Multi-pod & remote leaf deep dive

Hi, good morning, good afternoon, everyone. I'm Sakshi and welcome to today's expert inside series session. So we are going to cover in the session.00:10Topics Cisco ACI multi part design and deployment.00:20So today we are doing a continuation of our ACI series, which in which we have already covered two sessions and in case if you were not part of those sessions, you can go to our community page and check out the recording also. Now, before I start with the agenda today, I just want to briefly.00:23Introduce the team who will be presenting this session to you guys. So in today's session we are coming from customer success team from data center technology at Cisco and we have Experts and I will also be there giving you a quick introduction and setting.00:43At this stage so we can discuss ACI multi part and remotely design and deployment in this session.01:03So before we start the session, I just want to quickly take a minute and share this QR code with you. We have also pasted a joining link in the chat panel. So in case you want to join using your computer devices.01:10You can use that link and you can also use this QR code scan on your mobile devices and the passcode for this Slido is aCI-three.01:30So this will help us keep the session interactive and will give us a chance to know about our audience also. So I hope you can take out a minute and join the Slido interaction which we have. And during this session, please feel free to reach out to us using the chat panel or the Q and A panel.01:42And bring us all your queries and questions regarding the ACI topic and with that, let me quickly move on to the agenda also. So in this session we will start with introduction, we will talk about what are the supported topologies in ACI and how the deployment.02:02Of apic cluster happens, and then we will cover inter part connectivity, deployment consideration. We will talk about remote leaf and integrations. So all of this we will cover in this today's session.02:22And, I'll just take another minute so that our participants can join in this session. And meanwhile, feel free to tell us where you're joining from.02:36What part of the world are you joining from? So there is a Slido which is active and you can see the QR code on the screen as well. So yeah, I'll just wait for responses from everyone. So we have someone from India.02:51Sweden, so lot of people from different different time zones. So welcome everyone once again and I hope you have a lot of learning experience from the session.03:11And, if you have any doubt, once again, please feel free to reach out to us using the chat option and the Q and A panel.03:25So we have about nine responses. Let me also let me also activate the Slido in the app section. So if you want to directly join the Slido, you can go to the apps option in the Webex if you have the Webex application running on your device.03:43And you can simply, send us your response through the Webex as well. So this is now active on Webex also. So I hope. So I hope you have a great time in the session, so let's start with the introduction.04:03So if you have already used ACI, you might be aware there are more than one deployment options there. There is single pod deployment, there is multi pod fabric, there is remote leaf design. So when we talk about these three options specifically.04:21So despite it having different deployment design, one thing which is common in them is that these deployments have a single controller domain. We have one apic cluster which is managing all of these deployments and then we have.04:39Multiple, controller domains and multiple fabric domains. So in that case, you will have ACI multi site. So some of you might be using a single pod fabric design and some of you guys might be using multipod remotely for ACI multi site. So you.04:56Like depending upon that, you will have different deployment options, different disaster recovery scenarios and so on. So we will try to cover as much as possible in this session for multi pod and remotely. And we also had another session where we have covered these aspects a bit in detail.05:14Which will be available on our community page. So let's start with the ACI multi pod design. So Cisco's ACI multi pod design represents a single Cisco APIC cluster and it is connected, interconnects the portion of fabric referred.05:35Two as parts while each of the port has its own leaf and spine architectures so as you can see here in this figure we have part A and we have port n, and both of these parts are there combined and there is a one combined APIC cluster and we have some produ.05:52Running like ISIS **** MPBGP, right? So ISIS is there for infra like reachability within a port, so whenever an ISIS stops working in one part, it does not affect the isis in other part because it runs only.06:12Between leaf and spine of each of the port for the tep reachability towards nodes in the other port, the spine learns the tep range from the other port instead of individual IPs.06:32Using the OSPF protocol and we have the Coup database which runs in each part containing the endpoint entries like and and there is a shared and stored cooper database.06:46On the spine and then we talk about the **** database from the other part, we have the MPBGP EVPN protocol between the spines which gives the user the information about like which we which gives the system the information about the **** database.07:04Which is there in the other part as well. And we have the MPBGP protocol also running over here, which is for the L three out route distribution within a pod.07:23So even when the MPBGP is there running in one part and not working in another part, again, it does not affect the other because it establishes neighbors only between the route reflector of the spines which is and the leaf switches in each part and distributes the L three route within the port. So this is like just the.07:35Brief introduction like what is multi pod? We have least fine architecture within each part. We have one part which has here in this figure has two apic, and we have another part which has one apic. But.07:55It it is a single APIC cluster which is managing both the ports in this example and we have VXLAN also connecting the spines and the maximum which is the time which will take.08:11Is 50 milliseconds.08:31In in this scenario where we have the interpod network, we'll see a bit of this in more detail when we talk about supported topologies of multi port. So ACI multi pod, it can be connected using this inter part network which we saw in the previous figure also. But with the recent version that is.08:32Dot two and beyond. So we have an option of port being directly connected without an IPN also. So in this scenarios, the in this scenario, the spine switches in one port are connected to spine switches with the other port.08:52Directly and it it should it does not necessarily require any kind of an IPN connectivity in case if you have only a two part system, the only constraint is that like this is only applicable, this back to back spine topology is only applicable for two port system.09:07But both the method supports up to 50 milliseconds round trip time and.09:27Supporting like supporting geographically diverse ports with this round trip time. So either it is even if it is running on IP connection or it is running on the, like direct connectivity.09:34It will be having same round trip times like less or around 15 milliseconds. Now one more thing is there in terms of connectivity, when we use an IPN based design, it will use routers and switches, it will require the directional protocol also for bump traffic.09:51Can OSPF BGP to run, but without IPN when we use this direct connection, so we have fabric connectivity which is connecting the spines directly between different port.10:11And as I mentioned, when we are using direct connectivity, we are restricted to only a two part design and IPN requires external networking hard hard hardware like switches routers between the port by the direct connectivity eliminates the need for.10:24Any kind of direct like any kind of IPN related hardware.10:44So in case if you have a smaller, you know, requirements or if you have some kind of constants and you just want to directly connect the switches back to back spine switches back to back, then you can use this kind of design as well. And another thing is that when we talk about this.10:50IPN without I like direct spine back to back connectivity.11:11So any feature which requires an IPN connectivity or, deployment like multi site remotely or APIC connectivity via L three network. So these things will not be compatible when we talk about multipot spine back to back kind of a deployment.11:16And we have back to back plus IPN being supported only for migration purposes. So just to remember like this can have some kind of a disruptive impact, so it is only supported for the migration purposes.11:34Now, one thing is spine discovers the connectivity using the LLDP protocol and these point to point connection must either be directly linked or they will have directly link connection or they should support the tunneling of LLDP packets packet.11:53To have this kind of a multi pod spine back to back setup.12:13Now it is not mandatory for all the spines in a port to connect to all the spines in another port. So you can use this recommended option which is also called like full mesh between the spine design, so where you can see each of the spine is connected to the other.12:19The Spine in the other side of the port.12:37And so on. So this is the full mesh design which is recommended, but you can also use a partial mesh design between the spine where you only need one spine connecting to one s another spine in the other port and same thing with the other spine and the other spine in 2nd part.12:40So this kind of design is also supported. So reducing the making it like minimal to have like this multi part setup without any IP network, you minimum need this partial mesh set up.12:58Now, next we will move on to the APEC cluster deployment consideration. So, as you can see here on the left hand side, we have an ACI fabric which has the leaf spine, the virtual machine, and we have the apic cluster, right?13:18Which is over here, but when we zoom into it, like it can be a single part, but when we talk about multi pod kind of a setup, so this is how it will look like. So we have a pod one, we have a pod two, we saw a similar diagram before as well and you can see over here that.13:37Right the IPN connectivity is also there through the spine switches and we have here two apex in one part one and single APIC in the port two side and we have MPVGP EVPN running between the spine. So this is like a.13:55General like scenario where you have a two part system, you have an IP connectivity and you have a three cluster cluster of three apex supporting the control controlling and like configurations of the.14:14Policies and like how you want to send out the data into the network. So let's see how this data which we are sending into the network and how it's.14:32Correlating to the apic, which is there in the ACI deployment. So in this, one thing which you need to understand in the ground level is that Apic cluster, it uses a large database technology which is called as sharding and the apic configuration.14:47Database is going to partition the data into logically bound subsets which we call shards. So to visualize it, you can see in this diagram over here, we have these three and each of this APIC has this.15:07Small colorful blocks in the bottom so you can imagine these small colorful blocks as the data shards and these data shards are replicated up to three replicas regarding regardless what is the size of your APIC cluster. So.15:23Here in this case we have three apics, so each of the apic is going to have one copy of the data shard and it will get evenly distributed across the apic. And this shard assignment is based on a predetermined hash function and let's move on to the next slide.15:41So in case of a single part scenario over here, right? Now we're gonna see while you are deploying, there, there are some failures which can happen, right? And, how this is going to impact the data which is there in the apic, and what kind of like standby.16:01Design you can consider, right? So in case of a single port scenario, one of the common failure which can happen is in case one or more of the apex stops working. So.16:22Let's see over here. So in this we have apic one and Apic two which have some kind of issue and now they are NO longer available to share the data charts which are in them for retry operation. So in this case, only Apic three will be available and since it has all.16:37All the copies of data shard, but it does not have the overall like majority and like basically it does not have the quorum. So in that scenario, it will be.16:55In read only mode. So of course you can you know like keep on functioning within your network of leads and spine, but this will not give you the option to have the right ability. So for that you have to, you know, like, whatever like you have to.17:09Either, bring up your backup apic, or you have to like fix the issue which is happening with the apic one and APIC two and heal the cluster, and once all the notes are up again, after healing, the data flow will start.17:29Now, hey Sakshi, talking about failure scenarios of epic cluster deployment. So I'd also like to share one of my experience with one of my expert customer who has a data center and DR into different geographical locations.17:48And they could able to maintain somewhere around 50 milliseconds of round trip time latency between two sites. So the requirement is they wanted a single unified fabric with consistent policy management across multiple parts, specifically.18:08With separate fault domains. So in that case, we recommended them to go further with the ACI multiple multi part deployments, in which case you could have your.18:28Epic cluster deployed in one location and just turn up your fabric office spine and leaf nodes in your data DR DR location and manage all across the parts from the single epic cluster which is in your DC.18:42So, that's about it now share going back to you. Thank you for sharing this detail about one of the scenario which you faced and yeah, thank you. I I hope like the customers who are joining us might have some similar.18:59Situations and it can be useful for them as well. Yep. Thanks. Okay, so next I want to show same situation, but we have five cluster on the right hand side and again the dual node failure scenario happens.19:19Now in this case since we have more apex, but as I mentioned earlier also when I was discussing the sharding part, so one, each shard is replicated at most three times, right? So now despite having five apex.19:38It will not create more than those three replicas, but it will give you the option, like it will give you in a way like more stability, right? So in this case you can see some, e.g., you can see there is a green chart, there is a blue, there's a dark blue one, right?19:58So since green and blue obviously do not have the quorum, so these.20:18Shards will only be in the read only mode. But while the the blue one, the Apic four and the Apic five, this, they both have the shard, so this can be in the right mode. So this implies that connecting one of the three remaining apic mode would lead to an.20:25On deterministic behavior across the shards as configuration can be changed.20:45For the blue shard but not for the green and the light blue one. So this will kind of have like a, like a mixed kind of, scenario like some will be in read mode, some some data will be in the right mode While you will still be able to run the configurations which are already.20:50There are pushed into your network, right?21:10So this is again like showing what what data will be in the read only mode and what will be in the right mode. So next we have a specific multi part scenario where so this is earlier we discussed.21:13This in the single part scenario. Now we have this multi part scenario, right? So usually we have a big nodes which are usually deployed in the separate parts while forming same one cluster. So in this scenario that requires specific considerations, right? Where two parts.21:31Two ports are part of the multi partnet fabric and like it can be part of like as Shubham also mentioned, right? Separate geographical data center site or say like different building floors, whatever. So these options will be there. So in this case, there can be two main.21:51In challenges which can happen. So here the connectivity between the two different ports is different disturbed. So in this case both the port will get isolated, but since port one has the.22:11This two replicas of the shard, so it will be able to perform the read write operation while this epic three because of only single epic three and part two, it will go into the read only mode and.22:27Another scenario which can happen in this two two part scenario is one whole part goes into failure because Apic one and Apic two are not functional anymore. So in this case.22:43We have a pod hard failure scenario. In this case, the recommendation is to activate a standby node. So as you can see over here, we have a standby node and make cluster fully functional again. With this standby node being activated and the data being.22:58And redistributed and replicated into the standby.23:18Now this standby node can be a warm standby or a cold standby which we will discuss in the coming up slide, right? And next is same scenario, two part scenario, but in this case we have a five APIC in the APEC cluster. So again in this case what happens if the connectivity be.23:21When the two part is disrupted.23:41So in case this happens, as you can see there is a mixed like the replicas or the shards, they are in the mixed form. So on the part two side, some data will be in the read only and some data will be in the right only mode, right? And same thing.23:44Happens on the port one side?24:04And it and once the ports are brought back up and the data will get synchronized then everything will start running again. Now, another scenario which can happen is the whole port one is down because Apic one, two and three, they all go in.24:06Some kind of failure scenario. So in this case, there will be some loss of information for the that are replicated across the Apic nodes in the field board and.24:27The Apic four and five will still be functional and while we bring up the while we bring up the APIC back up so this part of the port one will not be able to send out that data to the port two.24:40And the possible way to restore such kind of a state is either you can take a snapshot and revert back to the latest taken snapshot or you can again as I mentioned, you can use like a standby mom standby kind of an apic deployment which we will cover.24:59Next and you can also contact the view and the tag who can help you with.25:20This kind of a recovery scenario. Now next is in order to minimize the risk of lose losing all the three replicas for a shard, the recommendation is to avoid deploying more than two Apic node in the same.25:25So starting from the Apic four.O1, it is possible to deploy a four node cluster that can support up to 200 or like around 200 leaf node switches across two of the ports. Now, the consideration is like.25:46In this case, the split brain scenario can happen so the same data, same, the same data which is already made like which we already discussed sorry same scenario which we already discussed. So in case some kind of for like connectivity.26:04Between the two part failure happens. So in that case, what will happen like since we have.26:24This two way on either side, so we will be able to perform the read write operation in part two as well as part one while having some mixed information, so some, some data might be in read and some data might be in read write, so until the connection is brought back up. But.26:31In both of the port, we will be able to perform functions independently as well. But if in case one part completely goes down, then in that scenario, once again like in part two you will be able to make some changes. And once the port.26:51One is brought back up, then the data will be like reestablished and the heart failure. Instead of the like complete failure, like since we have here also a majority and the forum is there, so we will be able to have a proper, like.27:12Call like redistribution and reestablishment of connection possible, right? And we can also have a standby APIC over here. So in this case, what will happen the standby APIC will become Apic one and it will copy some of the data which is there.27:33And then again it will bring up the option to be able to do right operations from this majority of the APIC while this APIC one and two which were originally disturbed can be fixed.27:52So once all the shards, all the apex are regained, the majority of full majority of all the shards is also established. So this is also another example how you can deploy a four cluster and like where you can use the standby APIC in this case.28:07Now, as I was mentioning previously something about warm standby, something about cold standby. So what is that? So basically before six.12 release, we used to have something called as cold standby. So what it used to do is, it used to be like a passive listener, it used to.28:27To be there in the apic cluster but it will not store any kind of shards or any kind of data within it.28:47And only when something goes down and you manually, you know, like, activated and make it as your main part of your main apic cluster, then it will start NO replicating and redistributing the shards and take up an active role. But with the ACI release six.12, what is.28:53Possible is now you can have an active listener in, inside your Apex cluster.29:13What that means is this warm standby Apic will also have one copy of all the charts and it will be actively listening. So what we talked about right to have a, to, to have recovery option, you can go to the latest snapshot, but you can also use a warm standby apic.29:19So in that scenario, if something happens to one of your apic or something happens in one side of your port and all the Apex in that side of your pod goes down, then this warm standby can, Apic can come up and it can.29:39Help bring up the whole like apic cluster back up using the data it has stored, right? So, just to mention if you are already on ACI release six.one or above six.12 or above, then this feature will be available to you. And if you are on any.29:54Of the like below this release, then you might have to upgrade to be able to use this particular feature.30:14Right, now let's also see a few of the scenario where we are going to use warm standby apic, and how it's gonna have a different impact than the whole standby apex. So 1st we're going to talk about this single part scenario. So.30:21So so we have this five APIC cluster and one of the Apic four goes down. So what will happen if you have a core standby or if you have a bomb standby? So as you can see visual in in terms of visualizing this two types of apic, so the core standard.30:39Standby will not have any data while the warm standby, it was actively listening. So it has all the four shards, one one replica of all the four shards and since the epic four went down, now it can take place.31:00Of this apic IV, so poor standby or warm standby both either of them will take place of this Apic four which just went down and then again the data will be rebuilt and the cluster will start functioning. Now we will see this whole process step by step also in the next slide and another.31:12Scenario which can happen is that more than two apics, right? So three went down or and in in this case, now Apic four and five, these two apics are only left, so this might not have overall majority over the data. So in that scenario, if you have a warm standby data.31:33With full data backup, it can come up and it can support and bring up this whole cluster into a state where it can start functioning normally will be able to perform read write operations and everything. So let's see step by step how that happens.31:53So as you can see here 1st, we have this apic IV and we have a warm standby. So it's currently in a standby state and the APIC four is also functional, but once Apic four goes down, so this will become a new APIC four and whatever full data backup it has will not be used.32:11Rather it will be synchronized from the remaining apic. So this is what happens if only one goes down in your apic cluster and you have a warm standby. But.32:31In case, so, just a second. So once it comes up, it will, you know, like take data backup from here epic one and Apic two and it will take place as a new epic for serve. Now in case if you have more than one go more than two epic.32:41Going down. So let's say the apic one, two, and three are all lost. Then in that scenario, Apic four and five.33:01Are also not that useful, right? So in this case, the bomb standby will become the new Apic one and it will it will be it will also clear up the Apic four and APIC five for whatever charts were there and it will give them the new copy from the data backup this.33:08Standby cluster had, so in this case this bomb standby will be able to support and bring up the cluster back quicker in compared to a core standby which does not contain any data, right? So what happens when we have only one failure, we don't want to waste a lot of time in, you know, like eight.33:28The new apic copying all the shards to all the active apex. But if.33:48More than like in in case three Apic goes down, then in that scenario epic four and five will not have the majority of the shards and will not be able to give the full data. So in this case this becomes new apic one and the Apic four and five will become sort of like Apic two and three and will be able.33:54You have a copy of each of the shards which are there in the standby cluster.34:14Now, we have here another scenario, so in this case we have two part one and part two, and we have this five APIC system and one warm standby, right? So in case this apic.34:19One two and three, all of these goes down. This will not be able to perform any read write operations in the pod one. Now what will happen in this.34:39Thing, how will standby epic, perform the job it has to do is similar to what happened in the previous scenario, right? So again, like the standby APIC will come up as an active apic.34:49And it will take place of APIC one. Now it already has the data, all the data because it was actively listening. So now what will happen is the you as a user you can clean report the Apic four and five and using the apic, new apic one or the standby apic, which we had.35:06You can rebuild Apic four and five and make them as Apic two and three and you can have an active functional apic cluster on this side, right? So this is another deployment option you can have.35:26And like which will also support you in case any kind of recovery is needed or any kind of, you know, like, issue out there with the cluster. Now, how do you, change warm or core standby? I'm not sure if any of you have tried this warm standby.35:41And core standby feature before, but when you go to your system in the apic, you can go to the system settings option over here and you can see.36:01In the fabric wide setting, there will be an option of standby type. So if you have called standby and you want to turn your APIC into a warm standby, you can just click on this toggle and select. But again, as I mentioned previously, to have this warm standby feature, you need at least six thoughts.36:11One.two in your like six.one.two version in your deployment. So if you are on a previous version, you can upgrade and you can use this feature. And with that, I have one question for our participants.36:31So have you tried using warm standby feature of ACI while replacing any of your faulty APIC controller with the standby Apic controller? Or you still using the cold standby or you don't have, you know, like any.36:50Epic for standby feature, you can let us know in the comment and we have the slide also to understand if you have used or maybe you are planning to use after this discussion and what is the status? So we'll just quickly want to know where you are.37:07So I'll just wait here for 30 s so most of our participants can answer so we have 13 votes.37:27So, please feel free to let us know anything if you have also any doubts in the chat panel, any specific scenario which you have, you know, might be different from what we are explaining here so you can let us know in the chat panel also.37:46So we have some votes coming in C, so I think so far we have 50 % not waiting plan.38:0740 % not used but planning and 11 % saying that they have used and it's like already in their design, ok. That's good to see that a lot of people, even though they are not using it but they are planning to. That is good to hear.38:23Okay, let me wait for 5 s more and then I will move on to the next slide so next we will talk about APIC connectivity over layer three network.38:42Work. So.39:02So we have seen the normal apic setup, right? Where Apic inside the port as shown here, right? We have one over here, sorry, two apic over here and one apic over here in the port two. So each part will be assigned a tech pool address and based on that the ports are discovered over.39:04Were the IPN network using the DHCP relay and with this kind.39:24Or set up the apic, with the leaves which set up offers direct high speed low latency connectivity, and it's best for standard local fabric kind of deployment where your security protocol allows this thing. But we also have a second option where the APIC is connected over the IPN.39:28And, there is like a standalone apic cluster, so you can see over here the port zero.39:48So this allows the controller to reside outside the fabric connecting via the layer three. This is ideal for secure remote and like our shared service zone without any direct leaf connection in, inside your port, right? So we have a tep pool over here of one part, we have another tech pool over.39:56Over here of another part while the cluster here is on port zero. And one more thing is that the.40:16Apic over IPN connection, which is this design, right? Over layer three is, like, is different than the traditional setup, right? It does not connect to these switches, and in this case the IP assignment of Apic are manually done and it does not need to be assigned from the.40:23What that range.40:43And it can be over a different subnet altogether. So as we can see here we have ten.fifty.123, and all and here we the are completely in the different subnet. Now, one common use case for apic over IPN is.40:44It is designed for environments requiring security inspection, right? Where you have firewall between the APIC and the fabric and this kind of setup must be done using the initial setup only. So you have to decide initially if you want to connect your apic cluster with the leaf or you want them in a separate part zero kind of a setup.41:05Right, and next we have.41:24The apic, virtual Apic cluster option. So the virtual Apic deployment, it is allowed from ACI release six.O.two onwards. So you can run an entire three node controller cust cluster as a virtual machine on your VMware ESXi or your AWS services.41:28So e.g., the ESXi server is directly connected to the fabric and it eliminates the need for any dedicated physical server reducing the cost also. This approach supports smaller.41:48Cost effective deployment. So if you have a smaller requirement deployment, so in that case you can have a mini ACI fabric kind of a deployment while it requires a dedicated resource on your SSD and vme storage and a proper VLAN and network connectivity for that communication to happen between.42:03In your server and your ESXi server and your fabric.42:23Right? And when we talk about deploying this virtual APIC on AWS, you can use the cloud formation template and for the virtual Apic on ESXi host, you can deploy using the OVF template on the VMware vCenter, but one important point is that.42:27But NO, like mixed cluster is supported. So you can either have only virtual apic cluster running or you can have only physical running. And currently.42:47The virtual epic supports deployment of remote leaf multi port multi site, all of these deployments. Again, like in this also, there is a two mode of deployments supported. One is over layered two and one is over layered three.42:59So the layer two deployment which is supported in this the ESXi host on which the virtual apex are deployed is directly connected to the leaf switches and the ESXi host will then the the ESXi hosting the apex can then connect the.43:15ACI leaf switches within active standby uplink or an active active uplink using the LACP port channel. While when we are using the layer three type of deployment, the asxi host on which the virtual Apex are deployed is remotely attached to the ACI fabric.43:35Via an external network. So, one more thing I want to share over here before I pass on the ball to Shubham to share the next part so that what happens if you want to migrate from physical to virtual cluster migration?43:51So from six.one.one, you can now migrate from physical cluster to a virtual cluster and vice versa. There is a migration workflow which can handle this kind of migration including decommission and commission for all the nodes automatically. But as I mentioned, a mix of physical and.44:11Cluster is not supported. So whatever is your requirement and how you want to.44:31Do it either in virtual or physical, so both is possible. Virtual is possible from ACI release six.one.one. And with that, there is another question which we have. So which kind of ACI deployment do you have currently in your network?44:37So, let me see some people answering. So single part, multi pod, ok. So we have lot of people with single part deployment.44:57And, let us know in the chat box also, do you have any standby apic? Or, like you just have a three cluster, so what kind of you know like Apic deployment you have also you can let us know in the chat box.45:16And I can see there's like a good margin like between single pod multi pod and multi site, so multi pod and single port still near about near around each other but while multi site we have lesser in number.45:34So yeah, so next we will be discussing about interpod connectivity and deployment consideration. So I will stop my screen share and I will move, I'll pass it on to the next person over to you.45:49Yeah, thanks. So, yeah, so I see there are some questions being asked about virtual epic. But yeah, in the interest of time maybe you can answer those questions and I can.46:13Start presenting the next topic.46:29Go ahead, thanks. Yeah, so you guys are able to see the presentation yeah quick check. Yes. Yeah. Okay ok so yeah so yeah quick time check we will have two.46:34Not yet. We are not able to see yours.46:55Okay is it? Yeah, ok you're able to see yeah ok ok so I will be covering the inter part.46:58Connectivity deployment as well as data plane and other considerations yeah going forward. So, yeah. So Ashakshi had mentioned that IPN is required for more than two data centers or 4012 data centers if you are planning to scale it up to a bigger.47:18Multi port deployment. So, yeah. So what does the IPN need to support? So basically it requires OSPF for BGP to be configured on the IPN network and we also need a multicast bi directional PIM which is required.47:38For layer two traffic we will also cover like how the layer two forwarding will happen across the two parts in upcoming slides and if you are using OSPF for your IPv four reachability, then BGP.47:58May not be required and vice versa. So that is here about the IPN network and we also need DHCP relay so that the apex can discover across the boards, their cluster peers as well as if you add any new switch to a pod.48:18And the APIC is connected across the IPN, then you can add the switch and the switch will perform the DHCP action of across the IPN window. So these are the important things that are needed from the IPN. So it can be a non Cisco device, it can be a Cisco device.48:38And the configurations will will be.48:58Managed outside of the apic. So the spines will configure your OSPF and BGP what is required on the spines, it will be configured by the apic. However, the IPN configuration need to be done outside of the.49:02So yeah, as we were discussing so BGP underlay is also supported across the IPN network now like so we can, but yeah, be mindful that only EBGP is supported and this is starting ACI five.two.49:23Three onwards. You can either use OSPF or BGP.49:41And, yeah, like since this is EBGP there are some considerations that you need to be careful of because ACA fabrics themselves also use BGP like part one will have its own BGP configuration pod two will have its own BGP.49:46So, yeah, and BGP will have like lower AD value EBGP will have AD value of 20 so when OSPF and BGP both are configured BGP will be preferred in case you have configured both.50:06So, and this is also supported in your multi pod remote leaf and multi site. Yeah, so in case you are having nexus, which is in the IPN network.50:26Right, then you need to be careful like of this feature in Nexus that does not advertise the.50:46Routes if the fabric AS is same across the ports like both these parts have BGP AS number X as six five double zero one. Then, and if your IPN is having 65010 as the BGP AS and the routes received from.50:58Part one may not be advertised to part two by default, so if you have an access switch in the IPN network, you need to disable as check that will, that is like a.51:17Built in loop avoidance mechanism in Nexus switches non ACI so yeah you have to keep this in mind as a caveat to disable the PRAS check.51:33Yeah, and another important consideration for IPN is MTU so we need like nowadays we see the IPN network are capable to handle a jumbo MTU up to 9000 byte or even bigger. However.51:49Like if you don't have a jumbo MTU support on your IPN network.52:09So then how will you design the MTU for your inter part traffic that is explained in this section. So in ACI basically a data plane and control plane have different mtus. So what do I mean by data plane MTU and control plane MTU?52:16So control plane MTU is for the BGP updates and other control plane traffic that is being exchanged across the ports. So that will be like maybe whatever of 1500 or a 2000 by.52:35Or, by default there will be a value inside ACI, so the default value is 9000 byte. So when this like within the pod the control plane traffic is 9000 bytes, but if you don't have a.52:56Support for 9000 bytes in your IPN network, then how can you like change the control plane MTU that is explained in upcoming slides and by data plane MTU, we mean here that whenever a packet from host one goes to host two across.53:16The ports, there will be a VXLAN encapsulation for when the traffic leaves from spine one here and it traverses across the IPN. So you need to have 50 byte additional overhead for your VXLAN encapsulation that will be done at the.53:36IP and traffic. So we need to configure the data accordingly.53:57So yeah, yeah. So as I was saying, like under system and system settings, there is an option to configure the control plane MTU. So this is 9000 by default. However, if your IPN does not support 9000 bytes of.54:06IP and traffic, then you can bring it down to the supported value according to your IPN.54:24And, yeah, next is ok. So, so yeah like in this example, the IPN is only capable to support 1500 bytes, so we have kept control plane MTUS 1400 bytes here.54:32And we have applied it to the apic, cluster and here this is a remote leaf site similarly, you can do it on a port inter part like part one to part two, where you will be configuring the control plane MTU on both the apic clusters.54:51Yeah so yeah so how the TCP MSS sync.55:14Happens, right? So in this slide like how the traffic when apic cluster is trying to manage the remotely, it will send a TCP sync packet of 1460 bytes which is configured here. Assuming the IPN supports only 14.55:20Hundred bytes of MTU.55:39The TCP send MSS packet is only like the remote leaf responds with 9000 bytes, right? So, when it comes back it is anyways like fragmented so the TCP MTU is 1500.55:44Rights when it is received on the right so it will be like fragmented and it will be.56:03Like a DF bit set alright so because it is do not fragment bit set here on the IPN. So, if, if your IP IP and MTU is less than what the apic is configured with, then there is a chance that you may not be able.56:12Able to discover the remote leaf or the apic, hardware which is connected across the ports or the leaves as well. So this is a important consideration that you might need to be aware of to configure the control plane MTU. Okay.56:32Yeah so ok so how do you configure the TCP MSS policy? So it is under systems and system settings, if you go, there is a configuration of TCP MSS policy. So.56:52There are different options and in here that you might need to be aware of. So if you apply to global, it will apply to all your multi pod multi site remote leafs and remote leaf to local leaf traffic as well. If you are only worried about.57:11About like if your multi pod has.57:309000 bytes of IPN support, but your remote leaf does not have, then you can select another option like remote leaf only, right? So it only applies to remote leaf and part one to port two traffic will still send that higher.57:34Yeah ok so yeah so here we are again explaining how the TCP MSS functionality is achieved using the TCP messages, right? So, if you see here like when the host sends a TCP syn.57:56Packet it's a three way handshake of TCP that is happening in the demo here like we'll show. So it sends the packet with 1460 bytes of.58:16Of TCP MSS in and then we have configured the syn packet like the TCP MSS MTU of only 1400 byte across the APIC cluster. So when the host sends it with 1460.58:30It will be like, sent across the IPN and the ACA like the remote leaf which is receiving the packet will check the source IP and the destination IP of the VXLAN packet. And then when it is trying to send it to the end host.58:49Of the port two, it will change the TCP send MSS to 14.59:09Hundred bytes because, the source IP of the VXLAN packet and the destination IP of the VXLAN packet is in different parts. So that's how the logic is configured in the ACI fabric that if you if the source IP is from the remote pod.59:15Like a different pod then the MSS packet will be sent MSS adjustment will be done before sending it out to the host. And then.59:35Again like this host is sending to the host two from port two is sending the TCP synac packet. It will be sent to pod one remotely here and when we when the lookup happens on the leaf, then.59:47It will be.01:00:07Again checking the same logic that it is from a different type pool and then it will adjust the same packet as 1400 bytes. So, once the TCP handshake between the hosts it's complete at that time they will like going forward now they will only send the data packet as.01:00:081400 rights and not eight 8900.01:00:28Or 9000 MTU, right? Yeah, ok. So, yeah, so moving to the next topic, it is control plane and data plane. So we like.01:00:32We have till now we have seen like how the apic cluster is deployed and how the TCP MTU and the control plane MTU and other policies are being set. Now we will see how when the pod is actually built up, how it sets.01:00:50Of the control plane of the ports, right? So assume you have one apic, connected in seed part one like we call it seed part because that's where we start the onboarding of the APIC and ACI switches. Okay, so APIC will discover all the local.01:01:09Present switches as the usual procedure one by one each of the switch will be discovered and provisioned and it will also provision like you will configure.01:01:30Multi pod configuration on the spines once they are provisioned, so it will configure whatever OSPF and BGP neighborships that need to be configured on the spines. And then the, we are assuming the IPN network is already having the DNS.01:01:46CP relay of the like it will relay the DHCP packets to the APIC cluster.01:02:05And again like the spine of remote ports, right? The port two spines, they are yet to be provisioned, so they will be sending out DHCP request as a usual procedure across the.01:02:12Across the IPN network as well, since they are like assuming they are in factory default mode, they will send out the DHCP request on all the ports which are up, right? So it will send a DHCP packet and it will get relayed to the apic. And then Apic will respond.01:02:32With the like assign the IP to the spines and then again it will carry on the discovery procedure of the.01:02:51Other leaves connected to the spine and the same DHCP assignments will happen across the for all the devices, right? So, and in turn it will also detect a new apic, which is connected to the port two.01:03:01Right, and form the cluster, right? So this is how like multi part will come up during the provisioning phase. And if there are multiple pods the same process will repeat as it is, right?01:03:17So, yeah, so in this slide we will explain like what all OSPF and BGP routing will happen across the IPN network and how the routes of the VXLAN tunnel endpoints will be exchanged, right?01:03:33So when you configure the ACI like spines which is for the IP and connectivity, the, like, the ACI fabric runs isis routing protocol as under the protocol. However, on the spines, there is.01:03:53We are configuring OSPF for BGP for the inter part connectivity. So, there will be like we are assigning the port type IPs from this subnet like 1000 00:16 on this part and for part two we are.01:04:13Explaining like for part two we are setting the type IPs as 1010062, ok? So when the OSPF or BGP neighborship comes up across the IPN network, it will redistribute all the.01:04:33Type endpoint IP subnet range to the IPN network. So this IPN network will have all the tunnel endpoints IP in their OSPF or BGP database, right? And once whatever is received from the IPN like the remote.01:04:48Type endpoint IPs will be redistributed into ISIS by apic. So this will be automatically done. We don't need to do anything special as soon as we configure the.01:05:08IP and connectivity, the redistribution will also be taken care. And if you see the leaf, routing table for the VRF overlay one, you will see the 10100 pod IP that the IPs of the remote port.01:05:22We will see the next hop as spine one and spine two on the leaves. Yeah, so and it will also perform ECMP for the same so your traffic will also be.01:05:44Load balanced across the links from leaf to spine, right? So, yeah, so there is one important thing which you might need to keep in mind when using multi pod network. So by default in ACI the spine are configured with.01:05:59Fabric wide value of 63 when the spines are set up. So this is the ISIS fabric wide style metric, right? So, like, it is configured with 63, so when there is a upgrade or.01:06:19In one of the board, how will it impact your network? So this slide explains about that. So as I explained in previous slide, there is a OSPF and BGP configured across the APM and there is a redistribution happening in the.01:06:39Individual ports for the type routes. So if you see under your settings and ISIS policy, you will see your ISIS metric of redistributed routes configured as 63. So this is the default value which is configured on your port.01:07:00Right. However, if you are like upgrading a spine, right? Then what will happen? So assume you are upgrading this spine of your network and your like this node is getting upgraded so it is in downstream.01:07:19And it will redistribute the remote routes with ISIS metric of 64, right? So effectively like this is still what do you say?01:07:39The traffic is anyways overloaded like I am talking about the isis overload bit. So, but this spine also sends it with a value of the same metric, right? So what happens is when the.01:07:51Yeah, when the leaf is sending the traffic, it still sends out the traffic to both the spines. But since this node is getting upgraded, it will drop the traffic. So there can be some loss of the traffic, right? So, you can set this value like.01:08:08Change this value in settings isis.01:08:28To any value which is lower than 63. This will help you like impact any traffic disruption when the spines are getting upgraded, right? So, yeah, that is one consideration while configuring multiple, right?01:08:32Yeah, so any questions that I need to take or I can continue quickly for the next day? I think I think all questions are answered go ahead.01:08:51That's.01:09:12Yeah, ok, thanks. So, yeah. So still now we have seen like how do we provision the parts how what all the redistribution happens in the background on OSPF and BGP now going further like we will see when, when the actual.01:09:12Pod traffic comes right from leaf in one part trying to send to the leaf in another pod, how will it send the traffic? So how the control plane comes into action we will see in this. So.01:09:33When we have configured the inter part network, even though like once the reachability between spines is established, the spines of pod one are able to reach the, the spines of port two, they also have their.01:09:53BGP VPN neighborships come up directly between the spines. So it will be a full mesh of BGP VPN neighborships across the spines, right? So when an endpoint one tries to reach to another.01:10:09Leaf in the remote part, how will the traffic flow? So this endpoint is trying to send the.01:10:30Packet right now, it is having like its cooper database and it is a proxy hardware proxy enabled, right? So endpoint one like the leafs will send a hardware proxy to spine and then there is a EVPN neighborship.01:10:39Between the spines so that will also be sent across to the remote port and then these spines will also update their EVPN like VXLAN tables or group database you can say where they will store the value that ok endpoint one is.01:10:58Proxy A.01:11:19Right? And then again they have their own, like, so all the endpoints which are present in part one here will be present in the Coup database of the spines here, right? So it will be sent to proxy A endpoint one and two, However endpoint.01:11:20Three and four will be presenting their local leaves and similarly, if you see the database here and .1 and two are connected to local leaves and three and four are connected to the proxy B here, right? So they will update that.01:11:40Database, right? So, ok. Now we will see how the traffic flows different kinds of scenarios right so with broadcast and unknown unicast as well right coming. So.01:11:59Yeah, so when an endpoint one sends the traffic to like wants to send the traffic to endpoint two, right now it is not known to the leaf here, right? So leaf will send a hardware proxy to the spine and the spine will also process.01:12:17The packet and it will also send the proxy to spine like a spine in the port two, ok? And it will send as a encapsulated packet with the class ID and the VNID configuration in the packet, right?01:12:37So when the spine of the remote pod receive the traffic, they will since it is a CPU packet, it will be processed and it will also update its.01:12:57Like a local database and the destination of this traffic like is endpoint two, which is present in the coope database of the spines, right? So endpoint two, it will send the traffic to leaf four the spine.01:13:12And we'll send it to leaf four.01:13:32And it will also update its own table that endpoint one is connected to proxy, right? And the leaf four will receive the packet and the like leaf here in part two will also update its table and the VM.01:13:34Database that endpoint one is connected to port one leaf four, right? So now it has the information that endpoint one is connected to part one leaf four. So going forward it won't do a proxy, right?01:13:55So and then assuming that the contracts are configured from both the endpoint one and endpoint two EPGs, the traffic will be forwarded to the leaf and endpoint, right? So, yeah. So now when the remote leave and.01:14:10Point is responding back with the reverse traffic. How will the traffic be forwarded? Now, like if you had observed the leaf five here will have the endpoint as part one leaf four. So it will, it won't send a proxy to the spine, it will directly send it the traffic.01:14:30Two port one leaf four here and it will be sent to the endpoint one immediately. So there is NO proxy involved in the reverse traffic.01:14:50Yeah ok so yeah and from this point onwards both the leaves will have their own like tables updated and they will directly send the traffic across the board, right? That's how VXLAN will.01:15:00Help us in this picture right ok so yeah ok so now like we need to consider the what all are the multicast like as we mentioned bi directional is needed in the IPN network.01:15:19So why is this needed? So, whenever there is a broadcast or unknown unicast or multicast traffic traffic basically, it will be sent as a multicast traffic across the IPN networks. So why it is being sent as a multicast so that and.01:15:39Specifically in that by pin bi directional so because the hosts can be present in any of the like endpoint one present here and .2 present here. Both of them can be the source for the bump traffic, right? So pin bidirectional gives us a.01:16:00Scalable star G entry. So if we don't have been bi directional.01:16:20It it might cause too much SG entries in the IPN network so that is not recommended for your IPN network. So pin bi directional is needed, right?01:16:26And yeah, assume the endpoint here sends out a broadcast traffic or a known unicast traffic, so this leaf will like flood it flood to the GPO address for that particular BD, right? Which is configured.01:16:42Here. So then the spine will also like one of the spine will act as a authoritative for that particular BD. So in this case we are assuming the spine one is selected as the authoritative spine, ok? It will also send a IGMP joint towards the.01:17:01The IPN network, right? And then similarly, we will also get the traffic from the remote part from the IPN network, right? So, that's how the multicast tree will be formed. So continuing like the discussion further, if we see the endpoint one is.01:17:21Traffic, it will get flooded by the GPO multicast address and then spine one will send it to the IPN network. Again, IPN will also forward it to the remote part and then.01:17:43Assuming the spine one here again is selected as the authoritative, it will send to all the leaves, ok? And then .2 receives a bump frame. So now, once it is received, then .2 will.01:18:01Yeah, sorry the endpoint two will respond with the corresponding unicast message and it will be handled by the interpod network and the ACI Cope database, right?01:18:17So yeah ok so what all configuration maybe needed on the IPN network for related to the bi directional, right? So we will configure all the like multicast groups and RP addresses on the.01:18:32IPN so here like we are recommending to have phantom RP concept of the like being bi directional RP, right? So one of the IPN device will act as a active RP. However, the same RP address is configured on the.01:18:53Another IPN three, but that is like longer prefix, right? So, like IPN three will not be preferred till the time IPN one is being handling the traffic, right? So if you see here/ten.01:19:12And/twenty nine. We are configuring the same IP address. So this is the phantom RP concept. So this will be.01:19:32Like IPN one will act as the active RP all the time, only when it goes now it will be transferred to IPN three, the RP role will be transferred to APN three. And this is another design where you can use multi pod setup. So if.01:19:41You see not all ports have APIC here. So if you have many smaller DCs spread across the.01:20:01No like across the geographies as long as you are able to achieve the latency requirement of 50 milliseconds, you can configure like you can have your apic cluster spread across the DCs and you can configure the DCs with the same.01:20:10Yeah, apic cluster using multipot network.01:20:31Okay so ok so network services integration. So, we will see here like what kind of firewall support is possible in multi part, right? So if you see here.01:20:36Like, you, you can have your active standby firewall spread across the boards that is like supported in ACI multi part, ok? And these are the like anyways.01:20:56You have most of the times the pods are spread in same physical building or any like very close geographies within the same city. This kind of design is generally found in a common multi pod deployment where active is on one pod and.01:21:16Or you can also form a active active firewall cluster across the ports that is also supported. However, like, independent active standby cluster across the board requires like asymmetric traffic handling and.01:21:37Other things to be kept in mind. So this is not generally used in ACI multi pod.01:21:56If you remember from our earlier EIS series, the last EIS which happened last week, this is a more common scenario for multi site network and it is supported in multi site but in multi part these are the typical use cases that are supported in AC.01:22:04Right, so, ok, with that I will stop my screen sharing and we'll go to for thank you for a great session today.01:22:25About and epic deployments across the boards. I think we have a hard stop in 7 min and I feel like it's pretty challenging to complete remotely in such a shortest pan of time. So what I would suggest.01:22:41Why don't we take this time last six to 7 min to understand our audiences? I just have a question for all of you today. Since we have covered multi site, we have covered multi part, would you be.01:23:01Interested in learning how, the, how we connect ACI single pod multi pod and multi site with external layer three networks using layer three out. Would you be interested? You can respond by just sharing a thumbs up.01:23:21Or you can respond by saying yes in the chat Nice. Perfect. So.01:23:39I have another topic for you and so this was about connecting your ACI with external layer three networks. The another side would be connecting your ACI multi single pod multi pod and multi site with layer four layer seven devices like firewalls.01:23:57Said load balancers. Would would you be interested in learning about that integration?01:24:17Yeah, we have lot of yes coming in. Yeah, this is pretty good. I think this is good. So what we can do, we can arrange a session on.01:24:23How we connect our external layer three networks with ACI multi part multi site and single part deployments and along with the network services integrations and send out another invite to you with a different expert inside series webinar.01:24:41There's also a question if the remote session based discussion would be taken up in a different session. Yes yes yes, we will, we will cover it in in a different session. If, if you guys don't have a.01:25:06Hard stop. We can push another 30 to 40 min today to have it covered. What do you say? You want it to cover today or would you like to have me arrange another session for remotely? Okay.01:25:25Okay, got it. So please watch out for an invite, another invite on expert inside series webinar or remote leave. Okay, yeah. It seems like most of the people.01:25:42People are saying for a different session.01:26:02Yeah, so we will consider that and we will we will thank you today for joining us here today. And also just before you go, I would ask you.01:26:05To have fill out this last slide as well. Yeah, so before we drop there is like on 26th of March, there is a 60 min live Q and A of.01:26:24About the topics which we have covered so it will be distributed networking focused Q and A So you can come up with your questions on multi pod, multi site.01:26:44And we will try to answer them in that 1 h session. So do join us on 26th of March same time. Yes. Okay, with that.01:26:57I would thank you all for taking your time out to learning a deep dive on remote multi part session today and I would also ask you to watch out for an, another expert inside series webinar on remote leaf and subsequent.01:27:13Topics which we have discussed today. So thank you team, thank you everyone. Thank you everyone. Thank you. Thank you, everyone.01:27:33