Integrating Cisco ACI with Cilium: best practices

Oh, Welcome everyone to today's session. And the 1st thing that we I would like to show is the list of the sessions that we have within this.00:00Series, as you can see on the slide we have a QR code as well as the timelines for each of those sessions. You will see the list of the previous sessions as well that were already discussed in the series. My colleague will be sharing the community page on the chat which you could.00:20See the registration for the upcoming session as well as this is where we are going to share our recordings in case you missed.00:40Any of the sessions and you would like to go over them, you can see the recordings there. With that, let's start with today's session expert insight series, integrating ACI with celium. And mainly where we'll be discussing the best practices.00:48For this integration. Before we we go deeper into the session, please join our Slido under the session name ACI-iso. And then you will see the back the passcode aCI-ISO. Let me share both.01:04On the chat. So those are the session name as well as the password, the passcode. So passcode is ACI-ISO. Also you can use the I'm sending to all panelists. Let me just send to everyone.01:24Iso is the session name and the passcode is ACI. Also you can see the QRR code here on the slides where you can use this QR to to connect directly to the search.01:45Okay, so what we'll be discussing in this session as well as what, what will be the flows. So the 1st thing that we will be speaking about is the show me how. This is where we will be discussing how the celium, integration works with ACI and what benefits we can get from integrating it. We will have an introduction.02:03About the Kubernetes and the surveillance, we will address some of the challenges, speak about them, and then we will dive into each of these challenges and speak about the best practices and things that we need to watch out for.02:23After that, we will be this my colleague Nico here will be showing us a live demo on the ACI and CDM integration itself, we will be, he will be also showing us some of the best practices that we will be discussing. And then at the end we will have a short summary about our session.02:37And you will see you will receive a survey in case you are interested for one to one consultation or you have any questions that you would like us to discuss directly with you.02:56With that, let's start about what is Kubernetes. So Kubernetes is an open source container orch orchestration platform. It's designed to automate the deployment scaling and management of containerized applications. Originally it was developed by.03:08Google and then Kubernetes is now maintained by the cloud native computing foundation.03:27So some of the key concepts, let me just try to get the highlighter here or a lazer pointer yes. So some of the key concepts that we will be discussing or you need to understand about Kubernetes are mentioned here in this slide. So the 1st thing is the nodes. Those.03:34Are the physical.03:54Or virtual machine that serves as either the controller node or we sometimes it's called master or the worker node or in other cases it's called the slave within the cluster itself. The control node is going to manage the worker nodes or the or the slave node.03:57Nodes and it will manage the deployments, the ports that will be inside that specific node.04:16The worker nodes are the nodes that are going to host these services, these pods. And this is where our application workload will be actually deployed. 2nd thing that we just mentioned is the pods. This is the smallest deployable unit within.04:24Coordinators itself. This is where we encapsulate.04:44One or more container within a shared storage and network. Then each of these pods is going to be inside something called deployment. This is where we specify the declaration.04:48For the, for the pod itself. This is where we specify the amount of replications or how roll rollouts are going to be done or rollbacks. So basically this is how we define how many instance we would like to have from the application.05:06Then we have the services. This is an optional type for the thing within Kubernetes. This is the concept that we will use in order to expose an application within Kubernetes.05:22For for us to get this application to be reachable from the outside world, in order for us to expose it, we utilize the service. You have multiple kinds such as the cluster IP, notebooked load balancer et cetera, that can be exposed to get these services.05:42Running. Lastly is the name space. This is the logical isolation within the cluster and it's useful for separating environments. So this is what is used for segmenting or have a multi tenancy within Kubernetes.06:01So now that we understood what Kubernetes is, let's discuss some of the challenges. So the 1st challenge that we will be discussed is the East west. This is the communication flows that is happening inside the cluster itself, whether if it's from port to pod or port to node communication.06:22This traffic is automatically handled by the CNI of choice that you have within Kubernetes, and this is where comes the 1st main challenge. Since this is part to port communication or part to note communication, the traffic is going to be handled internally. The, our network is not going to be participating in it. So.06:41So the challenge comes into visibility. The these flows are really hard to.07:01For us to capture on the overlays and this makes troubleshooting hard especially if the Kubernetes networks policies are used. So it will be hard for us to detect the issues or even understand where the issue is. Now for the solutions.07:06We can use a CNI that provides visibility in depth within the Kubernetes. As an example for it is iso Variant and this removes the overlay so that it allows the fabric to see what.07:23How the traffic is actually flowing. 2nd challenge is the north to South. These are the communication flows that are originating from entities that are outside of the cluster and needs to consume any of the services that we are consuming.07:41And the main point here is that we are going to expose services to the external world. The main challenge comes into 1st the service advertisement. How are we going to advertise these services towards the network? Then load balancing, how are we going to load balance this traffic?08:01Between the different nodes that we have, the worker nodes that we have, and lastly is the security.08:21Is there a way for us to utilize a firewall or some security control on the, on the traffic that's happening? All of that is is a challenge that the default Kubernetes CNI is having. The solution is that we are going to run BGP.08:28Between the Kubernetes and the network, this provides us with a way for us to perform our service advertisement and load balancing, and gives us the option to create service insertion within the ACI so that we can, insert the firewall for this communication to be in the middle between.08:47The, outside clients or the external network and our Kubernetes cluster. Lastly, we have South to north. These are the communication flows initiated by entities within the cluster itself that needs to communicate with an external service or going to the internet.09:06Now by default.09:26The pod initiated traffic will always be source natted to the node IP. This is how Kubernetes works. This brings the challenge of how can we secure the pod initiated traffic to an external system as well as how can we differentiate how we can utilize the multi tenancy.09:28Within Kubernetes since if we have multiple pods within multiple deployments and namespaces, at the end, once the traffic is exiting the pod it will utilize the pod IP.09:48And sorry the node IP, and this will create a challenge from the network side because the network will not then be able to differentiate between these different team spaces or these different applications and then we will not be able to apply security depending on which application that's running. Now the solution for that is.10:03Also within isovalent, we have the isovalant egress gateway, which we will be discussing shortly.10:23So we mentioned isovariant multiple times during the previous slide. So let's speak a little bit about what is it exactly. So the main part of isovariant that's in interest for us is celium. It's an open source cloud native solution that provides.10:31A secure and a secure and another way for us to observe the network connectivity between workloads within the Kubernetes. The Syrium was actually created by isovalant, and then Isovalant was acquired by Cisco in April 2024.10:49Now, the isobellent solution itself is consistent of three main applications, CVM, and EVPF. We will use each part of them.11:09To achieve the things or fix the the challenges that we are facing within Kubernetes, and all of these are or these different entities within it are open source, but the enterprise version for this.11:24Whole solution is called networking for Kubernetes. Now, what are the advantages or the benefits that we get from utilizing these different systems? So the 1st thing is the advanced network security. So this is a fine grained security.11:41It includes fine grained security policies that can be defined based on the application layer itself and we can get also scalability and performance, so by using EPBFF we achieve a high performance and scalability as it can run network policy directly.12:00On the Linux kernel without the need for complex user space processing.12:20Also we can get service mesh integration, so it integrates with services mesh architecture to provide a transparent high performance networking and security features that complement the services mesh capabilities.12:25We will get observability so we can get robust observability tools that allows us to monitor and troubleshoot the network traffic and security policies in details and also with metrics. Lastly is that all of this is built or designed with the with the mind of.12:44Having it running on Kubernetes, so it seamlessly integrate with Kubernetes to provide the native support for network policies and services.13:05So now that we understood mainly what are the challenges, What's and what's eyes of valiant, let's start to dig deeper into the each of these challenges and how the integration itself manages it.13:18And what are the best practices here? So the 1st thing that I would like to discuss here is the north to south. This is the traffic that's coming from the outside towards our cluster. So as we said, the main idea here is that we are going to utilize.13:34The CDM to actually create a BGP session between our ACI and Kubernetes cluster where we are going to advertise these services through BGP. Now in order for us to do this, we can do it through the layer three.13:50Out which gives us the option whether we do it with a normal or standard layer three out or we can do it with a floating layer three out. And what's the differences between those two? So with the standard.14:10Layer three out, we will utilize the layer three out logical interface configuration. We will set up our logical interface configuration on on the fabric and we can build this neighborship with our nodes that are holding the BGP within Celium.14:22Now this brings us to the challenge that if, e.g., we had an issue with the node and the BGP node had to move towards another host. This means that also I need to have this BGP configuration or logical interface configuration to be added on all the.14:40Actions that I have with my host, and this brings up the scalability concern is that if I have a small number of hosts that are configured with nodes, this is easy to configure, but with increasing or going higher with the scale, in the cluster, it becomes more challen.15:00To do that. Now this brings us to the floating layer three out where it has a simpler configuration whenever it comes to expanding the cluster. So we will re retain our peering, but we will have the.15:20Connection towards the, between the, our leaf switches and the host within the Kubernetes cluster does not require any manual configuration so that the movement of each of the nodes will not impact our BGP session.15:39We will discuss that in the next slide. So here I'm going to discuss, sorry, sorry. Here I'm going to discuss some of the new terms that we have terminologies that we have within floating SVI. The existing terminology is the primary IP as you.15:57Let's see here. So the primary IP is the normal layer three out IP address for the leaf node. This is the real IP of the leaf node itself, and each of.16:17Our leaf notes will have a unique primary. Now we have new terminologies that we will be discussing with the floating layer three out is the the floating IP, which is the common and internal IP that we are going to.16:28Push towards all of our leaf switches that are participating in floating layer three out, whether they are anchor nodes or non anchored nodes. And since we mentioned that, so what is anchor and non anchor? The anchor nodes are the floating layer three out border leaf that has the primary IP addresses.16:49Configured and they are the ones that are going to perform the peering with our, celium part or with the other virtual router part, not only for sealium. So in this scenario, as you can see here, we have leaf one and leaf two are our anchor nodes. They are the ones that are establishing the neighborship.17:08With our virtual router, this is where my layer three out is and I'm having leaf three and leaf four as none anchor nodes. And those are the potential.17:28Nodes that in case of any failure, this specific virtual machine or node might migrate to these different hosts that are connected to it. So how does change the behavior of the floating, layer three out? How do we change the behavior?17:39How we establish this neighborship. So the the floating IP is going to be configured on anchor and then anchor nodes. And this is going to be used in order for us to actually.17:59Program the layer three interface of the floating SVI, and this means since we have the same subnet that's configured, we will have the same MAC address actually.18:11Programmed on the anchor and then anchor switches, which will allow the traffic for us to be switched directly through the fabric to be received from between the external routers and into our main anchor nodes. So what happens is that even though my virtual router moved to an anchor node, I will.18:27Also keep establishing my connection at neighborship with my anchor nodes, but this time instead of going directly point to point.18:47I'm going to utilize my floating IP configuration in order to reach it through the fabric. So that's why we don't need to keep configuring our interfaces. We will retain the neighborship with our anchor nodes, but we will retire, we will utilize the fabric.18:55In order to retain this neighborship. Now, since we have this neighborship done through the fabric, this brings us to some of the challenges within this integration. So the 1st one is asymmetric traffic. As you can see, since I'm building the.19:13The layer three out with the anchor nodes, then, will any other destination.19:33As in this example we have the destination on host four tries to communicate with the services behind our non anchor nodes in this example is host three, it will need 1st to go towards leaf one and leaf two. Because remember the neighborship is still established with those. So the advertisement that we are receiving.19:39On the network for the redistribution of the.19:59The external routes are still done through leaf one and leaf two through my anchor nodes. So from leaf four perspective, I don't know, the only thing is that I know is that this external route that I have is reachable through leaf one and leaf two. So the traffic will be sent towards leaf one and leaf two.20:03Then once it gets to leaf one leaf two, this is where we get into layer two where we utilize the floating SVI and we will understand that, yeah, this.20:23Destination is actually Vmac A which is connected to in leaf three. So we will go again through the fabric to reach to leaf three in order for us to get this into the destination. Now, how are we solving this? The way to solve it is through a feature that we have inside ACI, which is called next.20:34Next hop propagate. It means that I'm going still to use the redistribution that I have from the anchor nodes. So I'm going to read to redistribute the external route.20:53From the leaf one and leaf two, but I will also add to it a direct attached advertisement that's going to be coming from leaf three. So leaf three is also going to participate now and advertise the direct attachment. So what will happen is that.21:04I will get the redistribution, which is for my network 1000 zero/8I will say if you want to get this, the next hop for it is 1721611. So I'm redistributing I'm saying this.21:22Network is actually reachable through the BGP neighbor that I have. And then since I'm also getting a direct attachment advertisement from leaf three, I'm saying my cellium or virtual router BGP neighbor.21:38Which is 1721611 in this example is actually reachable directly through leaf three. And this way I'm going to skip the fabric communication and the traffic is going to directly go through between leaf four and leaf three.21:53So this is the 1st challenge that we have within this type of communication. 2nd thing is that we will bring another node into the question. So the 1st part of communication we only had one node, 2nd, now 2nd type of communication we have.22:11Two nodes that are going, we are going we are load balancing between.22:30So the the main challenge here is that we are going to have unbalanced the traffic since we are still actually dependent on the redistribution happening from the leaf one and leaf two from my anchor nodes. What happens is that I'm going to receive.22:35If.22:55The two routes from the external router in this example, the virtual routers to the same destination which is ten 00/eight. The the difference here will be that for the anchor nodes, the metric will be always zero because it's directly connected.22:55Neighborship to to to my leaf switches, but for the nodes that are connected through the non anchor connection, this is where we will get a higher metric. So in this example, we will get metric 100. This means that whenever we get into the BGP election process to to perform that redistribution.23:16We will actually going to be only electing the the route that we get through the anchor nodes. This doesn't make sense as we are actually hitting the load balancer main idea here since if one of my nodes had a failure on host.23:36One and we migrated all of our pods into host three. This means that I'm losing all the host three capability for for me to balance the traffic between.23:56Now the solution for that is, introduced in ACI 6.12 and later. We have a feature now that's called ignore IGP metric. The main idea is that now we are not going to take metrics into calculation within BGP and when we receive the advertisement of from.24:07Both sides for the 1000 zero/eight, whether if it's from my virtual router one or virtual router two, I will not take the metric into calculation and I will redistribute both.24:27Towards my other leaf switches. So what will happen here is that leaf one, leaf two and leaf three will be part of the ECMP connection that I have for this communication. Okay, now on this slide, I'm also mentioning other BGP enhancements.24:39Things that we can take into consideration. The 1st one is the BGP dynamic neighborship. So since all of our nodes will be configured within the same autonomous system number, we can utilize this feature where it allows us to leverage this capability.24:59To automate the establishment of BGP neighborship between our ACI leaf switches and the virtual routers that we have.25:19The idea here is that the ACI will become the listener. So it's, going to accept any incoming BGP session from a predefined IP subnet, so we don't need to manually configure all of our BGP neighborship in case there is going to be any cluster expansion, you can just.25:27Just utilize the BGP dynamic neighborship. 2nd one is the BGP graceful restart. This is the feature where it ensures that the BGP speaker will maintain its routing table within the if any restart happens.25:47Or if the BGP session terminates unexpectedly and the neighbor in this situation, the ACI will retain all the previous paths and it will maintain the connectivity through the disruption, provided that the data plane itself remains.26:07Correctly programmed during the restart, so we don't lose the data plane traffic.26:28Or the connection. So in case of a controller node failure, we lose the BGP process or it restarts. We don't want to lose the, data plane connection unless there is a failure actually happening in the data plane connection itself.26:322nd one is BFD. This is only available within the enterprise version, so you cannot use it with the open source version. And this is where it BFD simply provides with a with a more rapid and dependable detection of any link failure.26:51Which will give us a faster failover and enhance the overall network resilience. Lastly is the increase for max BGP ACMP path. And this is used in case of you having more than 16 nodes.27:10Of ECMP traffic. So by default, the ACI leave or the ACI itself will have maximum of 16 BGP ECMP paths. In case you require more, you can utilize this feature to increase it to up to maximum to 64 ECMP paths.27:27Now, on the right side also I'm mentioning some of the BGP hardening features that we can utilize where we can set up password authentication, set maximum AS limit into one since this is going to be directly connected when not we don't have any multi hop BGP.27:48Also configure BGP import route control to accept only the the expected subnets from this Kubernetes cluster, whether if it's the subnet, node subnet or the services subnet, and also to set the limit of the numbers of received prefixes from the nodes.28:09That I'm getting from the Kubernetes cluster. So now that we are done with North to south, let's discuss south to north. So while Kubernetes was designed.28:28For multi tenancy. As soon as the traffic tries to leave the cluster, the identity will be lost. So by default, as I discussed earlier, the port traffic that's leaving the cluster is going to mess or often will be masquarated with the node IP. So it will it will become hard.28:43For the external network or the external firewalls that you have to, differentiate between the different traffic. The solution for that is within isovelant. It's called isoveillance egress gateway.29:03This features allows for redirecting the traffic that's originating from the ports and disting to a specific CID CIDR that are outside the cluster to be routed throughout a particular node. What will happen is that once this, these policies are in place, the packets that are going.29:15Going to leave the cluster will be masquarated with the selected and predictable IPs that we have within these egress nodes. So as you can see in the slides, we have different deployments, different pods. Each of them are within the Kubernetes cluster.29:35Now we can specify this through a specific name space or a specific pod deployment, and we can set up multiple egress nodes. And we can say pod A will utilize egress node one and pod B will be utilizing egress node two, and then we can.29:51Now, predictably know each of these applications, what will be the IP that will be externally seen by the outside world? What will be seen in that group? Now the consideration here is that the 1st one is the redundancy for to for you to deploy a minimum of two egress nodes as recommended.30:11And these nodes to be physically connected into multiple or different ACI leaf switches. In case of a node or leaf failure, you will still have another egress node that can hold your traffic, and also it brings the question of whether to use dedicated versus shared within the.30:31Node itself am I going to use only this node for egress services or I'm also sharing other services to to be shared within this specific node? This will be beneficial discussion during the scale and requirement part.30:46And if you would like to have more resource isolation and or more specialized hardware advantages, you will go up with the with the dedicated part. Now for us to build this integration between celium.31:06And ACI, we can specifically for south to north, we can utilize two ways or option for this specific integration. So the 1st option for us is to do this integration through esgs or endpoint security groups, and this is where we.31:21We can perform the integration. So the serium egress gateway is going to be configured to use the IP addresses from the secondary subnet within my ACI bridge domain.31:41And these will be become, these subnets will become the fixed source IPs for the designated egress traffic. Then we utilize the acisgs, we create them, we utilize the IP sector inside them to match these, predefined IP addresses for the egress.31:53To to to to to manage to capture them within the e ESG.32:13And then we can apply our contracts or service insertion or whatever to these specific esgs directly, and it will rep so each of these esgs will be representing a specific application or egress identifier in this scenario. So as you can see here, we have two different namespace.32:18Just to make it clear. Each of these namespace is going to be to be mapped to a specific egress IP or egress node. So we have, we know that there is a specific egress IPs here, and then we create our ESG that will have.32:38Or two different esgs. As you can see here, two of each one of them is going to have an IP selector matching that egress IP. So this way I'm going to have the egress node mapped to my ESG where I can apply all of the contracts that I need to to hold the communication between the.32:55The ESG and the external communication.33:152nd way to do it is that we can utilize the BGP feature itself to, to, to advertise the addresses that they have as the egress gateway. Now the ACI here instead of me using the esgs, I'm going to use the externally.33:19EPGs that they have within the layer three out, where I'm going to classify these egress traffic based on the advertised egress IP prefixes and then this gives me the option to allow or to apply contracts to to to control the traffic that's happening.33:38So so similar to.33:58Just we just previously discussed with esgs, we will create a separate external EPG per egress gateway IP or subnet if we want, and then we can apply contracts to it. If we have any concerns about the scale limit for the ACI external EPG number.34:00We can also have the option to create a single external EPG to hold all of my egress subnets or IPs, and then we can utilize the service insertion.34:21In order for us to point out this external APG towards a firewall where we will be doing our multi tenancy control or policy enforcement directly on the firewall. Okay.34:32Just last thing to mention here for the South to North the challenge is that we will also since if you go with a second option of the BGP integration, all of the things that we discussed specifically for tuning and hardening within the.34:481st challenge is also, can be taken into the south to north challenge. Now the last challenge is the east to west. This is where it's hard for us to actually capture the traffic, and this.35:07Communication is or this challenge is being addressed by directly where we have Abel, which is a fully distributed networking and security observability platform. It's built on top of celium. So it will utilize Celium and APBF to enable the visibility into the communication that's happening.35:22Inside the Kubernetes. You will see that you have three main.35:42Parts for habble here. The 1st one is UI, this is where you can you can utilize a user interface to visualize network traffic between services. This is where you can see services map, who can talk to who, and troubleshoot connectivity issues visually. 2nd one is.35:47This is a terminal based tool or simply a hubble command. You can do queries and inspect on the network flows directly. You can debug issues in real time from the shell itself or you can do filter traffic by whether pods or namespace et cetera. Lastly, we have hubble metric. These are the.36:06Tricks that we can actually export from.36:27Our Kubernetes cluster into an external tool. In this example here we are mentioning Prometheus. The main purpose here is that we can provide aggregated and time series metrics enabling alerts and long term memory as well as we can feed these metrics into dashboards when we.36:30Where we can get statistics and in this example, we are mentioning Grafa.36:50Now in this slide, I'm just showing how the user interface looks like. We will be discussing that more in the or we will also see it in the demo. As you can see here, we can see the pod to put communication and in this communication we can actually get more details whether what's the source name space, what's this.36:56What's the source identity, destination identity, what are the ports that are being used? Also, what's the traffic direction and the decision that was made by Kubernetes. In case of the traffic being dropped, then you can review this drop reason and go into more details. Previous to Harvell we didn't.37:17Have any chance to do this kind of troubleshooting.37:36With that, let me pass the ball to my colleague Nico to guide us that. Thank you Saif for the great information in the demo. We will be walking through a few things. I will be showing you the.37:41The floating layer three out connectivity with a silium cluster, we will be covering a little bit basics around the silium cluster and we will also check some traffic patterns from outside to ingress.38:00Towards the ceilium cluster load balancer address as well as egressing from a pod inside of the cilium cluster. In this case, we have the anchor notes here peering with the ceilium, worker nodes.38:16And the actual physical links are connected to the non anchor nodes. And this is because we want to see the routing optimization. We want to avoid the additional loop or the extra hop, going to the anchor nodes when the actual connectivity is through the non anchor nodes.38:33And, also something around the ceiling cluster, what we have all already or we won't be focusing on, but we need also a cluster interface for, or EPG or another network segment where we do the cluster communication.38:53In between the ports or the node IPs.39:13We have a virtual machine to, use kube CTL and commands for accessing the ceilium cluster and when i'm using the word ceilium, I have here enterprise a version of the Celium deployed.39:16So let's 1st go look at the layer three out of optimizations and I have a small reference here, that's the different optimizations. What do they do? The next hop propagation is meaning that anchor notes say don't come to me, go to the actual next hop.39:34Which in this case is behind the non anchor node. A direct address host route advertisement is meaning that the non anchor node is telling that I have the next hop. This is the best way to reach the silium nodes. Then the last one in our IGP metric is use all cilium nodes, not only the.39:53Anchor node connected with are receiving a lower metric.40:13Compared to the non anchor node connected ones that are communicating with the anchor nodes through the fabric and that's why they are getting a a more poor metric, but with ignore IGP metric we can avoid that. So let's go and.40:17Take a look at the layer three out objects, so we have the top level object where we have VRF, we have layer three domain.40:33We will further see that we have also a physical domain with the same floating as we have available and we have BGP selected. Then on the node profile we can see the router IDs.40:51And we can see also the BGP peer connectivity here with the/twenty four address that covers both the ceilingum side addresses, the peer addresses. They're actually configured under the interface profile. So I'm going under the interface profile where we use the.41:07Interface type floating SVN, we can see the primary addresses here for the peering. We can see MTU, we can see in here and if we are clicking on it, we can see the physical domain and we can see the floating address.41:27That we can use then for the response traffic. If you have ingress traffic towards the sealing load balancer, the return back is not actually using BGP routes, but it's using Linux route table. So, something to note, sydium is not.41:44Importing any BGP routes, so we we leverage Linux route table and we can point e.g. to the floating address, for the return traffic or the traffic going.42:02Away from the ceiling cluster. I'm going to show what routes we are currently having without any optimization, so I'm using, nodes.42:17One anchor node, then one compute node that doesn't have the floating SVI, and we do show IP route VRF, the tenant's name, column.42:33The VRF name and what we are seeing here without the optimization we are looking for the load balancer address and the load balancer address on the anchor node is correctly pointing to the interface address. But that's not the case for the compute node.42:49Where this load balancer address is pointing back or pointing towards the anchor nodes. And this is the loop or the extra leg that we want to avoid, and also on the non anchor node, we have also the load balancer address is pointing towards the anchor node.43:09So let's take a look at the optimization. I'm going to walk through the three different things 1st in the policy section and then we will apply it in the layer three out in the right places. So under the policies and protocol.43:28We have route maps for next up propagation as well as direct attach advertise, and the next hop propagation is having then max rules where we have the load balancer subnets and the egress gateway.43:43Address which are the ones that are being published or or advertised from the CDM cluster. So we can take a look, e.g., at the service subnet and we have the load balancer subnet here. There is also next hop propagate match.44:03Set rule here under the next hop propagate with the next hop propagation selected. Then the direct access advertise is only having a mass rule for the next hop/thirty two addresses which are the node BGP peering addresses.44:21And there is NO set rule for the direct at that advertisement. We have then the IGP metric policy which is under the BGP best path policy, in our IGP metric.44:40And we will also see how to leverage this. So the 1st thing is, next hop propagation. And the next hop propagation is set under the interface profile and the BGP peer, and scrolling down we have the route control profile.44:57And I'm going to add here the next hop propagation route map in the import direction and then for 2nd node as well for the BGP peer, I'm going to set the next hop propagation into the import direction.45:15And now we can go and we can see the routes changing. I can avoid, I can remove the anchor node which was already correctly set, so there is NO changes needed there. But then the compute node.45:34We can now see that the load balancer address next hop is correctly set. It's now the node address, and also for the non anchor node, we have the correct next hop address.45:50But we have still one issue that the compute nodes which are not part of the SVI, they're actually doing another route lookup, so to reach the next hop address, they are, in fact, they are going to check.46:06Another route lookup, so I'm doing show IP route for this address again putting the VRF here with tenant.46:22Call on the VRF, and let's see what do we get for the Talos sorry what do we get for the IP address here? And actually.46:40That is still pointing towards the anchor nodes. So we need to use the advertise connected host routes and we are doing it on the top level item on the layer three out.46:53And scrolling down we have route profile for redistribution, and here I'm going to add it in the type attached host and we had the route map for the direct attached advertise and with the with the next hop.47:10Addresses and now update and we can check again on the compute node and we are getting now the next hop correctly pointing for the celium node addresses.47:26Towards the non anchor nodes, which are having the the best route. And then the final optimization is that in our IGP metric. I don't actually have the use case where where we have the anchor node connectedum node and non anchor node.47:43Connected, so I'm only going to show that, but we find the configuration under the node profile, and we have here creates BGP protocol profile, and here best path control, we can select the ignore igp metric.48:00And this would then leverage all the celium nodes even if we have anchor node connected and non anchor node connected cluster nodes. I I don't have that currently, so we are just seeing the configuration. So ok, those were the optimizations.48:18So let's next take a look at some of the traffic patterns and we have different traffic patterns like we have from a client virtual machine on ACI leaf.48:35Towards the load balancer address. So that is ingress. I'm marking that with an eye, and then we have another traffic pattern from a ceiling hosted port towards outside.48:51Through the egress gateway and that's going towards a database, also sitting behind a ACI leaf. And then we have also a 3rd traffic pattern, also was talking.49:07We had the east west traffic, so just between the ports in the ceiling cluster, oh I forgot to mark the egress traffic pattern. Eh, but let's take a look at the silium cluster and this traffic traffic types.49:23I'm going to take command line for a terminal screen for our control node where we can see some basics of the ceiling cluster I can do.49:41And we can see then the ACI leaves here. We can doum BGP routes, and we are seeing the advertised routes including the load balancer address and the egress gateway address, and as mentioned.50:02Celineum is not importing any routes, so we use Linux routes tables for that. Then we could also check Kubernetes notes, kubectl, get ports. Let's check the pods, A.50:18So we can, we can see that we have the infrastructure and then we have another name space for some tests load, workloads and ok let's take a look at.50:36The traffic patterns then so the ingress from an external virtual machine sitting under ACI leaf. So I have here a net cat command to the load balancer address to port 80.50:57And we can see that we are getting some traffic now from outside towards the ceiling load balancer address and I can I can stop the command and I'm going to use the hubble user interface that Siph was talking about.51:11And, ok, I have hubble here and we can see that this previous traffic is coming from outside towards our star Wars namespace and the death star deployment on port 80 and we can see the specific details on this flow. We can see that.51:29Our source IP is 8787 8787. We can see our destination pod. If we also look at our Kubernetes pods, here we can see the death start pod this one.51:49And we can also see the pod address 102-44-1607. We don't see in this view the load balancer but it's it's doing its job there. Then let's take a look at the traffic pattern.52:08From, the ceiling cluster egressing towards the database address here and I'm going to use here, I'm SSH into one, I'm going into one of the pods.52:26Here and I'm going to ping out towards the database address. And I have a small issue with the lab. I don't get the ping responses, but we can see the traffic on the hubble user interface.52:42So we can go to the hubble and we can see traffic initiating from the Star Wars deployment and it's going out, and we can also see the specific lines for this traffic.52:59With the source IP, that is from our pod now, and the destination IP 3530 05:35 and also the protocol we can see here. So we have one more traffic pattern east west. I'm going to show that.53:15So while I'm on this pod that is running on worker one, I can also connect to, a port running next to me on a different worker. So I'm running ping.53:33Towards my next point and we can also see that in the hubble user interface. So we have a traffic pattern in the Star Wars deployment.53:49To the star Wars deployment and we can also see the details with the source IP, and the source, source port worker one, next should test worker one, source IP and the destination pod and the destination IP.54:04So we saw also the east west traffic pattern. I'm going to also show what we can see from Nexus dashboard when we have traffic passing through the ACI leaves. So let's look at the ingress traffic from Nexus dashboard.54:20So next is dashboard I have here and I'm going to analyze analysis hub, and traffic analytics, and I'm going to change the time frame for.54:36A little bit longer so that I can also show, a specific collection of, of flow, a troubleshooting collection. Scrolling down we can find our service address, the load balancer hosted on port 80.54:52I'm clicking on that and we can then find the flow details and the flow health for the time period selected. We can see the client address here and we can see the details were in ACI fabric this endpoint is leaving.55:08And we can also see traffic statistics for the timeframe selected. I'm going to show the specific flow collection which you have to initiate manually for a limited period of time and you can get even more detailed records.55:24And I'm going to click on the flow flow records. I'm putting them in the time order and I can click on one of the records and we can see the traffic flow.55:39From the clients and leave spine interpod device spine leaf to the destination load balancer address hosted on celium. And we can see some traffic statistics for this time frame for for this flow capture.55:56And then I can reverse the traffic, and I can see also the response from the server back towards the client. And if you had any issues in the traffic, any drops, e.g., then we would have it marked red here on the nexus dashboard.56:14So those are the things I wanted to show around the connectivity with cluster as well as some traffic patterns. So back to you Saif. Thank you so much.56:30A great demo. Let me share again my screen.56:47So I hope you guys can see my screen. So, some key points to remember here is that the easiness and scalability that ACI and brings together. So BGP is well established protocol and that's easy to configure and troubleshoot. It's highly.57:06Scalable with thousands of nodes that can be established within the cluster.57:26The visibility that we mentioned multiple times where every communication flow can be actually monitored within the hubble and also the traffic that's traversing the network can be also monitored by our convention tool conventional tool such as the ACI or the Nexus dashboard as Nico showed us in the demo. Lastly.57:32The security where this is the is a cloud native security.57:51Enforcement that's going to be done by EVPF and also we will be utilizing network level security for the node to node or north ingress or egress communication, leveraging the fabric contract or service insertion.57:59And we have it. All of this is going to be netops ready since the Kubernetes the ACI and also the NXOS can be driven by the API which means that we can utilize Ansible or Terraform to automate all of these deployments.58:13So that brings us to the end of our session. Please note that you will get a survey once you close this session, we would really appreciate if we can get your feedback. Also, if you are interested for one to one consultation, feel free to to answer one of the questions that we have in the survey.58:36Be checking if you want this consultation to happen where we can directly contact you and get this consultation running.58:56Thank you so much, for joining us for today and have a good one.59:07