Deep Dive: Data Center Networking for AI Workloads

Hello everybody. Welcome to today's expert insight series session on deep dive data center network for AI workloads. My name is Riley Norris and I.00:09We'll be your moderator for today. We are also joined by experts Dave mccarlan and Jorge Alonzo who will be taking us through the sessions best practices.00:20During the session, we will also be answering live Q and A So feel free to type any of your questions into the Webex Q and A panel or the Slido panel, and our panelists will be with you. Speaking of Slido, we do have a brief Slido question here for everybody.00:30And then we'll have one at the end for a quick survey. Our 1st question here is, what describes your primary role? It looks like we have a couple of network architects, network engineers in the call, some data center operations folks, systems engineers, solutions architects.00:49Awesome. Okay ready to start Riley? Yeah, go ahead and wrap up our 1st Slido and over to you Dave. Thank you. Thanks. So just a quick overview of the flow of the session. We are.01:17To be looking at the system, looking at what we can provide for AI infrastructure. This session will be mainly about the networking aspect of AI.01:37And how we can incorporate in the Cisco solution. We have other sessions that talk about the compute platform and the GPU platforms, but this specific would be talking about how we created a lossless network, how we can make use of the next Nexus dashboard controller for telemetry and configuration, how we can make use of existing ACI or.01:47VXLAN fabrics and then our new hyper fabric and hyper fabric AI cloud based solution. So we'll talk about how we can use AI and ML in Nexus dashboard.02:07Go over some overviews. We've got some pretty good demos and you can actually log into some of these and try them yourselves. So we'll talk about that as we go along. Again, we've gone through several sessions so far, receiving this.02:19Talking about the AI portfolio and we've got some additional sessions coming after today's session. There is a QR link that will go to our registration page and give you more information about the sessions that we've had and are coming up. And we are going to complete those with a.02:36Recap of everything. Again, today's session will be on networking for AI workloads and the follow on sessions on compute and real life use cases and autonomous networks.02:53So let's dive right in. We, when we talk about the IAI MO infrastructure, there are a lot of parts that are going on into the system. We've got our traditional front end network. Let me see if I can get this going. And this could be our ACI or VXLAN network.03:09We've got our storage network, our management system. This could be ACI or Nexus dashboard, and we've got our GPU compute back end network with our GPU nodes. The idea is to provide you with a lossless.03:29Priority flow controlled networks so all your devices can talk to all your devices with the least amount of interference. Let's talk a little bit about what could be interested to you.03:47And talk about the differences between training and inferencing, NO to node bandwidth requirements for training is very high because of your GPUs. Your GPUs want to be talking to each other with the least amount of interference.04:06And the lowest amount of latency and the highest amount of bandwidth. Most of the time the communication model needs to communicate with each other, but there needs to be a line rate and, you know, the highest possible bandwidth available. One of the key metrics for training is the training time. Gpus are very expensive, so you need to.04:19Complete task as quickly as possible without any congestion or latency.04:39Job job completion time is also an important key metric. An operational model that is most of the time offline, but for you is available when you need it. Each training or fine tuning requirement you can need a relatively large workload.04:44And we'll see examples of this in a couple of slides. We talk about inferencing, the node to node bandwidth requirements is relatively low, but you'll, you'll have it when it's available. You also want a low latency network. These are these are real time requests on usage of the train model and on your environment.05:00It needs to be be highly responsive and has to be, again, a very low latency and high availability between the user and the model. If you look at some of the key requirements and how they all work together, we'll talk.05:18Talk about how we can address some of the Cisco equipment.05:38So for training and fine tuning, we've already mentioned we've got a high bandwidth requirement between your GPU communication and there maybe large data sets that are moving between your GPUs. This traffic needs to be non blocking, lossless.05:41Any networking consistencies can affect the accuracy of training time, the AI modules. We also want to have some type of congestion management. A GPU is not only talking to a GPU. It could be one to many or many to one. And even though you have a one to one non oversubscribed network.05:59This could relo result in traffic ingestion. We need to be able to redistribute or move traffic along different links as needed. For inferencing, you need low latency for real time AI applications extreme require extreme low latency, so these are some of the techniques we'll be talking about in the.06:19Next slides like RDMA and some of our load balancing techniques to give you that low latency. Again general requirements are you want to use a validated configuration. Ideally, it'll be template based, so you can just pick and choose what you need.06:38And then it'll make the deployment much easier. You also need tools to provide you with visibility, monitoring, understanding where you maybe having hot spots in your network and troubleshooting and detection. So these are all the things that you want to have in your design to make it easier for your data scientists and AI.06:54The Engineers to work with the system.07:14We're introducing some new switches along the line. These are their 400 gig switches, several models mainly having to deal with the density of the ports and the size of the chassis. They all have the fairly high bandwidth. Everything is one to one on the fabric.07:20There is NO oversubscription and we're making use of some of the most advanced ASICs for our systems. All of them make use of quad port sfps, some are double density, and that's that's what gonna give us a 400 gigs.07:41They have both, cloud port and optical port as required by your network design. We're also introducing an 800 gig fixed switch, again, it's gonna be running on the.07:58Our silicon 1G2 hundred A six. We'll be talking about that in later slides. This can function as either a spine or leaf and it's a very powerful switch and available now. It is a fixed unit, fixed rack port unit, 64 ports of 800 gig, and this is on quad SFP double density.08:14And let's talk about what's required for an AI ready data center. So let's focus on the front end. You know quite a bit about this is our our traditional ACI fabric or our VXLAN fabrics using Nexus dashboard. This is how you interact with the AI.08:41System. You're probably very familiar with it right now. Our ACI fabrics and our VXLAN fabrics are designed to be any, any connectivity and the bandwidth between them is determined by how many links you put place between the leafs and spines. If you look at the front end, you'll see there are different networking types, where do you know how to build most of these.09:02From scratch.09:22But you can make use of your existing ACI or VXLAN fabric for your AI ML workloads. If you want to, you can also repurpose your ACI environments, redeploy a small EVPN fabric, and again, everything is visible via ACI or the Nexus dashboard.09:23So the Nexus dashboard can run on top of both ACI and our VXLAN fabric, provide you with both configuration control and management and monitoring. And in some cases we can do pre change analysis and display a historical telemetry.09:43In our back end, we have our classic design which is servers connected to leaves. The servers in our case generally have GPUs, and we want to be able to have those GPUs talk to other GPUs in the system. In our classic design, this does not give us the most optimal.10:06AI ML infrastructure. The industry is moving towards a rail based system.10:26In which case, each CPU node could have multiple GPUs. In our case, a CPU has eight GPUs and we want to connect the GPUs to the.10:34Back end infrastructure in a matter that each GPU is connected to each leaf. So in the case of node one on the left, if there's eight GPUs in this chassis, then each GPU is connected to a leaf. This is our rail based design.10:49And this gives you the ability to have everything, one to one connected to every leaf, and every leaf is one to not one connected to every spine. So again, technically everything is one to one, non oversubscribed, and everything can talk to everything.11:09But just because everything can talk to everything doesn't mean you're gonna get the best possible bandwidth or connectivity between nodes. And this is where some of our best practices, some of our protocols and some of our features will provide, help provide that without collapsing the fabric under heavy load.11:26Let's talk about some of the techniques that we have in our system. If you look at our fabrics, if we're doing each node has eight GPUs and each GPU is connected to a respective leaf GPU one is leaf one for every node GPU two is for.11:49Every node to every leaf. Communication seems pretty straightforward in some cases. If your, if your traffic is going from GPU one to GPU one in a in node one and node two, then it's pretty simple. It's gonna go up to the leaf and go back down to the next GPU.12:05So just a single hop to go from GPU to GPU. Now what if GPU one needs to talk to GPU two? It seems logically it would go up to the leaf leaf one and go to one of the spines and then back down to leaf two and then get to GPU two.12:25But some of the optimizations that have been going on with the industry and with NVIDIA, we can actually allow the traffic to go through the GPUs, so in the same node, traffic will transfer through GP two and then up to leaf two and then back down to GP two. So we eliminate.12:44An additional two hops in the system by making use of this traffic between this traffic order between GPU one and GPU two. This is part of nvidia's PXN or PCI NV link that allows traffic to go from one GPU to another GPU within the same node.13:04And allow it to talk to another GPU. Again, there's protocols that allowed this to happen, it understands it has to go a specific node and then there it can figure out the fastest path to get there. Again, this is a part of nvidia's.13:23PXM, it was introduced in NCCL 2.12 and it allows it's a feature that allows GPUs to talk to other GPUs on the nic, bypassing the CPU. And again, this is part of RDMA we'll talk about in just a minute.13:39How GPUs are talking to GPUs and how we can talk to memory without going through the CPU and add additional latency and handshaking. So this way we can make use of the Nick as the local Nick as much as possible to reach other NICs.13:57And other GPUs talk to other GPUs in this in two different nodes. The spines are still there, there are gonna be instances where you need to talk to different, maybe multiple sets of nodes at the same time and the spines will give you that traffic distribution.14:15So if we look at one of our validated designs, it's 256 GPUs in our back end networking going to a real optimized, network. We've got 32 port 400 gig uplinks, so eight links between leafs and spines.14:39So all of our leaves are one to one to every spot, and then we're gonna connect every GPU to every leaf. So again it looks fairly complicated, but the the idea is to give you one to one of every device.14:56Between every device. And again, even though this eliminates a lot of the issues in terms of the infrastructure, just because everything is one to one, it doesn't mean you cannot run it into congestion or issues with collapsing the fabric.15:12Without additional protocols and features enabled. So again, let's look good everything out together out of band, we've got our Cisco converged front end and not back end. We've got our storage leaves, our management leaves.15:32This.15:52Using spines connecting to our GPUs. Gpus are going to our inter GPU backend network. Everything is connected, everything is one to one. We've got an out of band management network system. And then on top of that, we've got our data center network that to get out to your network so you can actually make.15:53Use of it or your data scientists or AI engineers can make use of this back end network.16:13So if we look at Cisco as a whole, we can provide you with this silicon on the switches. We have our systems, our compute platform, our C 8805 GPU systems. We have optics that will give you the ability to connect everything together.16:22Our NXOS software and our unified management being either Nexus dashboard or hyper fabric or hyper fabric AI. All the critical components are designed by Cisco. We do work with partners like NVIDIA and Vast to provide with you with GPU support drivers and for storage.16:42But all the critical network infrastructure is done by Cisco. So let's talk a little bit more about what's going on inside the fabric.17:02The building blocks as we've discussed several times for an AI network fabric, it should be a lossless fabric. We don't want to start losing frames having them drop onto the floor. It wants to be non blocking. We don't have any congestion in the system. It should be high speed.17:19And it should be extremely low latency. So as much as possible, we want to eliminate things like handshaking, ink act in order to understand whether the traffic should be delivered or not. So how does this translate into our.17:35Protocols and best practices and features. Again, the design, we always strive for a one to one subscription or non oversubscribed network. We are providing the ability to RDMA over converged Ethernet.17:51Explicit change congestion notification. This is a way to mark traffic as being on a congested link and send back pressure to this the sender to slow down or to revert to a different link to eliminate night that congestion.18:09W Red is a weighted random early detection. Again, another way of using class of service to avoid packet drops and congestion. We also have priority flow control. This is a way again making use of classic service to mark traffic so it should be.18:28The, the last things that's ever dropped in on the network.18:47It'd be higher, higher value than another hunk of traffic that is not charring critical data. DCQN is another congestion control mechanism and AFD is adaptive flow dropping. Again, we'll talk about these in a little more detail, but these are all designed to forward the traffic as fast as possible without.18:52Dropping it. And then we will talk about load balancing techniques. If you saw the last slides, there's multiple links going everywhere. You know, how does the system determine which way to go to which spine should it select?19:12In order to forward traffic. Some of the transport protocols we have in our ACI cluster or AI cluster is traditional TCP for ethernet traffic and rocky V two, which is RDMA over converged Ethernet.19:25I always think of them as just buckets, the buckets contain the payload and the outside of the bucket has got the sender and receiver information. It's just a way of compart putting everything to a container.19:42And the container has an external address and an external sender and a sent a receiver. And then again inside the system, RDMA is used to reduce interference from the CPU.19:58And allow the GPUs to talk to each other or to talk to the memory as required and reducing that overhead induced by CPU requests. CPU will offload those requests to the RDMA system and RDMA will handle traffic between GPU nodes and storage.20:15Kind of a simplified example. I've got two servers here on the PCI bus in each system. I got my GPUs, my memory, my Ethernet PCI, and my RDMA negs.20:36Again, the idea behind this is to allow the CPUs to offload information, just make simple requests to allow the GPUs to talk to each other directly. This reduces a lot of overhead going back and forth to the CPUs, handshaking.20:50Mirroring traffic and mirroring memory copying memory back and forth. This gives you a very low latency system with high throughput and minimal CPU overhead. It's also routable traffic, so that means we can make use of this GPU to GPU or RDMA communication over.21:08A larger network as opposed to be closely coupled within the same chassis.21:28This connection between GPU to GPU or RDMA over converged Ethernet is UDP, it's assuming there's NO transmission, so we have to guarantee all packets will arrive in time and a lot of the features that we built in the fabric give you that reliability.21:34Let's talk about some of these lossless features. This is probably the, the overriding feature that we use. It's our explicit congestion notification, and when traffic is detected on a link or set of links in our end to end conversation.21:55To start experiencing congestion, we'll talk about how we look at that. We can actually mark that traffic and then tell the sender to slow down or basically apply back pressure to the sender to reduce the sp the amount of traffic that's coming from that sender over that link or to.22:13And other, other ways we can use that same congestion notification to move the traffic to a different link. The idea is to not collapse the fabric or not saturate the fabric so that we can NO longer send traffic over that link. Again, we will mark the traffic.22:33And let the sender know that there's an issue and to start looking into it before it gets to be a problem. We have priority flow control. Some of you are are very familiar with this. This is a class of service map marking so we can tell, we can classify.22:50Traffic as to what should be.23:09A high priority and in our case high priority and lossless or do not drop traffic. This is similar if you used a fiber channel over ethernet. We have a class of service which we call lossess or NO drop traffic, so the system will do everything it can.23:11Before it starts to drop that class of service. It's on a per priority basis, so you can have different classifications. We have a classification that we use for AI ML workloads. We also can do, we can.23:30Operate over lossless priority and dependent on other other priorities, so this will take precedence over another high priority class of traffic. Other traffic will be will be serviced as possible or redirected to different lengths.23:46Again, this is our our way of making Ethernet which is was actually initially designed to be slightly lossless to be a lossless Ethernet transport. So how does a priority flow control work? I kind of think of a.24:05A bucket that contains all your traffic and that's going to your egress port. We have some settings on this bucket. As traffic starts to flow into the bucket, as long as it's below the Exxon indicator, it will just drain out as, as necessary. Once we start getting into the.24:24Works on.24:44Limit we start to try to apply, we try to tell the sender up or send some back pressure to the sender to start slowing down frames. Once they exceed the X off, right? We're still making use of the headroom within the bucket.24:47To allow us to keep on accepting traffic, but again, we're trying to send notification to the sender or to have the sender try a different link in order to start moving it to a different egress port. But again, once, as traffic goes below that point, we'll turn off those toggles and.25:04Again just allowed traffic to slowly drain as required from the system.25:24And again, once it's below the exxon, it will just stop sending those priority flow control frames back to the sender. There are some considerations, priority flow control. It is possible, you can reach a state where the system can get saturated.25:31Your buffer can NO longer accept packets. Your sender is still trying to send traffic, and you can run into a congestion collapse of that, that link. At that point, we do have some washlock timers if we start noticing that the.25:52Buffers are full and NO NO traffic is getting through will reset the the sender and have it start over. This will allow the bucket to train and start send accepting new traffic. Again, we try not to ever reach this, but it is possible. We just want to make sure there's a way.26:07To recover the system without intervention. Rock EV two RDMA over converge Ethernet again is another part of our end to end lossless behavior. It, it's something that.26:27I think most people in compute world world are familiar with is remote direct met memory access. Again, it's a way to get CPUs to access memory without having as much involvement of the CPU.26:44And then remote allows it to be not local to the, the chassis and over converged Ethernet allows us to route that traffic over our ethernet cabling to different systems within the same data center or different data center. We can make use of layer two and layer three networks.27:01Again, the idea is to allow this type of communication to occur over our Ethernet network. We do make use of things like the explicit congestion notification, the wred, which is another class of service.27:19Method of marking traffic and what we call adaptive flow control. Again, familiar with the FCOE, some of these concepts maybe very familiar, and, so another technique is called weighted random early detection.27:36Or W red, and this works in conjunction with explicit conjection notification.27:55Again the same thing as a different set of markers where in the same buffer space. We have a minimum and maximum threshold settings. Once our bucket starts to exceed that W red minimum, we'll start marking the traffic and try to send, again, send back pressure to the sender.28:01To start modifying its behavior, either slowing down traffic.28:21Or select a different link. Once we've hit the W red max, again, the threshold is hit. We're gonna do everything to start marking everything with ECN, but nothing's gonna actually change. It's just another way to identify that that is a problem and we need to start looking at methods to bite.28:26Pass that link or bypass the bucket.28:46AFD or approximate fair drop, again is another technique that we have within the system. Not all traffic in Ethernet is is constant or might call democratic. It it can result in what we call large or elephant elephant flows or elephant frame.28:53And mice flows. As you can imagine an elephant in your egress can cause congestion and the idea is to start marking, you don't want the elephants to get in front of all the mice.29:11And then eventually again cause congestion or blockage in the system. AFD will start marking elephants and if necessary redirect them back into the queue to egress the buffer.29:27The idea is to make sure that the elephants do not congest the egress point of the switch and stop all traffic. So again, the whole point is to identify or allow the system to identify elephant on my flows.29:43And the, the idea is to, you kind of punish the elephants so they cannot, you know, form a congestion on the egress link. And they do this by using ECN marking. So how does it all work together? So it's not.30:01Not just one technique that's a combination of all of them. We have party flow control.30:21Weighted distribution and exchange explicit change notific congestion notification. There are different thresholds set for W red, and for PFC and then we also have our buffer to work with. The whole idea is to use use make make use of all these techniques.30:26To ensure that the buffer does not get full, that we apply sufficient back pressure or notification to the sender to either reduce or make use of a different link in order to send traffic out of the system.30:44We also include, this is traditional in our fabrics as load balancing techniques. If you look at our system, again, this is our, on the bottom, you'll see our nodes and they're kind of replicated, so we've got our four nodes and all the several GPUs.31:03You is connected to every leaf, in our case, every leaf is connected to every spine. So technically we are one to one non oversubscribed. But the reality is you may have multiple GPUs from multiple nodes trying to talk to a s a single node on a different leaf.31:23And our equal cost multi pathing algorithms, how do we select which link to spine to use? Because any of these spines can get to leaf four.31:43Oh, but the way our hashing algorithm works to use for ECMP selection is we make use of what's called the source destination, IP, source port and destination port, and UDP.31:58And in many cases, this can cause the hashing algorithm to select the same link because since there's not enough what they call entropy or unevenness in the the data to do that hashing algorithm.32:13So technically, even though we got multiple links going to every spine, we could push the traffic to other links to get to leaf four. Our hashing algorithm will may result that we're gonna overload this leaf to leaf one to spine one connection.32:29Again, this is something that we have to watch out for in our systems. We do have some techniques to help you eliminate these. And again, the whole idea is we wanna make use of these unused links to get to the same leaf.32:46One of the things we do is our dynamic load balancing is we'll actually start detecting congestion on these lead to spine connections. So again, in the case of our hashing algorithm, our standard ECMP hashing, we may end up always going to spine one.33:05But if we start seeing congestion on spine one, we can actually tell this, the system that spine one is starting to get congestion, even though the hashing algorithm selects spine one, start sending up traffic on spine two or say on spine three. This way we can maximize link utilization.33:21And still make it very automatic for you in order to determine which links are used being used in the system. So it will automatically select the best link with the lowest lowest utilization in order to move traffic off your leaf. Additionally, as we talked discussed.33:37Earlier, that's there's the GPU to GPU communication.33:58So we can keep the traffic on the light on the leaf as opposed to going to a spine in order to go to a different leaf. This does require the use of nvidia's spectrum X techniques and you can see the increase in traffic between just standard dynamic.34:02DLP and using nvidia's GPU to GPU communication.34:22These do require specific nvidia cards, and these are the ones that were at present at the time of this slide show, and you do have to have, there are some features you have to enable within the NVIDIA GPUs. And they will or there are some licensing requirements for this GPU to GPU community.34:29Coming up next, we'll do a demo on the load balancing selection techniques and some of the tools you can use to see that load balancing. I'm gonna pass it over to Hari. Thanks Dave.34:51Let me share my screen to my presenter, share my screen, screen two. Okay. Okay, so, my name is Jorge. I'm going to play a video for this part of the demo.35:06So let me start. Let me start showing you the topology. Sorry, I'm going to select, let me do something 1st. Okay.35:23We're going to share again, screen two. That's it. Okay, so let me start showing you the topology we are using. What you see here is the classic spinely fabric. We have three spine switches, 201-202-2203, and six.35:39Leaf switches L two oh one through L two oh six. Connected to each leaf from L two oh one to L two oh five, we have one host. Host one through host five.35:58These are our traffic generator simulating GPU notes. So the key element here is the leaf two oh six on the far right. Instead of a host, leaf two oh six has storage connectors as you can see at the bottom right, specifically on interface Ethernet 01:11, and that is the focal point for the.36:11Tire them.36:32The arrows show all five hosts sending traffic simultaneously toward the single storage destination through Ethernet 01:11 and the green-line as you can see represent 400 gb of links to the spines and the shown dashed lines represent 100 gb connection. So all that traffic.36:33On a single port on lift O six. That's is where congestion is going to happen and that's is exactly what we want to show you. So I'm going to play a video here.36:52After review this topology, this is the next dashboard, fabric controller view you can see we are working with is called Rocky version two. That name is of course intentional.37:07So the fabric is showing green and healthy nine switches in sync, NO alarms. Everything looks fine from the top, but that is exactly the problem we want to expose. Things can look healthy at the fabric level when packets are silently being dropped at.37:21Interface level.37:42So what we drill, when we drill into the topology and we can see the this physical layout, we can see the full physical layout, the the three spines at the top and the 6th leaves at the bottom, all green, not healthy, this much exactly the diagram we just saw. So we will now.37:43Navigate and now navigating to leaf two oh six, I'm going to click here on the fabric.38:03Going to click on switches, I'm going to select two oh six. The idea is before generating traffic, let's confirm that there is NO QS configured. So here we are using next to dashboard building switch show commands feature, so you can run CLI comma.38:08Against any switch directly from the from the GUI without needing any separate SSH sessions, so as you can see, I rule ipqos.38:29On leaf two oh six and as you can see, the outputs confirms there is NO IPQS configuration at this point. So NO policy maps and NO class maps. This is just plain default ethernet behavior.38:40Okay, now we are going to start, the traffic. So let me show you the the on on this screen on the left side of the screen, you can see iperf running on all five hosts simultaneously. Iperf, maybe you don't know what is ipf.38:58Iperf is an open source networks benchmark tool.39:16That generates high throughput UDP or TCP traffic between. So in this case, we are using it to simulate the heavy traffic pattern of GPU collective operations, so as you can see, you can see values, very high values.39:20As a 9000 mbps 1000 mbps. That's it. So this is all of us are pushing traffic as hard as they can, just like GPUs will do during training operation.39:38Here, look at the right side. This is leaf two oh six. There are three monitoring windows. I'm going to start for the bottom right window, that is the critical one, show, show interface Ethernet 01:11. It shows more than 500000 in these card packets and climbing, so.39:54This is how many packets have been silently dropped on the storage interface. So the switch buffers are overflowing at this point and there is nothing to stop it. And.40:14As you can see the top right window, let me let me scroll up for a second, the top right window is going to show you interface Ethernet 01:11 every second by filtering for PFC. So the columns shows.40:25RXPPPP and TXPPPP, those are the PFC post frame counters. So right now you can see packets on RXPPPPP, but on TXPPPP, there is NO any frames, so the switch is receiving some post signals but NO generating any back. So PFC is not configurable.40:43So that is not coordinate flow between flow control happening. And that is the top right window. So what about the middle right window? You can see it shows quote interface ethernet one every second and look at those.41:03ECM packets at ECM bytes. All zeros, can you see at the bottom right? All zeros. So the idea here is not a single ECM mark is being generated at this point. So the network has NO way to signal congestion to the sender. So the traffic keeps coming, the buffer.41:21Keep feeling and the and the packets keep getting dropped as as we already showed. So this is what standard internet does on the on their congestion. They drop 1st and ask question later.41:41So NO NO, now we are going to fix it. So we are going to edit a fabric and now inside the fabric dialogue in Nexus dashboard we navigate to the quote policy section and here you can see.41:54And a dropdown for nine K cloud scale platform policy and we have selected AI cluster QS templates. What, what is this? This is a a preview cisco validate template specifically designed for AI clusters.42:12So there is also a separate policy for our series platforms and another one for nine K platforms, but notice also that we are enabling enabled default policies check and and TCAM allocation is enabled, so that is required for some of the.42:29A QoS ticket based features to work currently at this point. Okay, now when we are seeing this, we already select the template, now we are going to save. So we save.42:47At this point, and the nexus dashboard is going to calculate, let me show you recalculate and deploy. So the next the the nexus dashboard calculates what needs to change on every switch.43:02So let's look at what is going to push. So this is the pending config preview for leap two oh six. This is one of the most valuable feature of next dashboard. Before deploying anything, it shows you exactly which CLI commands will be pushed. Line by line.43:17As you can see. So let me walk you through the key elements for this configuration.43:37At the top you can see the class map definition matching the SCCP values of 24 and 48. This is how traffic is classified into different QS group, the SCCP 24 maps to QS group three.43:44Which is the query we are going to protect with PFC. Now scrolling down, we have the policy map. As you can see the policy map, the highlight section on the query is critical. It gets.44:0160 % of the bandwidth reserved and most importantly it has a random detect line with minimum threshold, 150 kb. Maximum threshold, 3000 kb, drop probability seven by zero and ECN as you can see the last word ECN. So the last keyword is.44:22Everything Instead of dropping packets when the query did exceeds 100 kb, the switch will mark the packets with the ECN congestion experience bits. So signaling to the receiver and then back to the center to slow down, not drops, just signals.44:42As scrolling further down in the pending config, we can see the PFC configuration being pushed at the interface level. So the key lines are post PFC cost three PFC is enabled on cost three, which is all storage. And also.45:02The MTU set to 9216 humor frames required for rocky version two to work properly, as you can see. Now after that we are going to select the the priority flow control. So as you can see those three lines.45:20Remember we are reviewing the configuration that will be pushed on the switches. So far we haven't pushed any configuration to any switch. So priority flow control, mode on enables PFC on the interface. Priority flow control Watchdoc interval on, this is important. The the watchdoc.45:39A PFC deadlock.45:59Where pouse frames never gets released and traffic gets stuck indefinitely. And finally service policy type QoS, input QoS classification policy applies the classic classification and english. So please remember that this configuration that we already reviewed is being.46:01Push to every interface on every switch in the fabric levels leaps and spines alike.46:20Both of them. So let me show you the deploy configuration screen for fabric. I'm going to close here, and as you can see, you can see all nine, let me show you.46:29Just you can see all nine switches listed here, leaf two oh six leaf leaf two oh five, Leaf two oh one, spine two oh one two through spine two oh three. Every single one shows out of sync, as you can see out of sync status description, and that means the configuration.46:48Pending.47:08So you can see we have 86 lines, 128 lines of pending configuration and the spines have 30 lines each. So the progress bar are already feeling in green. Nexus dashboard is is ready to deploy everything simultaneously with a single click on the deploy button, as you can see at the botto.47:09Right deploy all button blue one. So this is the operational simplicity we talked about earlier. So one policy, one click, all switch in sync. So I'm going to.47:29Here and that is going to deploy the configuration and I'm going to see after the deployment, I'm going to see what changed. So on the left side, you can see the iperf windows. We already talked about the iperf, I'm going to do zoom here in order to.47:41To see better. So most hosts are now, as you can see what about the traffic on the host? So most hosts are now showing almost 0 kb. As you can see on your left, I can see 0 kb here, just let past some seconds here on your left, 0 kb.48:01Most of them are now tending to 0 kb or or very low values. That is PFC doing his job. As you can see, all of them are zero now. That is the PFC doing his job. So the switch is sending pouse frames back to the senders temporarily halting.48:21The traffic and let the buffer strains on the sender are compliant.48:41And they stop, so NO data is lost. On the right side of the picture is completely different from before. The top right PFC windows now shows internet 01:11 with RXPPPP and the Uplink interfaces showing TXPPP counts. So that means the PFC post fronts are being prompt.48:45Through the fabric and the flow control signal is is traveling back to the sources. In the middle by window, it now shows live EC encounters. So the switch is actively marking packets and signaling congestions. The sender now there is a congestion and now are adjust.49:06Hosting the rate and at the bottom right.49:25The discard counter remember that was very high, still shows 8080 hundred thousand output discards, but wait, the number is not growing anymore. That is the accumulate count from before we deploy QoS. So from this point forward the drops have been stopped.49:28And the losslets guarantees in effect. So this is the contrast we want to show you. The standard ethernet, the standard Ethernet sil silently drops hundreds of thousands of packets as we already saw. With the AIQS template deployed through Nexus dashboard, the same fabric.49:48Under the same traffic load operates without a single additional drop using ECM to signal and PFC to pause greater than this carton. So the key.50:09The key takeaway is not just that this works, it's how easy it was to get here. We went from zero QoS, hundreds of thousands of drops, and a fabric that had NO idea it was causing problems.50:21To a fully lossless congestion of our infrastructure, deploy across nine switches simultaneously with a pre validate Cisco approved template and a single deployed action in Nexus dashboard. That is the operational simplicity that AI workloads demands and that is what the solution delivers.50:36Now, I'm going to show you this is the Nexus dashboard here. So now that the QoS is deployed, let's verify it is actually running on the switch. I'm not I'm not playing any video here, this is the nexus dashboard. So we will do the exactly same operation that we, we did.50:57Without leaving the next dashboard from from with the fabric view I can select any switch and run show commands directly from the GUI. In this case we are going to to run them the same command, showroom IPQS against the switch.51:17So I'm going to click here on manage, I'm going to click here in fabric. This is my fabric, this is London London for London. London AI one, I'm going to click here, I'm going to click here, inventory, let me click here in inventory.51:30I'm going to select this switch. I'm going to click here, this is all the details about this switch. I'm going to click here in actions, actions, maintenance, and I'm going to click here in show commands.51:49And show commands are going to select an option show. I'm going to put here my command that is run IP, IPQS and I'm going to select this button at your arrive execute. So I'm going to execute that command on the switch.52:06So what come come what comes back is the fuel, the full QoS configuration now active on the device. So you can see the policy maps, the class maps, the DSCP DSCP by this classification.52:27That decides which query handles which traffic. This is where ECN notification is defined. This is where W red threshold are said, this is where PFC is enabled globally and applied to all interfaces. The watchdoc timer is also confirmed active per interface.52:40So the key point here is this is not just this switch in particular. This is the default for every switch in the fabric type. So one template consistently deployed everywhere. No manual per switch configuration, NO risk of one switch being misconfigured while others are correct.52:56Now let's go level deeper inside next dashboard. We're going to click here, I'm going to close this screen. I'm going to click here in overview. Okay, but in next when I click the specific switch, ok, I'm going to click here in this interface.53:15I'm going to get a full summary view, so the the all interface is connected. Let me click here in the statistics and this is the this is a dedicated statistic session for this interface showing real time data.53:34It refreshed every 10 s visually, but the data is being collected in real time. So what do you see in the traffic graph? Highly bursty traffic. So spikes up to 90, I don't know 90 something percent utilization followed by periods of lower activity.53:51This is exactly the pattern AI workloads produce. Collective operation like training a model is going to generate synchronized boards across all nodes simultaneously.54:11It's going to saturate the links for a few seconds and then going quiet. So this is a normal behavior, but this is the reason you need congestion management. Without it, these 90 % spikes become drop events. So below this graph.54:23And then below the authorization graph, you have all the metrics, as you can see here on your left, that matter for a lossless fabric, W red, PFC, ECN, and this e.g. I can click here and view queries.54:39I can see, this is we can see the PFC counters, show how many post frames, all that, ok? And finally I'm going to show you anomaly detection. As you can see here, I'm going to click home again.54:56Manage fabrics, going to click here, and this is the anomalies here and what about anomalies? Nexus dashboard does not does not show you current state is going to learn the normal behavior of your infrastructure over time.55:15And automatically is going to generate anomalies when something debase from the baseline. So you don't need to set a threshold manually.55:35So the the system is going to observe your traffic patterns, build a behavioral baseline and flags anything unusual. So let's go back in time and it's going to active now here and click e.g. last week. I'm going to apply here.55:44Let's go back in time and look at last week's anomalies, so you can see a significant number of anomalies and most of them are connectivity related. And something. I'm going to click here, e.g., I'm going to click here, e.g..56:01And, when we click into one of these anomalies, we get the full context, which two IP address were involved, what the traffic look at the moment, which devices were communicating. This allows you to go from something happening to here is exactly what happened and when happened, who was involved.56:15Without having to correlate logs. So, think about what this means operationally. In a traditional model, you will need to notice a problem. SSH into multiple switches.56:35Run show commands manually correlate timestamps, and try to reconstruct what happened. By the time you do all that, that event maybe long gone and the GPU training job maybe have already failed or degraded. With this approach consistent QoS deployed via templates.56:47Time for interface telemetry and automatic anomaly detection with historical context, you have proactive visibility instead of reactive. You know there was a congestion even before the application teams called you, you know which interfaces, which flows and which endpoints were involved.57:07And you have the data to prove that the lossless mechanisms did their job and preventing actual data loss. This is the end for this demo. Back to you Dave. Thanks a lot. Great demo and great reason why we should always be doing more demos and less talk.57:26So I'm just gonna do a hopefully a few sets of slides on Nexus dashboard, going to hyper fabric hyper fabric AI, and then we'll go back into the demos. You've already seen some of the capabilities of Nexus Dashboard controller. Again, it works with your existing Cisco networking environment, ACI or VXLAN. We incorporate into our.57:51AI ML workload.58:11Configurations. Who, those who you aren't familiar with it, nexus dashboard gives you that complete overview of the system. We have both management and configuration control and really really important in my mind is the ability to go back in time and see what happened maybe three days ago or four days.58:13Years ago when whether it was an instant.58:33We can actually pull that data up within the controller. It's being pulled from the systems constantly and then stored in the data lake on site Nexus dashboard. There's other things in terms of image management. You can do your upgrades from next dashboard against multiple switches. We can work with multiple multi site architectures.58:35And again, we can integrate those AI ML workloads in our our using our best practice QoS policies. Again, part of what you saw in the demos, we, we can actually look at some of these features we talked about earlier in terms of this congestion notification using AFT.58:55The or priority flow control or.59:15The other techniques we used for load balancing in order to determine where you maybe having issues and how you can actually correct those with some of our best practice templates. We are going to be releasing Nexus dashboard 4.2 very shortly.59:19One of the things we're gonna add into 4.2 is the ability to add in what's called slurm or the simple Linux utility for resource management. It's an open source, very scalable cluster management and job scheduling system. It's one of the industry standard tools for managing high performance computing and AI ML workloads.59:35Yeah, some of the, what we're providing with the Cisco solution is helping you create AI solutions to let your data scientists and your AI engineers work faster without having to worry about the infrastructure. We are introducing A hundred gig platform support.59:58You can leverage your existing ACI or VXLAN fabrics for the front end of your system and we are integrating in NVIDIA GPUs and vast storage to handle large data sets.01:00:18Now a brief introduction on what we call Cisco hyper Fabric and hyper Fabric AI. This has been previewed for early last late last year and is available now. We started with Cisco hyper fabric. It's a cloud managed networking for Cisco 6000 switches. So these are not the classics.01:00:346000 switches, this is our new line of data center 6000 switches. We've also introduced hyper fabric AI, which allows you to do use that same cloud management system to integrate in NVIDIA GPU and storage. Hyper fabric and hyper fabric AI are.01:00:54What we would consider is a fabric as a service. We got we Cisco will be hosting the management system and you have the ability to place your physical devices either on premises or maybe in a co working location. The advantage of this is that we make changes or add features and functions.01:01:13Functionalities to the hyper fabric infrastructure management tools, they are instantly available to everyone. Everyone's working from the same set of tools. You don't have to worry about which version of tool you're using. You still have control over the firmware versions on your exact specific switches, and we will, again, in our advisory.01:01:33Lets you know what, what versions of switch firmware you should be moving to. It's a way of doing a fabric of the service. Again, we will have the management tools, monitor tools inside of our cloud, and you can have your hardware on your own premises.01:01:54This is also now been extended to NVIDIA integration and vast storage, so hyper fabric AI. So not only can you manage Cisco Nexus 6000 switches in an AI ML infrastructure, you can also do that by including NVIDIA and vast storage.01:02:14So kind of the basic components, there's the cloud management system or fabric as a service. This is if you're familiar with something like inter site for compute, we, that's a very similar concept. The, the management for AI UCS AI IAM compute is in the cloud and your.01:02:34Hardware is actually on site. There's a connection between the two and management is configured. Monitoring is configured to Intersight.01:02:54But your hardware's on site. We've taken that same concept with hyper fabric and hyper fabric AI. We've added a lot more capabilities to hyper fabric. In hyperfabric you can actually do your design, your cabling layout, your SFP selection.01:03:02You can actually generate a bill of materials and then as you're awaiting, you can either even redesign or make changes. Once the equipment is on site, you would connect that equipment to that, that design, that blueprint, and you have a working network. We require you to you know cable it and power it up and.01:03:22And have one switch that's connected to the internet in order to complete the configuration, but everything can be done in hyper fabric without having anything on site. This is all hosted by Cisco. It's a browser access. We have API access and even mobile access.01:03:42It does make use of the Cisco 6000 switches. These are the sonic base switches. We do make use and configure this into high performance fabrics, again some simple topologies, either mesh or spine leaf. Again, the idea is to give you one to one connectivity between spines and leaves and including high.01:03:59Your fabric with.01:04:19NVD GPUs and vast and get and give you that real type configuration. In addition to the cloud controller, the Cisco switches and fabric design, we also have what we call helping hands or give you the clues or handsets to what you should be doing in order to get this to work.01:04:20Some sample topologies, you know, we can go anywhere from a single node or to a extreme large spine leaf. We top out at five nodes in a full mesh fabric. Again, everything is any to any, and as we start going to spine leaf, it's our our classic spine leaf architecture. Again, the idea is to give you one to one.01:04:43And non oversubscribed links between spines and leafs. Again, these these will only work with our Cisco 6000 switches using the sonic operating system.01:05:02Some of our current supported hardware right as the nexus, sorry, the Cisco hyper fabric six 6000 series. Again, these are 32 and 60 port switches. They make use of quad SFP dual double density and up to 400 gigs and 800 gig will be coming out soon. Again, these.01:05:16They're running Sonic, they are not running NXOS and they work with Cisco hyper fabric.01:05:35Talk briefly about kind of the the life cycle. Again, there's a cloud managed controller. It's very scalable. You can access from anywhere in the world, either using the GUI, using a mobile device or it could go headless with an API access. It's using our 6000 servers and you have.01:05:45Full visibility and design inside this tool.01:06:05So you can start with the fabric designer and select what type of fabric you want, what type of switches you want, what type of interfaces you want, what type of, cabling, do all that design within hyper fabric, even without having hardware inside. It can even generate a bill of materials that you can hand off to Cisco Cisco for what you want to.01:06:11Have and where to deliver it. Then once it's been.01:06:31Connected to your physical fabric, you can start monitoring, adding in the layer two or the external access layer two or layer three, everything can be done within hyper fabric from start to finish some hyper fabric terminology you'll run into if you.01:06:35Start playing with it, there's what we call the blueprint. So this is a detailed network fabric design, whether it's mesh, spine leaf, how many spines and how many leaves you select what type of models of switches, what types of ports, what types of connectivity, what type of interfaces you want.01:06:55That all goes in blueprint. We do have a way to organize who can work on the system, who's the administrator user, who's a viewer only user, and who can make changes to the system. As equipment comes on site, as it's spun up, you claim those into that blueprint.01:07:13And once they're claimed, you can bind the actual hardware into the system and then the configuration will flow down to those switches and push out the design you you created. So you don't have to worry about setting up virtual port channels or port channel links or multi links between systems.01:07:32It will take that design and do that work for you. All the underlay is done for you based on that blueprint. Again, kind of think about it the way it is, is a cloud based service. It is alive now. You should be able to log in if you have a CCO account.01:07:51It's hyper fabric.cisco.com. As you log in, you can, you can create organizations and fabrics. Again, you can play with the system and just see what happens. You can actually design a complete AI system without having any hardware, and if I say design everything, you can design everything out to the.01:08:09Interface type and SFP type.01:08:29Again, just a real simple interface, I will show up more in the demo, but you can actually create, as you log in with your CCO account, you can identify who you are and you can identify what fabric you want to belong to or organizations and additional people that you may want to have on that system.01:08:34After you've done that, one of the 1st things you would do is create a blueprint. Again, just think of this as your design for your fabric, you know, what switches you want, what topology you want, what type of interfaces and optics you want between the systems. Part of this hyper fabric and hyper fabric AI is.01:08:53We can do cabling plans. There's the ability to do auto cabling, which is.01:09:12Pretty straightforward. It's gonna connect between your leafs and spines. It will allow you to do. The only thing you really need to select is a type of optics or qsfps you want between speed the spines and leafs. Beyond that, it will start selecting it for you. You can always, always manually make changes.01:09:17And again, this will be included here in your bill of materials and just some simple port types. They should be pretty straightforward. You know, fabric ports or ports between spines and leaves port channels, again, you don't have to really worry about it too much. It's gonna create it for you if you have multiple links between spines and leaves.01:09:36And then host for endpoint connectivity and then routed for external connectivity to the AIML fabric and of course an unused port. All ports are generally admin up because you know it's expect you have to tell it what's gonna be be connected to what.01:09:56And so it has to be a connection that's admin up in order to make that connection work. It's actually similar to how ACI works is all the ports are admin up. But even though they're all admin up, it doesn't really mean they're gonna forward traffic. That's, that's more of a configuration than VNI.01:10:12Consideration later on. So enough of that and we'll go into our next demo. Jorge gonna demo hyper fabric and show you some of the things we talked about. Thanks Dave, I'm going to change the role, I'm going to share my screen again.01:10:29Okay, so in order to ingress to you the URL you need is hyperfablic.cisco.com. You will need a Cisco cco ID to login. That that is your standard Cisco account. So I'm going to.01:10:49Right here, my CCO. So the reason I want to show you this from the very beginning from the login process right there than jumping directly into the interface is you can see exactly what the experience looks like from zero. So I'm going to.01:11:09Quickly here continue.01:11:29Okay, now here, as you can see this is I'm in the fabric. I'm going to let me do a quick tour of the left navigation. This is just a test fabric by default. So, but this is the left the left navigation. So the 1st is here click here. This is the.01:11:30The 1st thing you are going to encounter after logging is the concept on an organization. An organization is simply a logical container is where you will create your fabrics where you will define your network blueprints and manage your users.01:11:50So think of it as your tenant boundary withinfabric. So when you create your 1st organization, the system automatically name it using your CCOID followed by the word scratch, but you can rename it at the creation time. So you are automatically assigned as administrator.01:12:08Of that organization. So as you can see, this is my organization and the words crash at the end. You can create as many organizations as you need and invite other people to, to them. You just specify the CCO email address and assign them a role. Okay, let's continue here. I'm going to click here. We are going.01:12:28To Return to fabric. This is inventory.01:12:48And the inventory section is where you claim devices will appear once you start adding switches to a fabric. Right now it's empty set of claiming devices set of devices bound to a fabric because we have not created anything yet. But once you have devices on board, this is where you.01:12:51To track their state and fabric assignments. Let's continue here on the left menu, subscriptions, so the subscriptions section is related to licensing. So when you start creating fabrics and deploying networks, this view shows you what license tier you have and how.01:13:11How much.01:13:31Of it you are consuming. And continue here administration. This is the most important section for the day to day operation. I'm going to click here, e.g., user management as you can see this is my name as administrator adding a user is.01:13:32It's as simple, you just need to click here add user.01:13:52And you just enter the the person's cco ID, I mean email address and assign them a role. So not complex LDAP integration required of the box, the system is going to handle the identity through Cisco existing authentication infrastructure.01:13:56Okay, I'm going to cancel here. So let's let's go to the, to the main policy here fabric. So this is the fabric. So let's go into creating a new fabric. So this is the the button and you can see add new fabric. You have three options. The 1st one is the new fabric. You build.01:14:16It you build the fabric manually from scratch defining every parameter yourself.01:14:36This is we are going to use this in a bit later because we are going to to build a small topology I will show you in a moment. Now import fabric is the last one is for when you have an existing configuration file and you want to bring in an new front templates.01:14:42This is the interesting one because the template option is going to give you some predefined templates that covers standard fabric deployments, and you can see we have a switch fabric and AI, AI cluster.01:15:02Which is the full stack option. That is so let's look at the switch fabric 1st and Cisco has, as I already mentioned, pre built several validate topologies to cover the most common deployment scenarios.01:15:19E.g., as you can see here on the on your left, two spines for leaves, that is the 1st one, two spines at the minimum viable spin leave fabric. So this is ideal for smaller environments or proof of concept deployments. As you can see here on your right, max spine leaf version one, this.01:15:38Four spines, 32 leaves, this is a full scale production fabric.01:15:58Oh, scrolling down, as you can see is max spine leaf version two. It scales up to 16 spines and 32 leaves for very large environments. And this one is three tier spine leaf. This is a very large topology for more complex data centers. There another one also here as you.01:16:02Can see even bigger. For all of them and use the HF 600 series switches. So these are the Cisco Nexus switches purpose built for. So the hardware is already pre select in the template. You are not starting from scratch.01:16:22On hardware choices. So what about the the another so let me scrolling up here. I'm going to click here now in AI cluster. That is the main topic here. So this is where hyper fabric AI lives the full stack options. So you have four sizes, as you can see, this is the small one.01:16:41And scrolling down as you can see medium, large and extra large. E.g., the talking about this the small one, as you can see it has two backend leaves for GPU to GPU traffic.01:17:01One management leaves for out of band management, two storage leaves for storage traffic, four GPU servers that those are the compute nodes and eleven storage servers. So, the key value here is that all the cabling and switch to switch interconnects.01:17:19Are already defined in the templates, so you are not designed the topology, you are just filling in the quantity. So let's select this small AI cluster as you can see let me wait for a second.01:17:39I'm going to reload here, I'm having a problem here.01:17:59Are you able to hear me? Yeah, I hear you fine. Okay, let's let's let's do, let's start again from scratch. Let me, something is happening here with the fabric. Oh, oh.01:18:14Oh, let me check. I'm going to start again.01:18:31Something are you able to access cisco.com? Okay ok ok now it's let's again.01:18:53Let's log in again.01:19:11We're having some access accessing problems to cisco.com. Oh my god.01:19:27Let's try again ok if that doesn't work I can also share my screen if looks like I do have access so I can.01:19:54Let you borrow my screen. Oh, ok just just give me a few seconds. I don't know what is going on here because I know I am able to access. So Corey, just let me know I can drive it, let me know where we click and and I'll just follow that.01:20:06I, I was trying I was trying to to explain the AI fabric so please click on fabric on your left. Okay. Select the, the AI fabric.01:20:22Oh NO, yeah. Let's add new fabric, add new fabric. Okay, let's add new fabric yeah ok NO new fabric from the plates.01:20:43No, NO NO NO, new, new fabric and the last one hold on let me go ahead yeah that's one. The click AAA cluster.01:21:00Okay, let's click on select small AI cluster template. Select small AI cluster. Okay, it's already selected there.01:21:18Okay. Yeah, I select a small AI cluster play button. Oh, yeah. Yeah, that's it ok so the idea is to to let's look the small AI clusters of the description. I already mentioned two backends.01:21:35Leaves so you can see two backend leave, one management leave, two storage leave, four GPU servers and that's it. Now let's click please save fabric blueprints, ok, please room cabling.01:21:54Okay, so once you click save fabric blueprint, sobric is going to process and save all the fabric tiers. So after saving, will you offer this option to room cabling, so the room cabling function automatically generates the complete cable plan for you.01:22:18All switch to switch connections.01:22:37And for air cluster, all GPU to switch connection as well, every port mapping, every cable defined automatically based on the blueprints. So this going to eliminate one of the most error prompt parts of building a data center fabric that is a manually planning and documenting coupled rooms.01:22:40So you can see in the topology, a back end network two switches, you can even you can see twelve links, 48 port to server.01:23:00So these numbers on the links are not the corrective. They represent the actual cable counts between groups generated by the auto cable function. And the legend confirms this includes a rail group connection with the DGPU cable pattern we discussed earliers. So just please click on.01:23:11The 4848 number. As you can see these those are the, the, the, the cable in the ports that I already what already already calculated ok the two more the ok now let's go go scroll up, please.01:23:31Up. Should I close the cabling? Yes yes, please close the close the cabling, click and review configuration, review configuration. Okay, that's the one of the main concepts that we are going to to to explain here because we have two.01:23:53Thoughts.01:24:13And there is two more mode models so there is a ruining mode that is currently deployed and active on the switches and there is an edit mode when you can make changes in a candidate configuration. Every change you make goes into a pending state 1st. So before anything get pushed to the actual devices.01:24:14You go to review configuration and we show you a human readable list of every chain. That is the, the, the list that you are seeing right now.01:24:34In this case, you can see a series of updates management port entries one per interface for every service management port being configured. You also have a Json diff. Please click on Jason diffs tab. So this show you the role configuration differentiate in a side by side view.01:24:45Exactly like a code review diff, so you can inspect every field that will change before committing. So when you are ready to push, you just need to click comment and push. Please push comment and push.01:25:04Yes, it's it's similar to using GitHub in when you are programming. So please write auto cabling here there, auto cabling and a push configuration. Hey Jorge, I just want to give you a quick time check. Yeah, I know. I'm going to finish in a moment.01:25:17So, and that's it. Every configuration push has a mandatory of the trail comment attached to, so at this point the configuration was pushed to the devices. And with that, I'm going to finish my demo here, back to you then.01:25:37So hopefully we get an idea of what's going on in the infrastructure. A lot of it is hidden from you. You'll see it if in some of the ACI and VXLAN configurations. Again, there are additional features within Nexus dashboard to help you control that.01:25:53Lossess network and a brief introduction of how hyper fabric and hyper fabric AI works with that system. So again hopefully this will give you some ideas of what's going on and what's available to you.01:26:11We have a lot of additional material on AI and ML workloads, and there are tons of blueprints and validated designs both from Cisco and Nvidia with Cisco hardware.01:26:25And then lastly, there's a QR link to our existing and future presentations that we'll be having within the next week or two weeks. So if you want to go ahead and scan that QR code, if you didn't get it earlier, that should give you a link to our registration page for information on upcoming.01:26:44Coming events.01:27:05And I think at this point we're done. Riley, I'm not sure if anyone's gonna do a ending slido or is at the end of the session. Oh nope, that wraps us up. Thank you guys so much for the insightful presentation and demo and thanks to everybody for joining. If you can see here we've posted a survey.01:27:07Day at the end of the session here and we would really appreciate the feedback to help improve our future offerings. Once again, thanks for joining this expert in sized series.01:27:27