Operationalizing Your AI-Ready Fabric within the Cisco DCN Ecosystem

Hi everyone and a very warm welcome to you all. I'm Lucy and I'm throwed to be your host for today's session. So today's session will be optimizing your AR ready fabric with.00:09Cisco DCN ecosystem. So this is 2nd installment of our beyond the series survival guide to navigating the complexity of AI networking and fabric. So we already have a fantastic session ahead and I'm honored to introduce our speaker today, the Cisco principal engineer, Paresh Gupta.00:20So before I hand the mic over to Paresh, then I will have a few quick housekeeping notes to ensure that you have the best experiences. So 1st don't worry if you miss the detail like today's webinar is being recorded and will be available on other DCN community page. So I will also paste them into the chat window.00:39For your reference.00:59And also if you if you experience any audio issues, try switch between your computer audio and the dial in option, and also we wanted to hear from you, so please feel free to drop your questions into the Q and A panel at the bottom of your screen throughout the entire presentation. And we will be keeping an eye on those and.01:00To address as many as we can. And with that slide just divert in. Harash, the floor is yours.01:20Alright Lucy, thank you so much and welcome everybody to the part two of 2nd in this series to the survival guide to AI networking. We talked on the same topic last week and today we'll be covering the part two. And just like last week, my name is still.01:28I'm still a principal technical marketing engineer at Cisco still this week and and you know, in continuation of what we talked last week, I'll cover, you know, a little bit more details about the GPU to GPU communication, a little bit more about collective communication congestion control mechanism.01:49How exactly Cisco and Nvidia are delivering the spectrum X ethernet technology to our customers. And then towards the end I'll I'll walk you through some of the demos using Cisco Nexus dashboard and hyper fabric. Right? Now, if you recall, I know it's been a week, so I'll quickly maybe build.02:09In three to 4 min I'll walk you through what we did last week. So essentially, if you recall, I mentioned that this is the entire AI infrastructure stack. You know, from Cisco we offered this as part of the Cisco secure AI factory. However, the focus of the session is on the AI networking piece to it, right? Essentially building.02:27The networks of an AI cluster. Just to be clear, we're not talking about using AI to operationalize your networks, right? That's outside the scope. The scope is really how you build the networks of an AI cluster essentially GPU to GPU communication and even GPU to storage.02:47Communication. And then also we talked about, you know, the different market segments where we fall and and how exactly Cisco as part of the Cisco Nexus one offering is providing a fully integrated networking stack that provides us the the choice of silicon be it Cisco silicon one or nvidia.03:06Our spectrum X ethernet silicon, the switches that are in 9000 switches from Cisco transceivers and the operating system on top of it using NXOS and Sonic. Right right? So again, of course I'm not gonna repeat everything that we talked last week, very quickly.03:25If you recall, I I talked about a typical GPU node, which is a, you know, a HGX kind of architecture. It has eight GPUs and each GPU has a dedicated nic. And these NICs as of today, they operate at 800 gig, right? So this is only for back end connectivity, but then we have somebody.03:44For front end connectivity for GPU to storage communication.04:05Right, and then, you know, I walked you through different kind of scenarios, some terminology of, of scale up, which is communication among the GPUs within the same node scale out, which is communication among GPUs of different nodes and also scale across which is you know kind of a network between, you know.04:10Fews that are geographically far away, right? I also talked about this switches, the 400 gig and 800 gig switches that Cisco is shipping right now, and very soon we'll be shipping the 1600 gig switches in air cool as well as liquid cool variant. I also covered.04:30The different rack form factors, the different type of OSFP transceivers, the one with the integrated heat sink, and the other one with the riding heat sink, right? And after that I explained some, a little bit more details about, you know, how exactly we create back end network design, essentially using two big concepts. One is the non.04:50Blocking network, essentially for for every downlink on a leaf switch we have a uplink. So that's a non blocking network design and the reason we create these networks is non blocking because.05:10When GPUs communicate, they really communicate at line rate in ingress and egress directions simultaneously. And when all devices communicate at the same time, you can imagine there's NO scope for any oversubscription in the network, so that's why we create a non blocking design.05:23As well as we create a rail optimized design, which essentially means within a scalable unit, the GPU one of all the nodes connect to leaf one, let's say, and then GPU two connects to leaf two and so on, right? And this really facilitates two big benefits. Number one, the GPU communication.05:40Benefits from the single hop forwarding of the leaf switches.06:01And then of course, the other benefit is the the amount of traffic that goes to spine switches, either, you know, it's completely eliminated or at least it's reduced. And then I also talked about the multi plane design options where instead of, you know, using a, a switch.06:04At, at let's say a high speed, what you do is like, you know, you reduce the speed on a specific switch. That means you increase the number of interfaces on the switch, and then the GPU connectivity remains the same because instead of connecting a GPU to one switch, we connect the GPU.06:21Due to multiple switches, right? So this GPU throughput capacity remains the same, but then they create multiple networks to connect the GPUs that leads to the benefit of high availability as well as scale, right? So this is all we talked last time too and I covered some of the, you know, that you don't really need, dedicated leaf switches for each rail in.06:41In fact, you can converge multiple rails on leaf switches especially if you're designing smaller GPU clusters, right? So these are again some examples, and this is one example of a front end network design tool.07:01Right? So again, look, what I just repeated in last 5 min, it's just like a crisp summary of of last time what we did. Now one point I will mention, and this is something I mentioned last time too, is that look this is a detailed topic. There's, there's a lot more detail into it. I have.07:13Content for two days. I'm trying to consolidate this within 2 h. So if you think there's a topic that I did not cover today, just let us know in the chat or Q and A panel.07:33And then, you know, we'll make sure in one of the following sessions we covered that or maybe by the end of this session, I'm able to answer that, right? So with that, let's, let's move forward and quickly focus on the network piece for the collective communication. So look, one of the things is that when these GPUs communicate with each other, let's say all these GPUs that you're seeing in.07:43This diagram, four GPUs, they are running the same job and because they're running the same job, they must be in sync all the time. And as part of that sync process, they communicate with each other using something called as collective communication. Now depending on the kind of library you use, there are different implement implementations to it. Nvidia has something called Nvidia.08:03Collective communication library.08:23Nickel AMD has Rockem collective communication library that's rickle. Intel has something similar, but from this session because you're focusing on networking, if you connect these GPU NICs to a switch, and then just like a typical network engineer because.08:25Lot of us in the networking field, you know, we we really care about, hey, what's flowing on the wire, and we, you know, our ultimate truth is that we like to put a packet sniffer and like to put that sniff sniffed amount of, you know, data in a wire shark and see how exactly the packets look. So if you do that, what are you gonna find is that.08:44That the GPU to GPU communication is really built off the RDMA operations, mostly the RDMA write operations. Essentially one GPU is writing data from its own memory to the other memory, and this is a RDMA as the name says.09:04It's a direct memory access without involving the CPU and the whole operating system stack, so that's why it's essentially fast. When these RDMA operations are carried over Ethernet network, it becomes RDMA or converge ethernet. And there's a version two of this protocol which really means is that there would be Ethernet header after that there would be.09:20The IP header and then UDP header, right? And after that, it's a whole infannie band transport. So the IBBTH header that you see that really stands for infany band based transfer transport header.09:39Right? So this is essentially the rocky V two protocol is now as you can see, there's NO rocky V two header as such, right? It's just a name, right? Which is one thing to remember. The other thing to remember is that for most of us, I believe we are very comfortable. Most of the network engineers are very comfortable in dealing with IP.09:53UDP traffic, right? So from that perspective, Rocky V two is nothing but you know IP UDP here, right? So that's something to be aware of. Now, the other some details that we should watch out for is that where exactly rocky V two is mostly used. Rocky V two is mostly used wherever high performance.10:12Data transfer is required. So mostly for GP to GP communication as well as for GP two storage communication. Now that does not mean TCP.10:32You know, is completely avoided. Of course, TCP is the, you know, most widely used protocol. So it's still used in both back end and front end networks. The other difference is TCP has a built in congestion control mechanism based on the explici explicit congestion notification. Essentially.10:43The networks detect congestion and then it informs the end devices about congestion and then the end devices adjusts its rate. So that's one way. The other approach that TCP uses for congestion control is based on the round trip time that's calculated.11:01Per session by the end devices. Now compared that to rocky V two when rocky V two specification was originally written, these details of congestion control was, were left to the implementation level. So what happened over the last, I would say around ten years back, a similar mechanism which was based on the ECN.11:17In Rocky was implemented. The name at that time was given was the data center quantized congestion notification as in DCQCN. And it works on a very similar concept. The difference is because UDP does not have a concept of a session, you know, the the one node cannot.11:37Not really informed the other node because there's NO session as such. So what happened is that in Rocky V two there was a new packet type. It's called congestion notification packet which serves the same purpose.11:57However over the last few years, you know, industry has realized that this DCQC mechanism for congestion control was really, you know, invented for storage traffic for GPU to storage communication, but the round trip time based congestion control algorithms are delivering better performance for GPU to GPU communication, so that's why.12:09Why, if you look at most of the back end networks, the the round trip time based congestion control algorithms are are very widely used. The other differences, you know, TCP mostly runs on a lossy or best effort network. What really I mean by that is, let's say if a.12:29The sender and receiver they are communicating and if the sender sends traffic faster than the receiver can ingest, what happens is packet drops. It's quite common in a, you know, most networks that we have seen. However, with Rocky V two, if you remember on the previous slide I showed you the entire, you know, sequence, the packet sequence number from.12:491250, if one of the packet gets dropped, the, the performance is significantly degraded. So to avoid that problem, we mostly run rocky B two in a lossless network. And how do we create a lossless network? We enable some kind of.13:09Flow control in the network with ethernet. This is priority based flow control or in short BFC, right? So how exactly you know we we run that. So this is exactly how things work when let's say in this example, if host one is sending traffic to host eight and if there's some congestion caused on spine one.13:25Spine one sends some kind of like, you know, pause frames, which is a mechanism part of the priority based flow control.13:46When the buffer utilization on spine one exceeds something called as a pause threshold. And then after that eventually these pause frames slow down, you know, the traffic that's incoming on spine one, but one of the side effects that happen is that as you can.13:52Imagine these pause frames spread.14:12To not just the culprits but also to all other devices that are sending and receiving traffic through the same paths. So this is why it leads to something called as congestion spreading. Right now this is a very well known phenomenon and to avoid this con condition, what we can do, we can have an end to end congestion.14:15Controlled working in the network. Now I already spoke about the DCQCN. This is the explanation of how exactly DCQCN. The way it works is that host one, let's say, which is the source in this case, sends some traffic, and it, it uses the ECT bits, which stands for the.14:35The ECN, the explicit congestion notification capable transport bit in the IP header, right? Now IP header has has eight bits for type of service and two bits are for ECN bits. So you see either the the source turns them as zero one or one zero and host one sends that packet.14:55Essentially, you know, the the packet reaches spine one and assuming spine one runs into congestion.15:15What it does, it, it turns on both the bits. So from zero one or one zero, it changes the bit values to one one. That's known as the congestion experienced bits. And essentially this traffic or this packet reaches the destination host eight.15:21Right? Now remember, host eight has to inform host one to slow down, right? But because this is UDP, there's NO concept of a session or a connection unlike TCP. So what host one does, it sends a special packet called congestion notification packet to host one and finally host one.15:39It just iterate assuming that the spine is is able to not run into a congestion scenario. So that's how DCQCN works, right? So essentially if you combine the PFC, which is a lossless networking concept with DCQC or with ECN together the.15:59These concepts, they deliver the best of both worlds which is a flow level scope because of the ECN mechanism, but at the same time PFC.16:19Avoids the packet loss, right? So it brings the best of both worlds. This and this is how it, it really works. Now, as I mentioned you know the RTT based congestion control mechanisms are being more prevalent in GPU to GPU communication. So that's something to watch out for. However, if you pay attention.16:29Well, I mentioned earlier that the GP to GPU communication networks or the back end networks we typically design in a non blocking way, which is zero over subscription. So that leads to the question that if the network is non oversubscribed.16:49Why would spine one run into congestion at all? Right let's say if spine one has X number of downlinks, it has X number of uplinks, why would spine one would run into congestion at all, right? And the reason for that congestion is because of the way IP Ethernet networks.17:05Do load balancing. So.17:25To understand that, if you look at this particular example where host one is sending traffic stream at 400 gig to host five, and at the same time host three is sending a 400 gig traffic stream to host six. Now what, what's possible is that when these two streams reach spine one, spine one uses something.17:27Call as a ECMP as in the equal cost multi path load balancing scheme. And because of that, it's possible that both the traffic streams end up taking the same link. That's say link number two.17:47Now as you can imagine, there's NO way we can send 800 gig of traffic, which is 400 plus 400 gig of traffic on a 400 gig link. All right, so this link or the link two runs into congestion due to overutilization While at the same time we have other links sitting idle.18:00Right? So this is the basic cause of congestion. Now, if you really think about why this problem happens because the intent of IP internet network is to, is to avoid, you know, out of order delivery of the packets of the same flow.18:16To, the receiver. So that was the real intent. But that intent doesn't really serve the best use case in IP in in AI networks. Why? Because, you know, these are some of the reasons, because there are fewer flows and also most of these flows are line rate, right?18:34Right, which runs into the condition, which, which is explained here. Now, this problem is, is not new. Most networks out there would, would see some kind of link imbalance problems. However, let's say if you have two links, right?18:54And one of the links is 60 % utilized, other link is 80 % utilized? Yeah yes it's a concern, but it's not a problem, right? The performance may not degrade. Why? Because most data center links don't operate at line rate, right? But for networks it's very different.19:11That GPU send and receive traffic at line rate.19:31So if two GPUs send traffic at line rate, even if one link is 98 % utilized, that means the rest of the two person traffic must be additionally sent on the other link. That means the other link is expected to carry a hundred and 2 % of the traffic which is not even possible and that's what causes congestion.19:34Due to over utilization on these links, and this is why the reason that despite having a non blocking network in IP net networks in in rocky V two network, it's possible to run into a congestion problem like this.19:54Now, how do we solve it? There are couple of approaches. One of the approaches that you increase the number of flows, right? So you compare on the left, where we have fewer flows, we, we, can identify a UDP flow for load balancing, right? And the number of UDP flows could be only so many.20:08What we can do on the Cisco switches is that you identify something called as a, as a, infinite band destination que pairs to load balance the traffic on different equal costs path, and because the number of que pair.20:27Flows are a lot more in assuming you increase the number of que pairs, the utilization becomes more uniform, right? I'm not saying it becomes perfectly uniform, but it becomes more uniform. And these are some of the results of the testing that we did before. So, you know, if you look at these charts from left to right, we increase the number.20:44Of Q pairs from the end devices, right? And then then you can find some details of how exactly to increase the Q pairs and you can also refer to nickel documentation where it explains that you can increase the number of Q pairs. But if you pay attention to those charts, you see the difference of the variation between the link utilization reduces.21:04As we increase the number of queue pairs, right? Now, I'm not saying you infinitely increase the number of que pairs, right? This is something you would have to experiment in your environment that what value maybe four or maybe eight or maybe 16 gives you the best values because look, you don't want to increase these values too much, otherwise you'll run into the resource contention problems, ok? But this is a very.21:23The valid approach. A lot of people have used these in in a production environment. In fact, I think Meta.21:43When public and mentioned about these this approach while training I guess their Lama three model with some 24000 GPU cluster and you know as you can see even in our own in house testing in Cisco, this approach has led to a very, you know, favorable results.21:50So this is the one approach of solving the problem of congestion due to over utilization caused by traffic imbalance. The other approach is instead of using a UDP flow for load sharing, what we can do, the Cisco silicon Ethernet silicon.22:07It can use something called as dynamic load balancing. And essentially what dynamic load balancing do, it, it has two modes. One is the flowlet mode, the other one is the per packet mode. In flowlett mode, the switches keep on inspecting the real time utilization of the links.22:27And at the same time, they also identify flowletts based on the inter packet gap. And if let's say the next packet that arrives, arrives after a certain duration, then those packets, that burst of packet is identified as a new flowlett, and then the new flowlett is sent on.22:45The least utilized link in real time, right? So because of that nature, because of the the flow let based load balancing, the utilization of the link becomes a lot more uniform as you can see when we compare the graph on the left, which is using the traditional ECMP approach.23:05With the graph on the right where which uses you know the flow let based dynamic load balancing. Now, however, as you can see, different flow lets may have different sizes. So the utilization of the equal cost paths becomes nearly uniform.23:24Not perfectly uniform, right? Now, if you want to achieve a perfect utilization of the links, you know, you could use a per packet load balancing. Now, as you can imagine, as the name says, this is a per packet load balancing, which means that the key, the, the switches keep on inspecting.23:41The real time utilization of the links, the equal cost links, and then the next packet is sent on the least congested link in real time, right? This is what is per packet mode of dynamic load balancing. This is also called as packet spraying. And this, as you can see again from the charts that this leads to perfect.23:59Link utilization and we have extensively tested this technology.24:20Now, if you recall a few minutes earlier I said that the whole intent of ECMP load balancing scheme was to send the packets of a single flow on the same link so that the receiver receives all packets of a flow.24:24In order, right? But if you send packets in a per packet load balancing approach, of course it's, it's very much possible. In fact, it happened most of the times that these packets of the same UDP flow arrive out of order on the receiver, and this is exactly where the partnership with cisco's, you know, switches.24:40This is called Nexus one architecture with the NVIDIA spectrum X adaptive routing has a role to play. Now the way this works.25:01Is that if you connect the Cisco nexus or N 9000 switches to nvidia's supernix, this can be nvidia's connectx seven supernick. This can be a Bluefield three nick or this can be even connectx eight nick. What happens is that there's a handshake between Cisco switches.25:09And Nvidia supernick. And this handshake enables adaptive routing as well as direct data placement capability. So the way it works, let's say, as you can see in this example, if node one GPU one has to send some, you know, it has to do some DMA or remote direct memory access.25:29To some amount of data to node two.25:50And if that data takes four packets, so the node one sends those four packets in sequence, of course, but it's possible when leaf one sends these packets, these packets go slightly out of order.25:54Based on the real time utilization of the four uplinks on leaf one. Finally, these packets reach, you know, spine one spine two and these packets arrive on leaf two, and then leaf two sends these packets to node two in the same order. So it's as you can see, it's possible that these packets arrive out of.26:10Order, you know, on, on the receiver. Now at this time, what the direct data placement technology does, it takes the payloads from each packet and places the payload directly into the memory of the receiving GPU.26:30The way this Supernix is able to do it because the destination memory address of each payload is directly encoded in the packet header, right? So it need not wait for reassembly to happen, right? Watch out there's NO reassembly happening on Supernix because if supernix have to do reassembly, Supernix need to maintain.26:50Oh, and have to have a large amount of memory. That's not what is happening here. What's happening here is the direct data placement.27:10Of the payload from each packet directly into the destination GPU memory, ok? And as I said, when Cisco switches connect to the Nvidia supernex because of the partnership between Cisco and in NVIDIA, this capability is automatically enabled.27:18Okay, so here you can see what are the performance gain out of this capability? You know, I have posted the whole live demo video on YouTube. You can find the link to it. I'll let you watch it. Essentially the summary of that video is that when we run.27:35CMP based scheme and compare it to the per packet mode of dynamic load balancing, we see 37 % performance gain.27:55This is just one data point. There have been other tests in other environments where people have seen some 40 % performance gains, some people have seen 60 % performance gain. Your mileage may vary right like but in this particular scenario where, where, you know, we connected 256 GPUs and ran this.28:05Particular test, we saw 37 % performance gain. And this video, you know, I'll let you explore at your own on pace.28:25Okay, the link is available here and we'll share the slides with you. Okay, so going further, what does really take to configure you know these backend networks? There are a couple of things. In fact, it's a lot more simpler than all the theory that we talked about because I think understanding the.28:33Why, like why we do things the way we do is more important but when we go to how, it's relatively easier. So the 1st thing we do is we configure IP addresses on all the NICs. Typically, we use IP addresses in/thirty one/thirty one subnets and then you can use any favorite routing algorithm of your choice to.28:53Create all to all connectivity. We recommend using BGP because of the feature called WCMP, which is.29:13Weighted cost multi path, but you can use OSPF or ISS to whatever you prefer. That's one. 2nd thing you should do is increase the MQ to the maximum value. Why? Because 4000 bytes for KB, I think it's just safe to increase the MTU to nine to one six bytes. The other thing you can do.29:19You you should do not you can do, but you you must do in fact, is enable as a lossless network. As I mentioned, you can enable, Ethernet network into a lossless network by enabling priority based flow control. Also, you should enable congestion management, maybe DCQC or or use a RTT based.29:39Control algorithm. Also, if you require a dynamic load balancing, you enable that feature. As I mentioned, there's a single command on the Cisco switches. If you enable that, this whole capability is enabled on the, on the switches and also it enables the capabilities on NVIDIANX. And also you should enable observability.29:59Which is getting the, not just about the network observability, but I highly, highly recommend to have the full stack observability during your testing or validation phase before putting your AI clusters into production, right? Now, all these points that I mentioned, you can enable via CLI.30:19Right, right. You can use NXOS CLI and you can do that. However, what, what we recommend is that, you know, there are a lot of moving paths, there are a lot of best practices, there are a lot of fine tuning. What we have done, we have integrated all these fine tuned values within.30:37Nexus dashboard and hyper fabric. So this is the reason we recommend you to use Nexus dashboard and hyper fabric and I'll show you the demo of that. Now talking about Nexus Dashboard and hyper fabric, these are two you know similar operation operation models of Cisco networking from Cisco. The 1st.30:53This one is Nexus dashboard, which is the on premises or the controller of the on premises network, to manage the Cisco and 9000 switches that are powered by Cisco silicon one or or Cisco cloud scale of silicon or or even NVIDIA spectrum X.31:13Ethernet silicon based switches, right? The other model is, which is the Nexus hyper fabric, which is cloud hosted controller of the on premises network. So one of the models is very similar to Nexus dashboard. Essentially you bring Cisco switches, you can bring your own computer or you can bring your own storage.31:31But now the controller is managed by Cisco. You don't have to install it. You don't have to run it on premises. Cisco is running it in cloud.31:51The other which is a lot more interesting model is from Nexus hyper fabric, which is a full stack AI infrastructure, including compute, network, and storage, right? Now this is really meaningful, especially if you're deploying AI clusters for the 1st time because look, there are a lot of moving parts like you know how many GPUs, how many.32:00Cables real optimized design, what type of transceivers, you know, as I mentioned, there are multiple networks of an AI cluster like, you know, you can have multiple backend networks if you're going a multi plane design, you have a front end network storage network, out of band management network. So there are a lot of moving paths and with Cisco Nexus hyper fabric, all these pieces get a lot more simple.32:19Okay, and let me show you, that, right? So the beauty of and which is the most interesting piece of of Nexus hyper Fabric is that you can try this right away, right? This is a cloud hosted controller. You don't need me or anybody else who run it.32:41You can just log into hyper fabric.cisco.com and try by yourself whatever I'm about to show you. Now, of course for the because we just have 1 h, I'll not be able to show you everything so I'll just give you a hint of what exactly.32:59Hyper fabric is doing and I'll let you explore it, right? So I've logged into Nexus hyper fabric. Now, one of the things you can do is like, you create, add new fabric. Now of course you can do new fabric in designer, but what I highly recommend is go to new fabric from template.33:18And you see some kind of networks here, right? However, for AI cluster, for a full stack AI cluster offering, you click on AI clusters and here you see there are some pre created designs. These designs are based on the Cisco enterprise reference architecture.33:36Which is in compliance with the Nvidia enterprise reference architecture. That means you can deploy these designs confidently. They have been vouched, they have been validated and they have been massively deployed out there, ok? And you can see based on the number of GPUs you need, you have multiple templates out here.33:55So let's say, you know, I decide, hey, I want 01:20 GPU cluster, right? So that's part of the large AI cluster. So let's say I select large AI cluster template. Now when I select that, as you can see, all kind of devices that I need, the number of devices that I need, it's all pre populated for you.34:15Right? Also the switch groups, the rail group, every, every design element that I talked today and I talked last week, it's all built into Nexus hyper fabric templates, ok? Now, here.34:34If let's say I decide, hey, there's NO change I need to make, you know, and I'm assuming you would have to make minimal changes, maybe some of the environmental changes, let's say what kind of airflow direction you want. After that you save save fabric blueprint. And this blueprint, as I said, it includes all kind of.34:50Details that are recommended as part of the enterprise reference architecture. Things like the GPU servers, like, you know, these are the kind of GPUs servers, the storage servers and the connectivity plan, ok? Now also if you scroll down, there's a lot more details, as I said, I'll let you explore this, like things like.35:12Like real groups and all.35:30Also bill of materials, now if you are a partner or even if you're an end user and you're trying to get, hey, what kind of components I need to, to host, let's say a hundred 28 GPU cluster, you have the entire list. And then this list you can export to CSV or even request an estimate ID from Cisco commerce.35:31Workplace or in short CCW, which is Cisco's portal for selling anything that Cisco makes, or or Cisco sells, right? So this is ready made. I know some of the, some of the folks who spend a lot of time in building estimates, so this whole, you know, task is simplified.35:51And again, this is validated, so you don't have to you you can you can proceed with confidence on this part. Also there's a capability called on site, so let's say you buy the equipment, you deploy it, now you have to tell, you know, your team members, hey, go connect it. Now, but how exactly people know which cable goes.36:10There, right? All these details are, are prescribed here in every detail, and when people are done with it, they can say, hey, hey, I'm done. So essentially, you know, you can assign the task to people and they can say the task done. And again, the, the UI and all the, all the details here ensure that there's NO error.36:30In connecting the cables.36:50Okay, now also you can review the configuration and pro and and push the config to the network when it's ready. But the other thing you can do is like, you know, let's say the cabling part, right? So of, of course you can export CSV because I know some people who really like dealing with the Excel sheet. So export.36:52CSV is the function, but you can run cabling, right? So it's, it's running all the cable and now it's telling you exactly, hey, this device, this kind of rail interface, this particular interface consume this kind of transceiver.37:11This kind of cable, the other end has to be this kind of transceiver on this interface and this device, right? So, this significantly simplifies buying an AI cluster and deploying an AI cluster, right? And essentially, if you think about how long it takes from.37:24From ordering GPUs to putting them in production essentially when you start generating tokens out of AI cluster, Nexus hyper fabric significantly improves that whole time, right? So that's a brief overview of Nexus hyper fabric. As I mentioned, it's a.37:44Cloud hosted, it's a Cisco cloud hosted controller. You should try like as I said, you don't have to install anything, just go to hyperfabric.cisco.com and try out some of the things that I showed you and there's a lot more you for you to try, right? You don't need me or anybody else for you to try. You can just click on these things and.38:03These are self explanatory.38:22Okay. So this was about the Nexus hyper fabric. Now, you know, in the slides you'll, you'll find some of the details, right? Like, what, what I talked about and a lot more. The other offering is about Nexus dashboard. So Nexus dashboard, we have significantly improved nexus dashboard over.38:24For the last few years and now it has, you know, the best capabilities for Ethernet based RDMA networks, right? And let me show you that like let me prove that to you what I just said, right? So.38:44This what you see is Nexus dashboard release four.two. For those of you who have been running Nexus dashboard, you, you may find this a little bit different. Why? Because starting four.two release, we have, as you can see, we have improved on the UI. There are multiple.38:57Teams to it, like there's a dark mode, there's a midnight mode, and of course there's a, you know, the light mode that you guys have been running, right? So next is dashboard essentially when you log in log in you land it here, but after that you can go to manage, click on fabrics. And in the fabrics, you can go to actions and say.39:14Create fabric and you see there are multiple types of fabric based on what you wanna do. Here you'll see see create new fabric and there's a new AI fabric type. This AI fabric type for most of the times you may wanna go with routed fabric, but even if you want to run VXLAN, that's totally ok. And note that all you have to do here is provide the name of the.39:34Fabric and the BGP isn't that's pretty much it. Everything else, all the best practices fine tune values all are baked in already in the, in the template.39:54But however, if you're a power user and if you want to customize these values, you can of course go to the advanced mode and then you see there are some settings enabled on the left. And then you can customize it. Like you can fine tune, let's say QS policies, you can enable dynamic load balancing, which is related to the nvidia spectrum.40:04Next adaptive routing technology, the per packet mode of dynamic load balancing. And you can customize all of the values that nexus dashboard provides. And still if that's not enough, you still want to push your own config, you can use a free form config, right? So essentially.40:24I hope you are able to see the simplicity part to it, which is you just have to put two values, the name of the fabric and the BGP ASN, that's pretty much it. Everything else, all the best practices are baked in, but at the same time you can customize it. Okay, so the fabric is created by this time. The next step would be.40:41You add the switches to the fabric, right? That's a typical workflow that we use in Nexus dashboard. So here you go to actions and then you say add switches. Now assuming all these switches have been deployed, they have been powered on, they have been cabled. All you have to do after that like provide credentials to one of the switches in the fabric.41:00After that nexus dashboard goes and discovers all the switches in the fabric, assuming a spine leaf or whatever that, you know, design might be. As you can see it has discovered four switches, two leaf switches and two spine switches. You can select them, what switches you want to add to the fabric.41:20And, it takes a while, for maybe 15 20 min and after that you go and look at the inventory, select the role of the switches and that's essentially your last step. After that, you know, you push the configuration to the switches.41:39In nexus dashboard we say recalculate and deploy, so the moment you finish this step, your, your whole network will be provisioned as per the best practices are recommended with the fine tuned values of, of congestion control, QoS buffer values. You.41:57Yeah, as part of the AI cluster template from Nexus dashboard to all the switches, right? So that's the provisioning of the AI cluster. The next thing you would do, and like something what we have added recently is integration with slum. Now, what is slum? It's a job scheduler. It stands for, as you can see on this.42:17Screen it stands for simple Linux utility for resource management. It is the most widely used.42:37Job scheduler. So what Nexus dashboard has done starting four.two release, it has added integration with slum. Adding the integration is really easy, you know, all you have to do just select it, provide name and credentials to the slum server. And yeah, it takes maybe five to 10 s to detect that you know the slum server is.42:44Is working well and as you can see the the status done.43:04Turns active. Okay, now if you're wondering, hey, what's the use of adding this visibility, I'm about to show you why or what problems the slurm integration solves for you. So by the way, before that, I hope you're able to notice for those of you who have run nexus dashboard before that starting four.two release.43:08You see a new card here which shows AI resources about your number of AI clusters you have, the number of scalable units you have, the number of AI servers, the GPUs you have, the number of jobs that are running right now and the GPUs as well. Of course, you can click on the fabric as well and start inspecting the.43:28Inventory for your AI resources, about your servers, your GPUs and the jobs as well, right? Now the job information here.43:47Is coming from slurm. So that's exactly the reason of that slurm integration. You can see there are some jobs here, there are 40 total jobs, 39 have run before. One is running right now. You can click on all the jobs and start going to details of each, each and every job.43:57And what exactly Nexus Dashboard is doing? Of course, it's showing you the entire back end network, and it's showing you what exactly might be wrong or might be problematic on your back end networks. So in this particular case, there are a bunch of problematic metrics that we have discovered based on our own.44:15Validations that we have done. And then what we do, we sort them out and show you the top ten most critical problems here. So in this case, you see there are some packet drops. Now, we don't expect any packet drops in the back end networks because typically, as I said, we call create them as a lossless network.44:35But still there are some packet drops, so as you can see, you know, detection of the problem is, is a lot faster. And by the way, all these problems are generated as anomalies and then the resolution of the anomalies are shown too. I'll show you that in a minute, but you can go into the detail of every job level.44:52Two, and then most importantly, you can actually visualize the the network path taken by the distributed job. So in this case, this job is running on 16 GPUs. You can drill down into each GPU, each server, and then you can find the its associated NIC.45:12And what link the Nick is connected to the switches, right? So if let's say a rail optimized design is misconfigured, you can see it. Also if anything that goes wrong on those links, it's correlated and shown here. You can also go at the GPU level, it's also not just the job level, the GPU level.45:31Well as well. And then all the metrics, all the relevant metrics of the GPUs are also fetched, as you can see here about temperature, about utilization, about the memory usage of the GPUs. Now, why this thing is important because.45:51A lot of times, if the GPUs are running hot, their performance degrades, right? Let's say if a GPUs is running eight 80 °C, its performance gonna reduce, right? If you suspect your jobs are slowing down.46:06And if you have temperature charts, not just the charts, Nexus dashboard also of course it's plotting chart as you can see, but it also has thresholds and anomalies and it creates those anomalies as you can see right here, and then it shows you, hey, what's the problem? When did the problem happen and what can be done to solve the problem? And.46:21As you can click, as you can see, right? How do I fix it? What's the impact of the problem? So a lot of this correlation and guidance.46:41Is built in within Nexus dashboard. Now I'll take another example of another job that's running. Now, as you can see, if any of the job is showing degraded performance is shown right here. As you can see, there's a live job with anomaly and there's one live job with anomaly on the right of the screen. So you click on that so.46:48Imagine finding the problems is so easier, is a lot faster, and the faster you you detect a problem, I don't have to tell you the faster you solve it. And as you can see, there are anomalies and they are correlated. If you look at the.47:09The text of the top ten anomalies on the right, it correlates network health. It correlates Nick health. It correlates GPU health with the job, and then it correlates if anything is is anomalous on on any of these entities.47:30It creates an anomalies and shows it to you, right? So I hope you're able to appreciate the correlation engine with the Nexus dashboard that ensures that the the the job or the health of your distributed jobs is is up to the mark, but if it's not, which happens a lot of times.47:48Then you are able to detect, why it's degraded, when it's degraded, and what's the resolution to it, ok? So this is one of a particular use case. The other thing is you can of course go in and start looking at the congestion details, right? All the details metric we put we bring in.48:06Like, you know, 20 plus metrics from every switch interface and then correlate it with the end to end data path, right? Now look congestion, as you can see right now, congestion is a, you know.48:26Is a loaded topic, right? And I'm sure some of you are experts into congestion, but not everybody in the team, you know, understands how to interpret the PFC pause frames, the ECN marking, the buffer utilization and all that. For those folks, what we have done.48:37Within nexus dashboard, we correlate multiple such counters and turn this into a simple to consume congestion score, as you can see on the screen right now, the congestion score values ranges from zero to ten. And what it really means is that zero to three is like.48:53Mild congestion. Four to seven is moderate congestion. Anything seven plus is severe congestion. What really means is that at severe congestion, the switch is dropping the packets. As you can see right now, there's a spike and at that time congestion score is nine.49:09Now at the time if you see why the packets are getting dropped, we also see some errors, right? Now, if you wanna need, ok, NO, let's say, ok, what could be the reason of that packet drops, right? So we go to each and every queue level and then we find that, hey, there's a queue default is dropping the packets. Typically, we expect.49:25Traffic to go in Q3, but we're seeing packet drops in Q default, and also notice that these are discards, these are time.49:46Transmit discards, these are not CRC drops. These are not bit error problems. These are transmit discards. So if I connect the dots, that's my guess is this is a kind of misconfiguration because of that misconfiguration, most of the lossless traffic should have gone to the lossless or a NO drop queue.49:53But in this case, it's going to a default queue and that's causing the congestion drop. Also, the congestion value bubbles up all the way to the fabric level, right? So you don't have to guess where the congestion might be. The the fabric will show you where the congestion is and then from fabric level, you go to the switch level, from switch level.50:12You go to the interface level and from interface level you go to the queue level, right? So there's a simplification of this information as well as the details are available to you whenever you want to go to that leave detail, as you can see from this particular whole workflow that that you see on the screen, right? And in this case, now you see all the traffic is going to.50:32Q3 yes there is congestion, but there's NO critical congestion.50:52Right? That means there's NO packet drops, right? So of course when there's too much traffic in the network, congestion is ok. What we don't want to do here is dropping the packets, which is clear from this example. And also you get to see the metrics per per queue level as well for all the congestion related metrics as well.50:56Okay? So I know there's a lot more details, you know, again, if you have more questions, if you wanna know more, if you wanna understand these details more, you know, reach out, right? Or or let us know what other topics you like to like us to cover more, right? Now the other.51:15Point that I mentioned is that because I talked about two operational model from Cisco, which is like a on premises, you know, Nexus dashboard and nexus hyper fabric.51:35Right? Sometimes the question is, hey, can you, can you bring these two products together? And the answer is yes, we are already working to bring these products together as part of the Cisco Cloud control, which is essentially all your Cisco inventory at one place.51:45All right, so yes, you still will continue to run Nexus dashboard on premises and Nexus hyper fabric in the cloud. But if you want to see a uniform management plane, which is seamless single sign on, Cisco Cloud controlled is the answer. Okay, so that brings me kind of like, you know, towards the end of the.52:01Session. And if you recall last time I started with saying that look, these AI clusters, you know, range in all kind of capacity. There are some hyperscalers who have.52:21Hundreds of thousands of GPUs. And on the other end, we have enterprises, and in between we have new clouds and sovereign clouds. In Cisco, what we're doing, we are offering a Cisco Nexus one full stack networking offering, which essentially serves the purpose regardless of your deployment model. And how we do that.52:31We offer the systems that are powered by the Cisco silicon one or even Nvidia spectrum X ethernet silicon.52:50Using our own transceivers that are and then these switches are powered by NXOS or sonic operating system. To manage these switches, we offer two operating model. It can be an on premises nexus dashboard or a cloud hosted Nexus hyper fabric. And finally, the most important thing is when customers are investing so much in these AI clusters, they will.52:59Want to see the confidence that, hey, this, this, whole model will will deliver the promise performance. And for that, we have published many enterprise reference architecture as well as the cloud reference architectures While working with industry ecosystem partners like Nvidia.53:18AMD, Intel, and as well as the storage vendors like Wecha, Wast, the DDN, pure storage, Hitachi and so on, right? So look, our intent is regardless of whether you are a hyperscaler or or you are a new cloud or you're a sovereign cloud or maybe you are an enterprise, we want to provide you a consistent.53:38For networking experience.53:58Okay, so that's all I had. I really like to thank you for being here. And as I said, if you have any further questions or any other topic that you want to dive further, feel free to reach out. Lucy, back to you. Okay, thank you Paresh. So that's for the hanging out with us today.54:02The hope you have enjoyed the session as much as we did. And before you head out, could you do us a quick favor that we will be love to hearing your thoughts? Like, please take a moment to complete our brief post webinar survey to share your feedback and let us know what kind of topics you're going to see in our future series. So your idea.54:22This will help us to keep things fresh and relevant and so don't be shy that we are all expecting for your answers and also your things that you wanted to listen. So thanks for being part of our committee and we look forward to seeing you at our next event, so that's all for today's session.54:42