Understanding the Unique Requirements for AI/ML Clusters

Hello and welcome to everyone. Thank you for joining us today. We're very excited to present the 1st of the two.00:08Sessions your survival guide to AI networking.00:20I'm your host Shannon movie. I'm the program manager for DCN at Cisco, and I'll be hosting today, but the the man of the hour is Paresh Gupta. He's a principal engineer.00:23Before I hand the mic over to get a few housekeeping items to cover, the webinar will be recorded and we will be posting it on our data center community pages. I will have the link for you during the next session, but it's easily available if you join.00:40Or visit our DCN excuse me community pages. And we'd love to hear from you today during today's present presentation. This presentation is all about you guys and so the more you engage with us, the more we cannot.01:00Answering your questions and maybe more, more knowledgeable. So if you have questions, please feel free to post it in the panel, the Q and A panel. It is down near the bottom window.01:18And if you have any audio trouble, we advise you to probably drop off and log back in or try using your phone. So with housekeeping taken care of, I handed over to Paria and thank you Shannon. Hello, everybody. My name is Paresh Gupta.01:34I'm a principal technical marketing engineer at Cisco and for the next 60 min, I'll walk you through the survival guide to AI networking.01:54This is a two part series session. Well honestly, there's a lot more to talk on this topic. I have content for two days, but I'll try to consolidate within 2 h. With that, what I strongly, you know, suggest is that.02:05Anything that you see that we are not talking about because of course we cannot talk everything, ask us questions if there's some topic that you would like us to go more detailed we can of course cover that next time, right? So with that, I'll go a bit further and like talk about the high level that what is the focus of the site.02:23So essentially when we talk about the whole journey of A AI, the artificial intelligence, there are three big stages in it. The 1st one is training. That's when we take a model, we build the model, and then we find the right parameters of the model.02:44Which makes it to predict when we try to use it. So that's called training. Then 2nd phase as you see is fine tuning, which is really about you know some of the organizations take a pre trained model and customize it a bit further for their own data set. So that's what is called as fine tuning.03:01And then finally, the most important stage of of this whole journey is the inferencing. That's exactly where we as end users start interacting with the models, right? Now to achieve these three stages, it requires the entire infrastructure behind it. So at the top level, you're looking at the AS software, these can be hosting the models. Now.03:18Most of these AS softwares are containerized, so to manage those containers, you need some kind of containerized management platforms so that's exactly where Kubernetes platform comes in play. Then you need the compute capacity to host these applications. You need storage, of course, without storage, AI.03:38And achieve much.03:58So that's where the storage has a role to play. And then of course the networking which connects all these things together. And of course, this becomes, you know, kind of a complex stack, so you must secure it to prevent unintended use cases, and also you must have the end to end visibility or the observability.04:00In the stack, so that's exactly where the Cisco Splunk observability comes in. And this entire stack, Cisco offers this as part of the, as part of the Cisco secure AI factory. Now there's a lot to go in details on each, but today the focus is and for these two sections, the focus.04:20This.04:40On the AI networking piece essentially how do we build the networks of this AI cluster? Now, just to be clear, I'm not gonna talk about how to use AI or how to run agent tech ops to operate the operations of your network or I'm not gonna talk about how AI works or how well it.04:41Work or even chat GPD, right? That's a separate conversation. So I just wanted to clarify the focus of the session because when we say AI, it, it means many things to many people, right? So with that, I'd like to clarify that.05:01How exactly if let's say you or or your colleagues or you know others who are building the AI cluster, how you should approach this. The reason is it depends on where you fall in that spectrum. On one edge, we have the hyperscalers. These are the organizations that focus on building tens or even hundreds of.05:18Thousands of GPU clusters, and on the other end end of the spectrum we have the enterprises who build smaller relatively smaller clusters, and in between we have the new clouds and the sovereign clouds. New clouds are the organization which are specialized into offering a GPU as a service offering.05:38And then sovereign clouds offer something similar, but they are more specific about, let's say, geographical boundary, let's say specific country where the government wants to keep this service within their own country, right?05:58The use cases are different too. Hyperscalers, do all, they they train the model, they fine tune the model, they also offer inferencing services, but enterprises, they are not really focusing on training a foundational model from scratch, but they are really taking up pre trained model and then fine tuning it.06:11And inferencing it. The locality is different too for these GPU clusters or AI clusters, that where exactly the GPUs fall across. So hyperscalers, some of the hyperscalers who have massive amount of GPU compute power. The the challenge is they are unable to find the power at one location. So they are building multiple GPU clusters which are.06:31Geographically far away, but they still want to connect them, right? So that's exactly the term that we use for scale across connectivity, but I don't see because the deployment size of of the GPU clusters is relatively smaller for enterprises.06:51You know, I don't see much focus on enterprises for a scale across. And by the way I'm going to explain these terms a little bit more detail in the next few minutes. The one major difference between hyperscalers and the enterprises that hyperscalers have massive R and D budgets. In fact, some of these hyperscalers are writing their own firmware for the nick. Now.07:06I don't see enterprises doing that. What enterprises are doing that is, is, expecting a vendor developed and a vendor supported software, right? And this is exactly where the whole value of Cisco secure AI factory comes into play.07:25And the opportunity size is different because the hyperscalers, there are few, you can count them, maybe five or ten who are doing foundational training of these AI models, but there are many many thousands of enterprises across the world. And look, the reason I'm explaining all this to you is that.07:42A lot of times when I talk to customers, the thought comes that, hey, you know, that particular hyperscaler did something, why can't I do the same? The reason is because their environments are different. You know, their challenges are different and therefore the solutions are different. So you must analyze what are your constraints and accordingly apply a solution.07:59Because what works for them may or may not work for you, right? The kind of challenges that people run into, let's say a hundred thousand GPU cluster or maybe even a few thousand GPU cluster if you are not scaling to that level, you probably won't run into the same problems.08:19And if you're not running to the same problem, it does not make sense to solve it. Right. So with that argument, you know, I'll go a little bit further and talk about how exactly the cisco, Cisco networking offering is trying to address the challenges of AI networking. In Cisco, we came up.08:35With this architecture called Cisco Nexus one at the very foundational level of this architecture.08:55Is the building pieces of what builds the network. The 1st most important critical piece is the switching silicon. So this is where we offer the Cisco's own silicon one or we have also productized the systems, the N 9000 series systems using the nvidia spectrum X Ethernet silicon. So that's the silicon piece.09:00Of course, then we make switches out of these. We also offer optimized optics or the transceivers for it. And then of course you can run NXOS, ACI or sonic. Now, to manage these switches, we offer two operating model. One is the on premises management of the on premises network using.09:20Nexus dashboard. The other one is you can use Nexus hyper fabric, which provides cloud management. And of course, NO network is complete without security and observability to, you know, keep it secure and, you know, detect the problems before they become problem performance impacting. So that's where we provide a fully integrated stack with end to end obser.09:40Right, so I'm gonna use these terms. I'm hoping that you are aware of how Cisco approaches the AI networking.10:01If not, I hope this explanation at least gives you some connecting dots that what exactly I'm talking or if I'm using NXOS, what exactly it means or if I'm mentioning Nexus dashboard, what exactly it means, even with next silicon one or the NVIDIA spectrum X Ethernet silicon.10:07Right? Now, starting from the very basics, if you look what exactly makes an AI cluster, the computation power of the GPUs is the most critical piece of an AI cluster. So there are multiple GPU nodes, and if you look within the GPU node, what exactly goes within the GPU nodes? I'm taking the example here.10:24Of the NV Nvidia HGX architecture. This is also called as NVL eight. The reason it's called NVL eight is because there are eight GPUs in a single device, a single server.10:43And these eight GPUs are connected to each other via high speed NV link. So that's why it's called as NVL eight. Now, if you're running a job that requires, let's say up to eight GPUs, then all the traffic remains local or within the device. However, if you're running a job that requires more than eight GPUs.10:56Then the traffic communicates with other GPUs and other servers.11:16Using the dedicated inter GPU backend network, and you see there's a dedicated one to one mapping from the GPU to the nic. In this case, these are black belt 300 or the B 300 GPUs from NVIDIA and each each GPU has a dedicated connectx eight nick. The connectx eight Nick is a 800 gig.11:22It provides two ports of or two interfaces of 800 gig, two interfaces of 400 gigs, so that makes it 800 gig.11:42Along with that, it also provides the the front end connectivity and storage connectivity for accessing storage that depends on the kind of next that you have. And then all these servers can be managed by the Cisco inter site you know as a, as a cloud hosted platform.11:51The other example is using the AMD GPUs, so in Cisco UCS 885A, we offer AMD three MI 350 03:50 GPUs series. They are similar to, you know, what I just explained, they are eight GPUs with a sing server and each GPU has a dedicated AMD Pensando Polar.12:08A 400 gig AI nick.12:28Right, and then of course the front end connectivity and the storage connectivity can also be there. Now, be aware of all these interfaces that you see on the servers, when GPUs communicate with each other, they really communicate at line rate. So that means when you run your, even when you deploy these servers and if you.12:30Run your 1st traffic test. You will see that all these backend interfaces get utilized at the full capacity. So that's very different with these GPU nodes or GPU clusters because typically we don't see line rate utilization on the interfaces, but in AI environments.12:50Although they are eight 400 gig nicks or in this case of eight, there are eight 800 gig nicks and all of them send and receive traffic at line rate simultaneously. So that creates some of the newer challenges and then we'll talk about how exactly.13:07We address those challenges. This is another example. This is the example of a rack scale system. This is the NVL 72 from Nvidia or the GB 300. If you look, there are ten compute nodes at the top, then there are nine NV link switches and and then they are.13:24Are another eight compute nodes or compute trays. Each compute node has four GPUs and each GPU has a dedicated connectx, eight nic. Now because there are 18 trays or 18 compute nodes, if you run the math, we're talking about hundred and 44 400 gig interfaces per rack.13:44Just for GPU to GPU connectivity. And in addition to that, there's also the front end front end connectivity. So for that, there are two 400 gig interfaces per compute node for front end or storage connectivity, so per rack, you're looking at 36.14:04400 gig, interfaces for front end connectivity. So as you can imagine, there's a lot of networking that's needed here in these environments to connect these rack scale systems. And as you will see you know as we progress through these sessions, I'll walk you through some of the network designs that, you know, we have created for some customers.14:21But going back to the very basics, this was our GPU nodes. Then in the AI clusters, you know, you would have some kind of front end network. This is how these GPU nodes are accessible to the external world. You'll also have a storage network. This is how the GPU nodes access the storage.14:42A lot of times customers decide to converge these networks because there's not much traffic on the front end network and storage network is what causes the most capacity utilization. So it does make sense for the affordability reasons to converge these networks. And if you're talking about a storage network, of course there has to be a high performance storage.15:01Now high performance storage comes from multiple vendors. We have partnership with Wast, we have partnership with DDN, with pure storage, Netapp, Hitachi, and the list goes on. There are a lot more details around it. For this session, what you need to understand is that depending on the.15:21The architecture, there might be multiple nodes and depending on the capacity of the storage you want to add to your AI cluster.15:40There might be multiple nodes, and most of these nodes offer dual ports of 200 gig and now they're even moving to 400 gig. So when you talk about network design, you must account for that connectivity. Now some of these storage vendors also have a storage backend network. The storage backend network is dedicated for.15:48You know communication among these storage nodes. This is different from.16:08The the inter GPU backend network. But for, you know, connecting the GPU nodes to the storage nodes, all you need to remember is that there might be 200 gig or 400 gig connectivity on the storage nodes. In addition to that, there's some also standard storage. This is really for storing your, you know, some data that's not really required.16:12That high performance, let's say logs, you know, storing your image files or, you know, ISO which are large files. So that can go on the standard storage.16:32And then we have some control nodes. So these control nodes is where your cluster management applications run. This is how you can configure your server. You can monitor your server. There are some examples like you can run your Kubernetes cluster from here. You can run your scheduler like slum from here. You can run nvidia base command manager, from here, you can.16:43Run Cisco Nexus dashboard for network management.17:03From these control nodes. And then, there would be a dedicated inter GPU backend network. Remember most of the times the this inter GPU backend network is dedicated for GPU to GPU connectivity, and then because there are a lot of devices, there would be an out of band management network.17:06Right? Now, this is how what you see right now is a different kind of terms that we use. I already talked about the communication among the GPUs of a single device. That network, which in nvidia's case is NV switch in amd's case is infinity fast.17:26Fabric.17:46The intra device, the within the device communication is mostly used as a scale up networking. But if you're connecting multiple such devices to each other, that's called scale out networking. The other team name for that network is the inter GPU backend network, right? Or people also call it as.17:46Compute fabric that's called scale out networking.18:07But as I said in the beginning, some of the customers do not have enough power capacity at one location, so they create one data center or one AI factory at one location and then the data center two at or AI factory two at another location, and then they want to connect these different environments. Now, if.18:09You know the the backend networks are connected with each other. There's a term for it scale across network. Also, I would say traditionally, you know, while connecting different data centers, there's another term that has been floating in the industry for a while that's called data center interconnect or DCI.18:29Alright, so you can, you can argue whether scale across is similar to DCI or not. Well, I mean, I'm just trying to convey that be aware when you come across these terms, what exactly people are referring to if you are more inclined of using one term, that's totally fine with me. I'm just like trying to convey the point that.18:47Scale up really means the communication within the device. Scale out really means communication between two devices and then scale across really means some, some long distance connectivity. That can be INTERGB backend network, but people may overload these terms.19:05Right? Now, you know, when we talk about the storage network and different kind of network, there are some different requirements, out here. One is the highest speed connectivity and throughput, right? Of course, you know, you want to have 800 gig or 1600 gig connectivity.19:23The.19:43When you want to provide the GP to GPU communication because one of the unique requirements or the unique observation with these environments are is that when these GPUs communicate, they really communicate at line weight and they communicate at the same time. So that means there's not enough scope for.19:43Any lower speeds or lower throughput network design. So that's why the highest speed available and the highest throughput available in the industry that you should go after. 2nd requirement is about resilience because of course this is a massive investment and then anytime things go wrong, the the environment should be able to detect and recover from that failure.20:03The low latency and jitter function. Of course this goes back to the very 1st principles or the basics of network design, but in these cases in GPU to GP communication, one of the biggest phenomenon is about jitter, but really about the variance in latency. So I'll take the example like.20:24Let's say if you're running a job and if that job requires thousand RDMA operations, if one of the RDMA operation is delayed.20:43The entire job gets delayed, right? So essentially, in GPU clusters, the performance is dictated by the worst performing operation, right? In other words, nothing is complete until everything is complete. So for that matter, what we really recommend is have a network design.20:53Or have a system design which minimize the jitter, which minimize the variance in the delay, right? So consistency and predictability in the performance with low jitter and low latency is a, is a top requirement. Also lossless networking because RDM traffic is, is, you know, get runs into heavy.21:13Performance degradation when packets are lost. So one of the major reasons of packet loss in Ethernet ethernet network is congestion and to avoid packet loss due to network congestion, we can configure lossless networking using priority based flow control.21:33Of course, when we are talking about network, and high performance network, when all the end devices, all the GPUs are sending in traffic at line rate simultaneously, there would of course be some network congestion, so there must be congestion control mechanisms. All these networks must be secure to prevent unintended or even intention.21:47Attacks. Also, if there are any vulnerabilities found you should be able to mitigate against them.22:07The simplicity is very important because a lot of times people just look for performance, but remember once you buy these clusters or when you deploy these clusters, you have to operate them for a while, right? So it, they must be simple to operate and then the architecture must be open so that you can avoid any vendor lock in. Also because of the.22:14Open, you know, standardized way you can benefit from the latest innovation. And that also directly correlates with the co coexistence of different workloads. That means not just training, not just fine tuning, but your infrastructure may must be able to support even inferencing all kind of use cases. There are some customers who are thinking how.22:34How they deploy Nvidia GPUs and AMD G GPUs together, right? So that they keep their options open. And finally, you know, of course this is a massive investment, so customers often ask for reference architectures and validations so that they can put their product clusters to production, which is.22:54It's really a token generation stage much faster and minimize their effort in, you know, doing R and D or or optimizations, they expect the vendors like us in Cisco to provide those optimizations, right? So what I'll do, you know, as we progress further, I'll explain how exactly Cisco is addressing.23:14All these requirements, right? This is a very methodical approach that we have taken. We, we keep on getting the feedback from our customers. And then we go back to drawing board and think about how do we address these problems at a very fundamental level because you see some of these requirements you may think that, yeah, these requirements were there even some 2030 years back.23:34But, the scale of the problem is very different now because of the GPU to GPU communication, right? So for that, the 1st thing I'll mention is that how exactly we are offering the connectivity piece to it, which is using the switches. So, the 1st offering is on the 400 gig switches. These switches have been out.23:53In the industry for many years, so if for those of you who have these switches in your environment, these are based on the cloud scale silicon, the 9364 DGX two way.24:13And the, and the 9332 DGX to be switches they can be very well used. In fact, we have validated them and published white papers on the performance for GP to GP communication. However, I would say most newer deployments as of 2025 and 2026 have gone in production using the switches that provide 800.24:24Hundred gig ports. So these are the two models. These are based on the Cisco silicon 1G2 hundred silicon. Essentially in two RU form factor, they offer 64 ports of 800 gig. Or you can break these ports out to double the port count, which is hundred 28 ports of 400 gig or maybe two.24:4456 ports of 200 gig, right? So that's how it works. The maximum number of interfaces you can create on these switches is 512, of course, then the link speed would go to a hundred gig, but that's something unique functionality of the Cisco silicon one silicon.25:04And the 3rd offering that we have is, the two RU switch, very similar port, you know, front panel port design. It's a two are you switch offers 64 ports of 800 gig, but then this 9164E NS four O switch.25:20Is powered by the nvidia spectrum four silicon, right? Also this the NVIDIA spectrum four based switch, it has the front panel port with OSFP form factor, but let's say for some reason if you're looking for QSFPDD form factor, we offer one of the switch, as you can see the 9364E.25:37SG two Q switch, which is powered by Cisco silicon one.25:57Right? Now, all these switches have a, you know, can be used in front end networks, the back end network, the scale out networks. Also, they offer a consistent operational experience as I talked about in the very beginning of Nexus one, you can run NXOS on, on them, you can manage them using Nexus dashboard. The differences, which is one of a very common question.26:01Questions hey why or which one should you pick? The difference is.26:21If you are looking for Nvidia Cloud Partner compliance, if you are looking for Nvidia Cloud partner reference architecture, then the Nvidia spectrum four based switch, which is the 9164E NS four O switch, is compliant with the NCPRA.26:25Right? However, if you're not very strict on NCPRA, we in Cisco have published the Cisco reference architectures, which are technically exactly, you know, same of the NCP or the NVIDIA cloud partner reference architectures. So the rest of the switches that are powered by Cisco's.26:43Own silicon, maybe it's cloud scale or silicon one.27:02You you can use these switches as well, right? So that is the major difference whether you're looking for NCPRA compliance or not. Now, all these switches that you see right now, the 800 gig switches, they are available right now and we are shipping them and a lot of these are heavily deployed out there by customers.27:06Now going forward, I'd like to give you a small glimpse into what's coming in the future. We'll be going to the next iteration of the Cisco silicon 1G2 hundred silicon by productizing the switches based on the Cisco silicon 1G2 hundred X.27:25Remember the difference between G 200 and G 200 X is not much. Both of them offers 64 ports of 800 gig. The difference with G 200 X is it supports the LPO optics, the linear program, the linear pluggable optics.27:42Right, that leads to significant power saving. I'll talk about that in a few minutes. And also, you know, the G 200 X base switches offer latency improvements. So this is a 800 gig switch, will also productize.27:59Another 800 gig switch based on the Cisco silicon 1P2 hundred based silicon. Now, this switch has an has a high bandwidth memory that the chip can use. All right, so you can think of that as a deep buffer switch.28:17And this is primarily used for scale across scenarios because if you're sending traffic over long distance, you know, some deep buffer switches can absorb the congestion and avoid the packet loss leading to better performance, and this switch also offers line rate maxsec capability in all the ports which.28:33Of course, if let's say, you know, customers are sending traffic to a different geographical boundary, or which will cross the geographical boundary or their, their own buildings, they like to encrypt the traffic, so this switch will offer maxsec function.28:53And then after the 800 gig products, we also are productizing the 1600 gig, one point 60 port switches. So you see there are two variants to it. On the top you see the hundred percent liquid cool switch. They are 64 ports. Each port can operate at.29:091600 gig line rate and these are OSFP form factor.29:29And then we do realize that not all the data centers are ready for liquid cool yet, so we offer you know air cooled version of, of that switch too that you see at the bottom, right? And then just like what we did with G 200 and spectrum four silicon going forward as well.29:32We are prioritizing a 16 gig, 1600 gig switch using using the NVIDIA spectrum six silicon, right? And again, the difference between these switches will be same what I explained before. If you're looking for NCPRA compliance, you can use the, the switch that is powered by.29:52The nvidia spectrum silicon, otherwise you can use the switches that are powered by Cisco silicon one while all these switches have the same operating models so your operating operational experience will not change. You still can use NXOS, you can still nexus dashboard and provide consistent operational experience to you.30:13Your teams and your colleagues. Right now some of the points that I mentioned here is like, you know, you see some nuances around their different rack form factor Orb three rack MGX rack. I talk talked about LPO optics. I talk about OSFP or form factors, so what I'll do, I'll explain that real quick.30:32Of course there are a lot more details to it. So, you know, feel free if this is your area of interest, you know, if you do web search I'm sure you'll come across tons of informations on it. But, the quick point that I want to mention is that the the the 1600 gig switch that we're productizing, it will be liquid cool switch powered.30:52By Cisco silicon 1G2 hundred G 300.31:12This switch, we are productizing that is compliant with the Orb three racks. Essentially these are the racks which are, you know, standardized by the OCP community. As you can see, the height of the switch is slightly higher and the width is 21 ".31:15But if you compare it to the switch that is powered by the NVIDIA spectrum six silicon.31:35That switch is more compliant with the NGX, rack architecture, which is really, you know, optimized for nvidia ecosystem. The height of the switch, the, the 9164F switch is slightly less, and the width is less two.31:40Right? Again, just to make these switches compliant with the kind of racks available out there. Otherwise you can see there is a in the very center of these switches, there's a, you know, bus bar. So essentially these switches work very different from the air cool switches because air cool switches have fans on their rear.31:58And they also have power supplies where you go and connect a power card.32:18In these switches, there are NO fans, as you can see, and also, there's nothing to connect at the rear of the switch. The way these switches that get installed that you insert them in the rack, and when you insert, the bus bar gets connected in the power bus bar of the rack. So that's how the switches get powered. There's NO capability.32:22To connect and also you see the liquid, inlet and outlet. So even those connectors get connected in the rack and they start you know getting the liquid inlet and outlet for liquid cooling.32:42Right, so that is a different operational experience with the switches as compared to the, the air cooled switches. The other difference that I mentioned is about the OSFP transceiver. Remember this is not OSPF, which is the name of the routing protocol, the open shortest part 1st. I know the names are very confusing.32:56So I like to clarify this is not oh open shortest part 1st. This is Okta.33:16Small form factor pluggable, OSFP it's a kind of a transceiver, but then with these transceivers, be aware there are multiple variants of this transceiver. One is the integrated heat sink. So if you zoom into this one, you see there's a heat sink on top of this transceiver. So that's why it's called.33:22That has integrated heat sink or IHS. The other variant of OSFP transceiver is you see there's NO heat sink here. So how exactly the heat gets dissipated? Essentially the heat sink is riding on the device in which you plug this transceiver. So that device could be a liquid cool switch.33:42Or it can be neck, right? So this is another difference you should be aware because if you try to order, let's say, a riding heat sink transceiver, maybe you ordered it for and if you try to insert that transceiver in an air cooled switch.34:02It will not work, right? So be aware all order the right transceiver. Both of them are OSFP transceivers, but as I said, one has the integrated heat sink. The other one does not have an integrated heat sink. It expects a riding heat sink, and because of that there's the the height of the writing heat sink.34:18Transceiver is slightly less, right? So that's another difference. The 3rd difference what what I mentioned is about we are productizing a switch that will support the LPO, the, the linear pluggable optics.34:38The difference is that, the most important difference that you should know is that LPO consumes less power and how it consumes less power. Essentially, you know, all the most of the transceivers or the most of the traditional transceivers that we are aware, we call them as FRO.34:52Essentially stands for fully retimed optics. It has a DSP inside it, so it does encoding decoding, and then it converts the signal to optical light to send it on the cable. With LPO, the encoding decoding happens within the silicon one you know from Cisco and.35:09And because of that, it, you know, there's less work needed by the transceiver and therefore it consumes less power.35:28Right? And you see the power saving, which is a significant number, right? And if you multiply that by 64, because there's 64 of them that goes within a switch. And if you also think that if these transceivers emit less heat, the fans of the switch have to run, you know, a little bit less less speed. So that also leads to some.35:35Power saving. So overall, this is a very major, I would say advancements that's happening and these switches will be soon available from Cisco as you see details on this, this slide.35:55Right? So you see there are multiple type of OSFP transceivers available from Cisco. The one that have the L prefix before the 800 gig, these are the LPO optics and they will be supported with the, the.36:07G 200 X, the Cisco silicon 1G2 hundred X base switches. The product name would be the 9364E as G2xo, right? So again, if you don't remember the product names I know there's plenty of them. What you can do of course refer to the slides, but one another.36:22The favorite location that I personally refer to because I mean it's impossible for a single human. At least it's impossible for me to remember all of this. So what I typically do is like I go to this link, the TMG matrix.cisco.com and I just like search with some basic letters and then it has a, you know, auto complete list and it provides.36:42So all the connectivity matrix, what transceiver is supported on what switch, what transceiver can connect to, what other kind of transceiver on the other end, what cable type it needs and all, all the good details.37:02Right? Now while we are on the screen, I'll also mention another thing that you need to watch out for on the OSFP transceiver is that if you notice the 1st row and 2nd row, they have a very similar looking optics type transceiver types. The only difference is, the top one says DR eight.37:15And 2nd one says DR eight P The difference is that.37:35The one without P has two ports or two links. This is where you can connect actually two MPO cables within the single transceiver. So if this is an 800 gig port, each cable can carry 400 gig or each cable can carry 200 gig, right? If you configure it like that.37:39But However, the transceiver that has the suffix of P, it only consumes a single MPO cable, alright? So be aware of these differences too, right? So overall or when you're working with OSFP transceivers, be aware there's an integrated heat sink.37:59There's a riding heat sink, then there are variants where you can connect two cables to a single transceiver or you can connect only one cable to a transceiver, and then there's something called a LPO version to it, and there are more nuances, right? More details to it. And all these details you can find on the TMG matrix.cisco.com.38:18Okay, so going further, what, what I'll do, I'll talk about some of the network designs for INTERGP backend network alright? So when we talk about INTERGP back end network, there are two more significant design decisions that we do, alright? The 1st one is the non blocking network.38:41Design what it really means is that if you're connecting multiple GPU servers.39:01With each other. Essentially for every leaf switch, assuming it has 64 interfaces on it, what you'll do, you'll take 60 half of the 64 interfaces, which is 32 in this case, you connect them to the servers, and then rest of the 32 ports will connect to the spines, right? So any amount of traffic that.39:06Comes into the network is the same amount of capacity that network has to send the traffic out. That's what makes it non blocking. Why do we create a non blocking network?39:26The reason is because all these GPUs, when they communicate, they really communicate at line rate in ingress and egress directions simultaneously. And of course when the GPUs are sending in traffic and receiving traffic at line rate, there's NO scope for any over subscription.39:35So that's why we create non blocking networks. The other design principle that you probably already noticed that there's a, you know, weird way or a different way of connecting the cables. This is called a rail optimized design. In rail optimized design essentially what we do, we.39:51Take the 1st GPU of all the servers and connect to the same leaf switch. Then we take 2nd GPU of all the servers connected to 2nd leaf switch and so on. So if there are eight GPUs, we'll look at eight, you know, leaf switches and then each leaf switch connect the exact same GPU ID on all these servers.40:09The reason we do that is when these GPUs communicate, there's something called a collective communication using libraries which are written by NVIDIA, it's called the Nvidia collective communication library. In short, we call it as nickel. From AMD there's a.40:29Rockem collective communication library. It's called Rickle.40:49And then from Intel, there's a Habana collective communication library. You can guess its name. What do they call? Of course, they call it as hickle. So nickel, rickle, hickle, these are all collective communication libraries. When they talk or when these communica you know collective communication libraries, they communicate.40:54Or when they make the communication work among the GPUs, what they do is only the same GPU ID communicate with each other. And when that happens, if you connect all these same GPU IDs to the same leaf switch.41:14The traffic actually benefits from the single hop forwarding of the leaf switches. That also means traffic does not go to the spines. So essentially you are looking at two benefits. Number one, you, you reduce the network latency or the network delay to the GPU to the GPU communication.41:31And of course you also reduce traffic going to the network and if the network has less traffic, of course, it won't run into the problems that you know some of the challenges that I talked about like congestion, right? Now, one of the questions that keeps come, ok, if the only G GPU one of node one is talking to GPU one of node 32.41:51What happens if GPU one wants to talk to GPU two in the other node? In that case, 1st the internal communication happens from this one GPU to the same GPU ID within the same node using the scale up network, which in Nvidia's case is NV switch or in amd's case it's it's infinity fabric.42:11And then, the traffic remains within the same.42:30GPU IDs, or another term we use for this communication is rails. So there are eight GPUs, there are eight rails, and then in this design at least we are dedicating one leaf switch per rails. Remember it's not mandatory to dedicate one leaf switch per rail, but.42:35In this design we have dedicated leaf switch per rail.42:54Now, what I mentioned before about traffic benefiting from the single hop forwarding of a leaf switch and traffic not going to the spines applies only when, you know, only a limited number of servers because.43:00You know, based on the number of interfaces you have on a switch, you can only connect so many GPU nodes to the same set of switches. So in this example, assuming there are 64 interfaces on a switch, assuming non blocking design, I can only connect 32.43:17Servers to eight leaf switches, we call them as a scalable unit. If you want to create a larger GPU cluster, let's say 512 GPU cluster, you must have another scalable unit and then connect the leaf and spine switches with each other. So that will create a larger GPU cluster.43:35But now if you're running a job, so now the traffic cannot benefit, as you can see, I've drawn the traffic path, traffic cannot just forward from the leaf switch. The traffic has go has to go to the leaf once, go to the spines and then come back and then, you know, again, go to one of the nodes in 2nd.43:54Scalable unit.44:14So for this challenge, of course, you know, this is how, by the way, you know, you can keep on extending this design and and connect thousands of GPUs to, to, you know, your network. If the two tier network as in leaf and spines are not enough, there are deployments that have even introduced super spine.44:15Spines in the network, you know, some people call them as a three tier network, right? And if you go to super spines, you can actually connect half a million GPUs there. I mean, you can run the math if you, if not feel free to reach out. You know, these are.44:35That's how some of the hyperscalers large environments have connected. I'll say it again in a three tier network, you can connect all the way to half a million GPUs if you keep on extending this design. But then, you know, a two tier network or a three tier network, you know, people, there's benefit, there's merit in keeping the number of tiers in the network to as low.44:47As possible. So for that, there's something happening, you know, we call it as a, a multi planner design evolution. So to explain that at the very fundamental level, I've taken the connectivity example, right? So, I mean, what you see here on the screen.45:07It's a same UCS ATTA server. It has nvidia B 300 GPUs and every GPU has a dedicated connected. Now in a single plane design, the design that I just explained on the previous slide, what you would do, you would take the transceiver, there would be, you know, this dual interface or dual port.45:24But, you know, transceiver, so you will connect both the transceivers to the same switch and then, you know.45:44Combining both the cables, this link will operate at 800 gig. So this is one way. The other way you can do actually send each cable to two different switches, right? Assuming these two different switches are part of two different networks.45:52These two different networks, we call them as a different planes, network planes, right? Network one, network two or you can call this plane one or plane two, right? So this is the very fundamental level of how exactly a multi planner design evolution is happening.46:09Now how exactly it serves the purpose or solves the problem that I just talked about is that remember, this is where I started from, right? Like if you have 02:56 GPUs cluster, you create a scalable unit, you know, scale this cluster to 512 GPUs, and then you can connect two scalable units to each other, and now traffic goes to the.46:27Spines. We already talked about this. But if you connect the same network using a multi planet design or a two plane design, now instead of sending the full 800 gig capacity to the same network, we are splitting 400 gig to plane one and 2nd 400 gig to plane two.46:45And now the design that you see on the top, that design benefits from the single hop forwarding among all the 512 GPUs of that scalable unit. So that is the benefit. The other benefit is.47:05Instead of a single network, you have two networks. So what happens if some of the, something goes wrong in one of the networks? You actually benefit from, you know, the availability of 2nd plane. So that would keep, you know, your operations.47:20Ongoing. Remember when I talked about resilience, you know, as a one of the requirements of AI networks, this is one of the ways of bringing resilience in your environment, which is not really put a single network connecting all the GPUs or rather have multiple independent networks connecting the GPUs.47:38And which also gives you high availability. Okay, so these are some of the details about multiplane design. I'll let you explore these. I already talked about the scale benefit. Look, one of the points that you need to realize is that.47:56If you connect a single plane network, there are only limited number of interfaces on a switch. But if you create multiple planes, each plane operating at slower speeds as in instead of operating a switchboard or a link at 800 gig.48:14If you operate the link at 400 gig, the number of interfaces on the switch doubles, right? If you operate the speed at 200 gig, now the number of switches, number of interfaces on the same switch.48:31Becomes four times, so you can have 02:56 interfaces operating at 200 gig. And because you have more interfaces on a switch, you can connect more devices to it, right? So if you run the math with the two plane network, you can connect 8000 GPUs at 400 gig.48:47In a two tier network, alright, with the leaf spine without bringing super spines. Of course, with the super spine as I said, you can actually go all the way to half a million. But this is just one, you know, idea that I want to bring it to you, of course, you know, if this is an area of your interests, feel free to draw some network design, run some math.49:03Or feel free to reach out happy to discuss this. The other benefit I already talked about the scale or the high availability benefits.49:23The traffic from the Nick is load balanced across both the planes, right? So the one GPU, still sees the other GPU and the two network or the multiple networks. That part is taken care of by the, by the nick. Both the planes are identical, right? And final point, it's not.49:31Mandatory if you have smaller environments, it is not mandatory to have multiple planes. Things will equally work well with a single plane as well. But again, you know, you should.49:50At least consider the benefit of the multi plane designs. Okay, in the beginning, I talked about the examples of the NVL 72 GB 300 rack scale systems. So in Cisco what we have done, we have published reference designs on these systems as well.50:00So there are, if you do a web search, you'll come across the reference designs. Essentially the way we connect these environments is that we create a scalable unit of two such racks, and if you recall, I mentioned that if you want to connect these two systems on a single rack, there are hundred and 44 ports of 400 gig, right? So two.50:19You multiply that count by two. And then you again you follow the same rail optimized design. There are four GPUs in a, in a compute node, remember? And then there are 18 such compute nodes in a single rack, but because there are four.50:39GPUs on a single node that translates to four rails and that's why we create, you know, a diagram or a network design like the way you see here. Of course, this is only 144 GPUs if you want to have more GPUs connected with each other.50:54You can actually scale this design all the way to 4600 GPUs in a two tier network. Of course, if you wanna have more GPUs connected using NVS 70 twos, you're talking about like even either going to a four plane design. That's one way.51:10Or you're talking about like going to a super spine, a three tier network design. That's another way. Of course, we recommend to go multi plane design, but remember these are all depends on what firmware level you are running and what kind of capabilities are available to our system.51:26Right, but refer to the reference architectures that we have published because we we backed that up and we have validated those environments while working with Nvidia on, on these network designs. Okay, but look, the other extreme is not everybody may have the resources to deploy.51:42The rack scale system, so a lot of time I get asked, hey, you know what what happens if I just want to deploy 64 GPUs? Do I still need those dedicated switches per rail? Well, the answer is NO. As you can see the the network design on the left, you can actually merge.52:01Traffic of multiple rails on the same switches. So in this design traffic of four rails go on the same switch. The design is still rail optimized and the design is still non blocking, so the basic principles apply, but we are not dedicating one switch per rail because it's not needed.52:19For 64 GPUs.52:38For hundred 28 GPUs, now you need few more switches, but again, we are not dedicating one switch per rail. We are using one switch for two rails, and then, you know, again, we follow the same design principles of, of non blocking network design with the rail optimized network connectivity. And of course even these designs that you.52:39You see on the screen right now, they are part of the Cisco reference architectures that we have published. So if you do a web search, you'll come across all these reference architectures.52:59So I hope it conveys the point that regardless of what scale you are planning to deploy, maybe a smaller GPU clusters of 64 GPUs or even larger GPU clusters of hundreds of thousands of GPUs. We have a reference architecture.53:08Our design available for a successful deployment. Okay, so that was about the back end network. I'll quickly reference about the front end and storage network. Now look the front end and the storage network design, they're all kind of variants on it, right?53:23And based on the customer requirements I've created network designs which are, you know, non oversubscribed. Some of the customers asked for two times over subscription, some customers asked for four times over subscription. The design that you see on the screen, this is a design I created for a customer who asked for.53:411 gb/s per GPU throughput, right? So I, you know, this is I think some of the reference architectures that we have published, they also follow a similar design, right? The other factor you need to account for, the front end connectivity is considering your front end network is.53:59With your storage network is that what is the connectivity of your storage nodes? Because some of them, I think most of them connect at 200 gig today.54:20But they are very fast, they're moving to 400 gigs, so based on that, your answer would be different. Also, what kind of NICs that you have for front end connectivity? Some of the NICs offer two, 200 gig interfaces while some other NICs offer a single 400 gig nick. So accordingly things might be changing.54:27But again, a lot of these details are mentioned in the Cisco reference architecture designs that I keep on mentioning if you do a web search, whatever search engine you use, use that and you will come across all these links. Otherwise, you know, we can, you know, reach out and we'll be very happy to share those reference architectures with you.54:46Okay, so look, what I like to do, I think we have reached to a milestone. We have talked about a lot of, you know, requirements. We have talked about the network designs, the backend network design, the front end network design. What I like to do is like, you know, take a pause here because before we start the next topic, we probably will run out of time today.55:05Okay, so I'll start the next topic next week, which is the part two of the series. But by the time you you know let us know, maybe you can post your, you know, questions or feedback in the chat or Q and A Let us know what topics that you would like me to cover that you have not seen me covering.55:25What I plan to do next week, explain a bit more about how the operations work, how this connectivity is really configured. So I'll go into the configuration operations piece. I'll also show you the demos of how you can use Nexus dashboard to operate these environments, how you can use the cloud managed.55:45Access hyper fabric from Cisco to, you know, operate these environments, and, you know, to achieve a successful deployment.56:05Right, but again, look, there's a lot to be discussed about these topic as I said in the beginning, the I have content for two days somehow I'm trying to consolidate within 2 h, so of course there's NO way I can cover everything, but.56:14If you have any specific areas you'd like to focus on, I'll be very happy to cover that, right? So for that, I'd like to pause today and hand it back to Shannon. Okay, thank you. That was great. There's just so much to learn and for you to be able to squeeze.56:30Piece what you can in this hour. Really appreciated.56:50Next week's session is gonna be is going to continue the education please feel free to reach out we'll try to keep you up to date and we will be lining up more topics in the coming months.56:54So we would love to have your input on what would be beneficial to you. With that, we'll end the session in that session. Have a wonderful day and we hope to see you next week.57:14