Deep Dive Data Center Compute for AI Workloads

Hello, and welcome to today's Cisco Expert insights series.00:10Where we are going to take a deep dive into data center compute for AI workloads. Now I am Joe Francis. I'll be the moderator for today's session and joining us we will have.00:16Two csss, Mohammed Khan, and Alan Clark, that will guide us through today's presentations and demonstrations. Now here's the flow of our session today.00:32We're going to start with a, explore with us how to set up an AI pod, and we'll transition over to show me how for some live demonstrations, as well as help me succeed where we'll go through some AI pod use cases.00:48Now the you'll see this slide a couple of times during our presentation, and each time you'll notice that one of the three sections is highlighted brighter than the others. That will give you an indicator of where we are in the flow of this session.01:05Now at this point, please join us@Slido.com. You can use the meeting code compute-AI and the passcode compute-AI, and we'll be handling our polls today through that Slido.01:24Now we've got our 1st question that we are curious about for Slido. If you can join on Slido.com, use that meeting and passcode, and what you should see is our 1st poll of where you're joining us from today.01:45If you can go in and just free hand text type city state zip code, something indicative of where you are joining us from today. That would help us out so we get to know our audience better.02:02Once we have this polls start to populate, I see results coming now. We've got Kansas City. We're gonna go to our second poll question. So keep typing with the 1st and we'll go to our second one. 2nd one's gonna be which best describes.02:22Your current AI compute setup. Now this is gonna have multiple potential answers there, and you're gonna select the one that best describes your current setup.02:40And as far as where folks are joining us from today.03:05We'll take a quick look and there we should be able to see that we've got several states reporting in. I see Texas a couple of times there.03:10A couple of cities within Texas, Dallas and Houston, that's good. We've got our friends in San Francisco. I noticed that we've got a couple of provinces in Canada there, Bridge Columbia I believe, as well as.03:27City of Montreal. We see someone from Belgium there, a good international audience for us today. Thank you very much for that. Now our second poll should be available to you as far as which best describes your current AI compute setup. Again, this is to.03:42Help our CSS team members here during today's presentation to get to know our audience better. It'll help us know what to focus on and better gauge the audience that we're gathering for these sessions. So as our second poll begins to populate.04:01Let me share that with you and we'll take a look and see the results that we have. Currently looks like most folks are still evaluating their options, second place we're not using GPUs today so far.04:22A few handful of folks planning to use GPUs within the next six to twelve months. So it looks like we've got some good voting there and looks like most folks are in the boat of either evaluating options or not using GPUs today.04:42Thank you again for that information. That will help us know what the target today's session for. And as I mentioned, you're gonna see this slide several times and right now this is gonna indicate that we are on the explore with us. How do I set up an AI pod?04:59And at this point, I'm gonna hand it over to my good friend Mohammed Khan. Take it away Mohammed. Joe, can you hear me ok? Yes, I can. Thank you. Thank you for Joe for your warm introduction.05:20To us and also running the slido. And I do wanna thank all the attendees joined this session and taking time in your schedule, really appreciate it.05:40Today we're going to have an exciting session on AI pods introduction and also hands on demo from my colleague Alan Clark after my presentation. So let me start with the AI pod introduction.05:51The Cisco AI AI part, the AI parts are the foundation of the Cisco secured AI factory providing a full stack modular infrastructure platform for enterprise AI, so developed in partnership with Nvidia and leading storage vendors.06:09AI parts combined with Cisco UCS compute, Cisco Nexus Networking, advanced GPUs, and robust software to support every stage of the AI life cycle from training and fine tuning to high throughput inferencing.06:29AI ports addresses the key challenges facing enterprise, like scaling AI workload ensuring data security and compliance and simplifying infrastructure management. The architecture features.06:48Like a high bandwidth, looseless networking and based in class GPU servers and pre validated designs to reduce the setup time up to 50 % and accelerate time to value.07:06Integrated with Cisco Intersight and Nexus dashboard, AI parts deliver unified management, automation, and operational visibility. Flexible deployment options, it supports on premise and premises and the hybrid and cloud integration.07:22Empowering organization to innovate with AI confidently, securely and at a scale, whether optimizing large language model, deploying real time analytics at the edge or supporting multi tier GPU cloud environment, Cisco AI parts.07:42Enable IT and AI teams to move from pilot to production with ease.08:02What I just talked about, I wanna summarize pretty much in this slide why Cisco AI pods. So 1st of all, security 1st by design so built with enterprise grade security to protect data, models and workloads. So you can adopt AI safely and configure.08:13Secondly, high performance AI infrastructure. Optimized for demanding AI workloads, enabling efficient model training, customization and inferencing at scale. And lastly, pre validated.08:34And ready to deploy, say fully pre validated AI stack that dramatically deduces a setup times and simplifies deployment so you can get the value faster. So in summary, it's a Cisco AI port from the AI infrastructure building blocks for secure AI factory in.08:52Includes, I'll tell you, we're gonna show you in the demo section as well. It includes NVIDIA, AI software, Kubernetes platform, Cisco AI networking, Cisco compute with nvidia accelerated computing and partner stories like data apps, Nutanix, past, etc.09:12To try.09:32Let's talk about this slide. Cisco AI port provides a scalable all in one architecture designed to simplify the entire AI infrastructure journey from training and optimization to inference.09:38The comprehensive solution integrates advanced networking, accelerated compute and storage to help you efficiently build tune and deploy any AI workload. So here like training optimization and inference, so optimization in the context of AI and machine learning refers here in this particular.09:56Case, the process of refining and improving air models or systems to achieve better performance accuracy efficiency and reasoning resource utilization.10:15And then this is interesting slide. I would like to talk more about it. And if you look at this slide in the middle, right, in the middle, the central block, there's a rectangular block here in blue color. So in the central block, this is the core Cisco airport capabilities, the heart of offering.10:31Cisco Airports provide a robust foundation for AI workload covering in this case. Operations, automation, AI software, Kubernetes, accelerated compute, high performance networking. So in terms to operation, we leverage Cisco Intersight and next to Nexus dashboard.10:52Support for unified management and monitoring. Regarding automation, it incorporates corporates tools for streamlined deployment and orchestrations such as you can use ansible, Terraform, PowerShell or Python. And AI software.11:12It's featured the NVIDIA and Coda for accelerated AI development and and deployment. Kubernetes is supporting container orchestration with openshift, Ubuntu and Rancher.11:31Regarding the accelerated computing of Cisco UCS for high performance processing. And lastly, the high performance networking utilizing Cisco Nexus for efficient data transfer.11:48This is the central, like a block I just talked about. Now extending to the storage platform ecosystem, there will be the bottom block. So this, the solution in this case it supports a diverse range of storage platform to meet the varied data requirement. E.g., the the.12:03Platform talking about, the 3rd party past storage, net apps, pure storage, Nutanix storage, and Hitachi as well.12:23And then if you look at the left side, the left block of this slide, the Cisco AI ports are designed for seamless integration and extension into various critical enterprise functions, Cisco security, e.g., enhancing AI.12:33With capabilities like aid defense, hyperscale and firewall Nexus switch, like a smart switch. And then if you look at this like right below security of observability, so integrating with Splunk observability cloud and supporting open telemetry extension for.12:51For insight. And then the last one from the left bottom left block, workload management and ops.13:11So here that would be the it facilitates efficient resource management and solutions such as isovalent and then Splunk as well. And then lastly I'm gonna talk about BYO AI tool on the right column.13:20So what that means, so build like like compute, meaning that bring your own AI tool and framework including like Q flow and the Jupyter et cetera.13:39So you can definitely read up more about this slide in our Cisco documentation we have if you are interested to learn more. Alright, now we are moving on to the Cisco compute AI portfolio. What do you offer?13:56As part of the Cisco Compute AI portfolio, Cisco offers two primary deployment options to support AI workload. Number one, the 1st one is our UCS dense GPU server portfolio on the left side, which includes C 8080AMH server C.14:17C 885AM8 server and C 845AM8 server.14:37These platforms are purpose built for high performance AI and HPC workloads, offering large scale GPU acceleration, high memory bandwidth and high throughput networking. So I'm gonna a little bit deep dive into this particular specific hardware I just talked.14:45About 1st of all C ADD AM8. So this.15:05Hardware or platform built on the NVIDIA ace GX platform and this is ideal for LLM inference advanced AI training and large scale data processing when maximum GPU density is required.15:11And 2nd one I talk about C885AMH server, so also based on the NVIDIA eight GX platform and it powered by dual AMD, epic CPUs.15:29It also well suited for high throughput AI workload and scal scalable and clustered AI environment. And the last the last like a profile the portfolio talked about, which is C 08:45 AMA.15:46It is based on the NVIDIA MGX modular design and this is most flexible AI server in the portfolio because it supports two to eight GPUs across a broad range of NVIDIA and AMD accelerators, making it ideal for AI.16:06Use cases, pretty much diverse AI use cases, evolving workloads and custom configurations. And on the right side of this slide, you have UCSX like a full stack AI with computer networking.16:26And this option actually like server and it provides a modular scalable and flexible compute that can support a broad spectrum of AI analytics. And for today's session, we'll be.16:41Deep diving on on this particular platform, which is Cisco UCSX platform.17:01All right, so now we have a demo. I wanted to share. Let me share my screen for that, screen two.17:14Let me play a video and I'm gonna pause the video.17:36And then I'll talk about different, component of the AI pod and then different modules front and back of the chassis Okay let me pause.17:40Here, this is the physical chassis and it's actually UCSX 9508 modular chassis with a seven RU and eight flexible slots. You see the numbering 128 in the bottom.17:59And what happened is liquid cooling ready, total 06:20 800 W power supply, 54 power distribution, and 400 404 400, like a dual rotational fans.18:17Total four fans each 400, do all traditional fans, and then we got here.18:37Like as I mentioned like six PSCO and it's providing 54 v of power to the chassis with N or N plus one and n plus n redundancy. The higher voltage allows efficient power delivery with less cooling wire wiring needed and reduce power loss. And then.18:44So chassis, by the way, I'm gonna show you from inter site how to configure the chassis and you'll see the similar like a view from inter site as well. I'll walk you through more in depth. I just want to give you like overview how the harder looks like if you open these nodes.19:04And what's inside all the, like some intro about this, the physical hardware. So if you look at UCS X 210 CM seven compute node on the left far left on slot one. So this is high density enterprise plus compute module.19:24Designed for the Cisco UCSX modular system of course it delivers strong performance flexible configuration option, and support for a wide suitable range of workload in data center, cloud, and hybrid environment.19:44It also suitable for virtualization, database analytics as well plus AI ML and accelerated workload. And after a compute node, we have a PCI node. So this this the this is X 440P, the model number I can show you in the in in.20:01In the interest from inter site that the the serial number and then the model is like X 440P PCI node. And this node actually provide two to four additional PCI slots to a compute node in the adjacent slot. These compute nodes are insert.20:21Into the art slot is for number one, but a PCI nodes into the event slots. Total four PCI nodes can be added in the chassis.20:41Yeah Moving on, the front connectivity. If you look at the chassis and then on top of that the chassis have two fabric interconnect, the yellow cable connected, these are 1L2 cables connected, and this cable are for.20:50And high availability for fabric interconnect.21:10This is actually the the back of the the fabric interconnect and let's play video. I just told you about this L one and L two cable the connected back to back.21:14Between the fabric interconnect FIA and FIB. And then here in this is the back of the chassis and right here, let me play a little bit more. So on top we have we have Fabric Interconnect.21:31Okay, and then the 1st two slides we call IFM, IFM module, and then the we got four fans here, 1234, and then right below four fans, we got XFM XFM module.21:50So the, what IFM does in this case, it's a like a two ethernet module modular fabric, which connect chassis to upstream network. It it has like hundred gig fabric or it could be 25 gig as well, but total if you configure the port channel it will be like.22:08A 200 gig fabric going to upstream.22:28And the FI could be four G and also the one we have in the demo is the latest one. I'll show you later on. And let's talk about the XFM. What is X XFM in this case? It's an X fabric module. It provide the connectivity between compute and PCI node.22:32So there is gonna connection between the inside the chassis that connect the compute node and the PCI node. All right, let's move on with the video. So I already mentioned all the details of.22:51From the IFM it goes to the Fabric Interconnect over here.23:11And then from fabric interconnect, we have a SFP interface or port that goes to the nexus, two Nexus switches upstream networking purpose. Now we're opening up the IFM.23:16Module here, you see like a three fans, the green color, 1st, second, and 3rd fans right here, and on the front orthony direct connector to all the nodes.23:33Now now we're opening up the X fabric module. It's pretty similar looking if you see, it has like three fan as well and our second 3rd, and then on the front we have our terminal.23:49Connector to interconnect the nodes. Now we are opening up the compute node. So compute node, we got Intel CPU.24:07And then you got memories for the CPU and this is adapter, and then this is the adapter slot, that's the one connect between node and the PCI node. The NO looking at the PCI node, let's open the.24:24See what's inside. So we got two GPUs here, GPU one and GPU two. This is the cabling that goes back.24:45And then, at the back we have the card that connect to the nodes to the XFM XFM. And keep in mind.25:04That this PCI notes support two different riger cards. So riger A riser B So riger A is two X dual with CPU and riger B is four X single with GPUs. So GPUs yes. So in this case the one we just showed you, it's.25:21We're using A because these are like a dual width of CPU and GPU we're easy.25:41Yeah, that's the brief intro about how the physical harder looks like with the context of this physical overview, I do wanna share from inter site perspective how to configure, I'll take like next five, 6 min from for this matter. Then I'm gonna.25:51Hand the ball back to our demo specialist.26:11But let me close this one and then bring back the next, ok, right here. So this is Cisco inter site and I logged in as a, as let's see, as administrator.26:14I think as myself as well I have the right so here what happened if we ne we need to configure like profiles, right? So chassis profile and then server profile.26:32And the domain profile. So regarding the chassis profile here, this is the chassis. And if you look at the chassis if I hover over my mouse, I can see the model number.26:50Which is two the compute node 200 and CM seven and right next to on the slot two I got PCI node two. Also it the model shows up UCS X 440P And I have another compute node and then PCI node. And these are like, you know, the.27:05For the PSU total six PSU, but two of them don't have the AC input, that's why it's showing yellow. But if you guys see the 4th slots we got for the PSU.27:25And if if you look at the rear view, as I mentioned showed you in the video, so in right this is the fabric interconnect module in this case, the one and two, and then these are four fans, and this is XFM.27:41Or terminal connector for the connectivity between nodes. And the profile was if I quickly go over the chassis, the chassis profile. So here look going to the chassis and the UCS chassis profile, so you got power profile in.28:00This case we can view what profile has been configured in this case is N plus one.28:20In this configuration, and after let me go over to the domain profile. Domain profile is nothing but configuration for fabric interconnect. Let me filter out the information what I'm looking for.28:27And in this case I'm gonna go to my preferred org, which is AI workload. So these are the two fi fabric interconnect A and B click on the 1st one the model is 9S9910 800 gig.28:42And I have all the information on the left, the serial number management IP address and what not the varsion it's running, but I'm gonna be focusing on the domain profile in this case. So domain profile how how do you, did you configure domain profiles? So these are the port configuration and for that.29:00Already got this policy configured if I see what's inside of the port policy.29:20You see that in this case we configured the port one one not 114 actually 143 and 14I believe lemme double check. Yeah, port one/three.29:26It's Ethernet uplink port channel and also one for part of the port channel. So this is the configuration for the port. And also we have the configuration for the VLAN, you can view the configuration.29:45Any VLAN you have configured as part of your need, you can view from here and then you have the VLAN configuration for fabric A and fabric interconnect B for both FI and then it says domain configuration such as NTP and system QoS.30:03So looking at the NTB, we have NTB server detail, also system QoS, we got the configuration detail as well. So here, this is the basic domain profile and now going to server profile.30:22And here, let me also filter out the information I'm looking for the servers in the AI part, AI workload. These are five servers we're using for this demo. Alan will be working on this demo, on this particular AI part, so let me bring click on the server.30:38So, you see, so this is the if you click on the server one, it see the green color, it shows up this is the server we're we're looking at right now. And.30:58You can see the top view and then from view as well, and the server profile in particular so here all the policy like a boot policy, a biased policy, the media policy, IMC access policy, storage.31:08This policy land connectivity policy, these are all the policies.31:28We have to configure proactively and then attach this policy to the server profile and then you deploy the profile. So we have done the all the work before. Also, if you configure all these profile proactively in case your server arrived later on, you can actually just add add the server to be configured deployed.31:32Right away so without wasting any time.31:52Yeah, so, this is the profile. Yes, I just I talked about server profile chassis profile and public interconnect the domain profile, and this is if you are familiar with Cisco IMM configuration, it's pretty much same thing. You you.31:54So you can really get into it quickly if you know Cisco IMM configuration best practices.32:14And you can follow up we have a wide range of documentation in Cisco website, you can definitely look into that. But I'm gonna quickly turn back to the to the presentation. So let me go back to the presentation quickly here.32:21Let me stop sharing 1st.32:51All right, so here in in our upcomings like a delivery time like next hour, almost an hour, we'll be talking about OpenShift setup.32:53And then also nvidia microservice setup Alan will be going over all the detail. I covered UCS infra setup, create and deploy domain profile and server profile, Testi's profile, all the good stuff.33:13And so just to before I hand over to Alan, I wanna just quickly wanted to show you that domain profile has all different policies. You don't need all the policies so in our configuration in my interested.33:28And the view we see saw a few of them, not all of them, but this is all the policy you can configure for domain profile. E.g., the domain configuration, you have switch control policy, NTP policy, network connectivity policy, QS policy, and then also for the regarding the port configuration, you got different policies.33:44VLAN VLAN policy multicast policy and server profile as well. Just to wanna quickly go over, we got compute configuration, management configuration, stress configuration, network configuration, and we have we got different policies for each configuration.34:04And you can configure as part of your need. And lastly, the chassis profile. So we, I think you know you can configure we IMC access policy part policy, SNMP policy, thermal policy and IP full policy under IMC access policy. So in our industry demo.34:22I only showed you the power policy because that's only needed but others of course you can customize as your need, see what you needed and you can go from there. So, well, this is the brief, you know, intro about the profile and how to configure. With that, I'm gonna be passing the ball back to Joe for the slider and then Alan will take over from here too.34:43Thank you so much. Excellent, great. Thank you for that wonderful presentation and demonstration Mohammed. We'll bring our audience to our next poll question that we handle through Slido. You should see that either@Slido.com or see it as a pop out.35:02In the window of your Webex session, and we are voting right now on what's your biggest AI compute challenge is at the current moment. So we've got our audience.35:21Answering that particular question that's gonna help our CSS engineers today better guide the conversation and their presentation and demonstration based on the findings here of where it's gonna be most useful to you.35:38So we've got a couple of more votes rolling in. Let me show this as we speak so we can look at it together so what we've got is.35:56At the top of the list, security and compliance, 43 % of the respondents indicate that that's their biggest challenge for today beneath that cost along with power and cooling.36:16So we can see where most of our audience is. I'm sure our CSS engineers are gonna have some great information that are tuned to those challenges coming up in the rest of our presentation and demonstration, and with that I can turn it back over to our.36:30Next CSS engineer for our next session on show me how. So Alan I'm gonna pass the ball over to you, let you take it away, thank you Joe. Give me 1 s while you get my screen shared out alright.36:49Share it out here. So I'm gonna show everybody how, this is our next step. We're gonna look at kind of deploying to the infrastructure, deployment of OpenShift, installation of some nvidia microservices, and then some actual demos of things you can do and how, how we would.37:09Use this once we kind of have these deployed. What we're not gonna do is a full walkthrough of deploying UCS. We we've covered that in previous expert insight series. Some of you are interested, we can get you the videos and links. So.37:29What we're really gonna start with is how are we gonna deploy OpenShift and and the Nvidia services? 1st we need to look through our our basic requirements for openshift. You need an account at Red Hat and then we're gonna need to need to get your account set up for Nvidia.37:46For these of you are gonna be your two links, console.red hat.com ngc.Nvidia.com set up your logins generally speaking, you want to go ahead and set up and generate your API keys and store them ahead of time. You can do this as part of your installation.38:04Just the important thing here is that when we generate our keys, we make sure we keep them safe, secure, but also don't lose them, because then you may have to regenerate your key and rerun your install. For deployment, DNS is probably the most important thing for getting the deployment to run smoothly and correctly.38:22You're gonna need to set up a couple of records, we'll need a record for API.cluster name, domain, domain, and then a wildcard for the apps and then depending on how many workers, how big of a deployment you're using, you're gonna need to put in your, your records for that as well. So.38:41For a deployment like the one we have in our lab where we have two nodes enabled with GPUs, we have three other nodes that are there to be kind of our control cluster. We would need these five entries, two for the worker nodes, three for those control clusters. Now this would be kind of an ideal scenario, what we're gonna.39:00See today's a little bit different. We're looking at a few different environments at different stages of deployment, and most of them are gonna be single node and the reason they're single node is we've got, I think, you know, 40 engineers trying to share these five servers right now and so getting all of them to myself for this demo was was a little bit of a big ask.39:20You're also gonna wanna set up your SSH key for openshift. So just SSH key gen, and then create an openshift key. This gives you your public and private key. You'll need them for the installation and for passwordless SSH once you've got everything set up.39:41Keep them close and keep them safe, like you would any other SSH keys. And with that, we're gonna take a look at the OpenShift assistant installer. And this is we're gonna integrate Red Hat and Cisco with OpenShift and Intersight to really streamline this process.39:58So we're gonna start by going to console.red hat.com and we're gonna set up and create an account and I'm gonna go ahead and show you what this is gonna look like live. Give me 1 s and we're gonna swap over, right? So here's that same inter site Mohammed was showing you earlier. I'm gonna go grab something real quick.40:17From one of our servers. So we're gonna go into our servers.40:36Filter down to that AI pod and we're gonna be working with the server in slot three today. So here's the server, just a quick inventory of what we're looking at, what's in here, for the GPUs we have a pair of those, those GPUs mounted through that PCIe node.40:41These are nvidia L 40 S's but the real thing I came here to grab is actually from our network, our server profile Phoenix, I need to grab my Mac address and this is gonna be important later when we're automating our installation.40:58I'm gonna grab the Mac address and I'm gonna put it over here in a notepad file. I've got kind of off to the side of the screen. We're gonna keep that for later. And now we're gonna go into our red hat console, so console of red hat.com, we're gonna click on openshift.41:17To get to the open shift console, and we're gonna do assisted installer. I mean, you, you can see here where we've created clusters in the past. This is actually the a cluster that's currently created in the lab. We're gonna walk through the steps for how the setup would go.41:35This takes a couple of minutes, but the actual installation has been taking us about an hour to an hour and a half depending on how long it takes to download everything. So obviously we're not gonna do that part Live go to.41:53When naming your, your cluster, it's important to note that it's lowercase letters only. For those of you who like to do cample case and kind of have that as your natural inclination, it is going to be you and won't let you do that. We can give it our base name, which is.42:14Yeah, we're gonna select our open shift version. Generally speaking, you're gonna want to go with, you know, kind of the latest stable release, you can use the developer preview.42:32So we're gonna go with 4215 you have the option to select your CPU architecture, we're using, you know, X 86 architecture here, we can integrate with platform partners. So if you are.42:48Mixing this with the Nutanix or with vSphere VCF you we have there's some automated integration there. We're not doing that. We're just going straight to the node, and then for us, we're doing a single node openshift today. Partially just so we don't have to type all the extra addresses here for the demo.43:06And also because this gives us that flexibility in, in our lab and and possibly in your locations as well to get started with a single node, and then over time come back and maybe redeploy as a highly available cluster. You will get the warning that a single node does not have high availability.43:23And then we're gonna do static IPs because our lab is set up for that. Click next. You can edit this as a YAML file. I prefer to do the form view, put in our DNS server.43:42We've got our IP subnet range, which is the range of IP addresses for the host, which should be oh and our default gateway.44:04And we're gonna go next and then it's gonna need some host specific information. So for host one, we're gonna give it that Mac address we just grabbed right here.44:29And we're gonna give it IP address, we go next. And then we're gonna have some options. We're gonna use the OpenShift for AI. This selects six operators here.44:47For the part of the initial install, so it will install some stuff for us that we don't.45:07That refresh.45:29Hopefully it saves our spot ok so we're actually actually took us a step ahead. Let's go back for a second. Yeah, so that's just our operators. We'll go next.45:32We want to add a host. So what physical host are we doing? 1st I'm gonna go ahead and create our minimal ISO image to fetch content. We'll generate the discovery ISO. If we wanna go mount that ISO directly, we can. There's a W git command here, there's a, a URL. But here's the cool part. We can add.45:49Got a host from Cisco Intersight. Because I'm already logged into intersight, it's not gonna send me through the login or anything else. What organization we are in our AI workloads organization, and then we were just gonna select our server that we would be doing server three.46:08Save, and then if we wanted to, we could hit execute right now and it would begin the process of bouncing that ISO running through the automated OS installation and setting up the cluster for us. Like I said, that process has been taking us about an hour, an hour and a half depending on.46:26On our network bandwidth to get the download and to get everything loaded in. So I'm gonna go ahead and cancel that out right now. But what you're gonna get when that starts is you will have a request that shows up in intersight. And it's gonna look something like this. You'll see a.46:44Add server, from ISO, it'll start building here and you'll be able to see the execution flow. So it'll go through all of these steps and we can, you can monitor through here, doesn't take too long once we get to the final point, but it.47:03Does take a little bit to get here. So once that server is deployed, we're gonna have our OpenShift console locally, so I'm gonna cancel out of here then we're actually gonna go over to our server and see what this looks like. So give me 1 s.47:23Here we are given me 1 s I'm gonna close that window and.47:46And reopen it real quick.48:06Okay about that. Give me just one more second here.48:34There we go. So here's gonna be our our home screen and our overview for that single node cluster once it's set up and you're gonna be able to see here we've been trying to set a few things up. We've got, you know, one node in the cluster, there's a hundred and 87 pods deployed for our in our Kubernetes here, we have warnings on files.48:58This is gonna be what we log into start, this is where we're at. What we're gonna wanna do at this point is we're gonna want to check a few things. We're gonna 1st make sure that NFD is working. Now NFD is gonna be our our sorry.49:21The node, I will just look it up here because the ecosystem installed operators the node feature discovery operator. So this is what kind of.49:40Allows us to discover what the node and our hardware is capable of and then allows everything else to use it. So one we want to make sure that this is installed and running correctly, so we are installed. Lusters good.49:59Condition everything looks ok here. And then the next one is we want to make sure that our NVIDIA operators are installed. So by default you'll get this Nvidia GPU operator. This is gonna be.50:16The drivers and cluster policies. This is you know how our our GPU is gonna work. Here's what we've got in, the OpenShift web console. I always like to check this one on the CLI as well, so we're gonna open up a quick session here.50:33And then we are gonna connect to openshift. So we're gonna copy our command our login command, so this always is a token based login, Grab our token.50:55And because I know that I don't have certs set up properly on this, I'm gonna need to make sure that instead of just that base command, I also tell it to get the.51:11TLS security check. So I've added this instance here skip TLS verify, you don't have your cert set up yet for a pod that we for lack of a better term, reset every five days getting the cert working is kind of a bit much.51:28So now we have our console access, so we are gonna see get, and we're gonna check on NFD. Okay, so we see all of our pods for NFD are running. This is good. The next one we're gonna check.51:46Is that Nvidia GPU operator? Okay, and so we see everything in here is running, and that's why we want to see. We wanted to see that things are running. We can see they've been out for about.52:04Almost two days now, which, you know, sounds about right for us. And then we've got it working, we want to make sure that we can see what it's actually doing. So for those of you who are familiar with Nvidia and running, let's say Nvidia SMI, we can still run that from here. It, it, but we've got to kind of run.52:21That is through the container, so the way that command looks is we actually do OC, execute, and then we've got to give it the driver forget to select in a second.52:41Not which namespace we're in, and then the command we actually want to run. And so we can see here that, hey, we, we are detecting our 2L40 S's that we expect to see. Right now there's nothing really loaded or using them.53:07As we get further along in the demo, we're actually we will see this kind of fill up. One thing that's an important to note though is you probably don't want to have to come in here to monitor this. By default there is not a monitoring dashboard for the GPUs within the open shift environment, but it's pretty easy to create.53:27Provides a JSON file for Grafana export just on their GitHub. I'm gonna put a link to it in the chat after towards the end of the presentation along with some additional commands for how we create a quick.53:47Dashboards, so the JSON file, pretty much curious looks like I don't have a JSON view right here. It's about 800 lines of code, so just You could build one yourself, I suggest just grabbing the one off of videos website or off of videos GitHub.54:05And then we'll just run a command to create and export that this information into the dashboards. And what that's gonna look like when we're done, if we come down here to observe is we create a new dashboard.54:23Nope, maybe we didn't create the dashboard in which case we're gonna go ahead and build it. Oh, we are in the wrong operator. Here we go. This is what that dashboard's gonna look like once it's built, so we can track our GPU temperature utilization, we can track power usage.54:50We can track the the clock deep utilization, memory utilization at tensor core. So you can see I haven't really been using it much in the last 25 min that should change when we switched actually running our live demos and we'll, we'll see some usage in here.55:11Once we've verified all this, the next thing we're gonna want to do is come over here to the software catalog, and we're gonna wanna install the Nvidia NIM operators for Kubernetes. You can see here we already did the install for this just because.55:30It does take some time, takes about 20 30 min, but you'll come in here, you'll click this, and instead of just saying uninstall, it will simply say install and you'll do the install option there. The reason we want to grab the NIM operators is it gives us some additional tools to manage our models. So.55:46Now that this is an installed operator, we can come over here, there we go. And there are APIs that we can use in here to create some additional pods to run. Nim cache and NIM services are probably gonna be the two most important ones. This.56:06Just lets us cache our data locally for our models that we're running. So when we're talking about doing inferencing and and stuff like that, we're just trying to speed stuff up as much as possible. So when we can have things local, it just works a little, you know, it's gonna be faster, it's gonna work a little better. NIM pipeline is the ability to manage multiple NIM servers.56:25So, this is not something that I've gotten to play with work a lot with because we're not running multiple models multiple services concurrently on a single node.56:45But as you expand out, you'll be using kind of how you'll manage those multiple services. So with all that said and with all this here, I kind of want to move over now to a different environment where in addition to having all of this installed, we've set up some vector databases, we've imported the models from hugging.56:57So we can actually show you what a, what this environment looks like and how you can use it once it's running.57:16It is gonna be somewhat up to the user so this is a separate setup that we have where we installed installed some additional information and it's.57:23Or some additional services. And so what we'll see here under our deployments inside the project we're looking at today is that we've installed, we've got an Oama pod set up, we've got a VOLM pod set up, we've got an open web UI pod set up.57:42Open web UI is probably gonna be the most common way you start developing your web interfaces for your your your models.58:01Right now, the VOM pod is running. We will swap to the Oma pod and then we'll actually look at a workbench, Jupyter workbench as well. And the reason we're only running one of these at a time is they do require a GPU and so.58:10We're freeing up and and moving the moving around which pod has the GPU as we do our different demos. So let's take a look at what this Open Web UI looks like. So this is Open web UI, it's a very simple interface.58:27We've got Mr. loaded as our model right now. We have a couple other, we have one other one in here, which is the, this foundation AI from Cisco's security model. So we'll be doing a demo with both of these today. So for Mr., this is where you're gonna use this is kind of an example of how you you would use your business intelligence.58:44For some retrieval augmented generation, so I'll ask what is a Cisco AI pod? So right now it's saying it doesn't know anything about Cisco AI pod. That's fine. We will give it some information that we have. So I'm gonna go into.59:04My folder here. I've got some PDFs that describe an AI pod, these are from the Cisco website. We've got the spec sheets for the 210M7 with the spec sheet for the PCIe node.59:23We're gonna add those in as a context for this to, to answer these questions. I'm gonna ask it the same question now that it has the context of that information. And so it's gonna give me this type of information.59:38One thing I would suggest when doing something like this, and just is that temperature is to change the temperature, and what temperature is, it's really the how creative of an answer do we get when we're talking about our business logic and our business intelligent? Sometimes we don't want it to be super creative. We just want it to be super accurate.59:59So we'll turn the temperature down a little bit to, to make it a little less creative, but a little more focused on the information that's actually been given, and then and.01:00:17Please forgive my my bad typing here.01:00:37So this will give me the list of what's available in the 440P so we've got some of these are maybe not quite accurate, some of them are that that's just the model will need to be kind of refined a little bit once you get started, but this is kind of how this is gonna work.01:00:40Option two here that we wanted to look at though is I want to look at the foundation of security foundation AI security model that Cisco released that's out on a hugging phase that you can use. And we are gonna start a new chat here.01:00:59Nope, Nope, new chat. Sorry. And we're gonna switch to that model and I'm gonna upload a different set of files, and in this case I'm actually attaching some files that are a list of Json based alerts that we've gotten from.01:01:13Oh a ticketing system and with with this we're gonna go three year right.01:01:33Prioritizing.01:02:03Right or perfect.01:02:19So I'm gonna tell it to prioritize which of these alerts we should, we should look at and why, and the 1st thing it's to do is probably gonna loop through, hey, here are the alerts, here's the suggestion for them. In summary, high priority alerts 145 and eight, which could lead to an immediate threat, medium alerts, 23236, and seven. So this is just one of those use cases where.01:02:31We can kind of use this for our security posture for kind of looking at.01:02:51Assisting our engineers, our NOC or SOC staff on the admin side. It looks like we're 1205, so I'm actually gonna skip Theama demo for today. I kind of want to take a look at the Jupyter notebook and.01:02:57Because I wanted two demos over there and one of those is because I saw that some of you were saying that, hey, we, we're not sure what's yet, we're concerned, we don't have them yet or we're looking at costs so with our Jupyter notebook here.01:03:14So actually let me show you how I got to there. In your openshift console, there's gonna be an option to go to the open shift, AI. Yes, this is being recorded. I just saw the question come in. So.01:03:31If we go here to our OpenShift AI console, we can set up some data science projects, and we should have one in here that we're gonna look at today. Oh, gonna be on 2nd page, so our little RTP project.01:03:48Inside of a data science project, you're gonna set up workbenches to do various things. We've got two set up in here to run Jupyter notebooks with pytorch and Kuda. Somebody can several people mentioned you don't have GPUs yet, you're still kind of.01:04:04Figure that out. You can actually do AI workloads without GPUs. So I actually have this one running right now fire it up, and it's gonna launch our Jupyter lab notebook. Give me 1 s here.01:04:22So here's our notebook. We're gonna open the right file here.01:04:40Upload, notebooks. So this is a project somebody else had built, everybody knows that already exists even better.01:04:57The top, there it is. So this is something someone built to just kind of give them the estimates on who would win for F one and F one this year, you know, kind of point scores stuff like that. So this is gonna be a CPU only based inference thing that we're gonna run through real quick.01:05:17Well, their Jupiter notebook, and I'm actually gonna go ahead and tell it to just run research run all cells once these packages should already be updated.01:05:39Wait down here yep so it is giving us driver numbers alright and here we go. We've got our predictions for grid position for.01:06:0526 for the Australian grand brief.01:06:25So, this is a Jupyter Notebook actually built to run just on and to leverage just the CPU, so you can actually get started with this with just CPU based inferencing while you're evaluating which CPU options you're going for or you know just waiting to get them. I saw some people said, hey, we're getting this.01:06:28Six months.01:06:48You can actually get started with with some of the setup now, and things like the Jupyter notebook are exportable, you can move them over to those GPU based servers once you've got your, your system up and running. Now if we wanted to instead use a GPU for this, I'm gonna go back over here and I'm actually gonna stop this workbench.01:06:49I'm gonna go back over to my deployment model and I'm gonna go into VLLM because VLLM is using the pods that currently have my GPU, and that's what we're using for that open web A the Open Web AI. I'm gonna shut off that pod so I can free up the GPU resource.01:07:11I'm gonna come over here in back into my workbench. This workbench does require a GPU, so I'm gonna tell it to start. This failed message is because there's not an available GPU right now. It's still being freed up from the other side. It should auto recover and, and pick that GPU up here.01:07:29About a minute.01:07:49So we'll just give that a quick second, there it goes. So that's gonna take a little bit to start while it's starting I'm gonna switch back over here real quick.01:07:51And I'm gonna go down to our dashboards and we're gonna look at our nvidia exported dashboard. Hey, we we can see right here when I started asking questions the GPU, we actually saw.01:08:12GPU usage.01:08:31Generally speaking, having a a single GPU dedicated to one individual using using it for a, for inferencing is probably not ideal. When you would have this opened up to hundreds of employees, you would see this kind of maintaining almost full utilization, that's kind of what we're, we're after and what we want to see.01:08:35But yeah, you can see we just have a small, you know, we have a spike in power, we have a spike in usage as we use it spike in the, and there's our tensorcore utilization as well. So this should about be started. Okay, we've got our our new notebook up and running.01:08:53So, NO user, give me a second So we started up.01:09:11Our new Jupyter notebook.01:09:30This one's now got access to the uselet GPU. So we're gonna do slightly different model and a slightly different work here, but one of the other things I want to do is actually want to come in here.01:09:33Where's my launcher?01:09:52We have Root here, Viptia. All right, still seeing our single GPU that we've been assigned here, we're good to go. Now I would like to.01:10:33Run this Jupyter notebook, which is a, a simple agentic workflow, it should run pretty quickly. Restart and run all, and what we're doing with this is setting up a.01:10:51A quick little logic workflow.01:11:11And this is gonna show something a little bit more complex than just, hey, we're, we're trying to pick a winner for an F one race. This is gonna be how you would set up, say, interactive bots or something like that for your workspace, so we're installing Lang Chang where we've got.01:11:14Pulling in a HuggingFace pipe from the HuggingFace pipeline. We are installing some transformers. It's gonna set up the graph here. Okay, so it is gonna do the download again.01:11:33And then we should eventually get some chat output, and this is kind of what we're, we're after to see or out to see. And the prompt here, so you can see here the prompt you're getting your friendly chatbot that interacts with users, answer questions to the best of your ability truthfully and friendly, keeping it to three, three sentences at maximum would be concise.01:11:48So, current location that is just a weather apps for you, can you provide more general information? So let's try this again. Whether.01:12:10And I think we had people in Austin today. Yeah that's what this is right. Should have sunny climbing average temperatures in seventies and eighties, so it's just giving us a general information, but this is.01:12:28Kind of what that would look like. Now I'm gonna go back over to my console, still not seeing anything here, right? Like we it's kind of hard to catch what's going on there, but we should in our metrics dashboard, see some more utilization.01:12:45Pulling interval might be a little slow here, but that's gonna be kind of an example of using a Jupyter by setting up an agentic flow. We have a variety of different things you can do once you hit this state.01:13:04I, the the kind of numbers use cases get pretty large. We have two two sessions I believe coming up that are later on as part of the CIS series that are gonna really gonna deep dive more in once you're at this point, how do you translate this into a business use case and then solving your business needs?01:13:20So with that said, I'm actually going to switch back over to our slide deck. Give me 1 s here, find the right slides.01:13:39And so this is kind of the the last little bit right is how do we use those GPU accelerated data science agents? That that Jupyter notebook over there is kind of a simple one. It just hey give me give me a quick response.01:14:09We have, the ability to use a variety of different software packages that are out there that are already existing, either models or or things like preexisting data sets that we can use to start with this. We can give it our own.01:14:25Business logic and our business data once we have our governance in place, to kind of act upon that for that retrieve minute generation reports and we can have it kind of interpret natural language or a little better. If anyone was looking really closely, you kind of see I'm.01:14:43You know, sometimes I have a misspelled word, I I don't have proper grammar English in some of these queries, still gives me a pretty good response. That's kind of what it's designed to do. That's one of the big, big pieces about this that makes makes AI kind of work for a lot of people is that it's good at picking up kind of that intent even if we don't wear to.01:14:59Exactly right sometimes. And we get some pretty good analytical results.01:15:19So common questions we got, this has actually been delivered in one of the other theaters, somebody asked, hey, can I monitor everything? Intersight can monitor the metrics on the GPUs, OpenShift can monitor it. Splunk can monitor it, and I will actually show you what all of this looks like in Splunk here in just a second. And the other one was, is there a way to automate everything?01:15:25Thing and the answer is is yes.01:15:45Almost everything we we deploy here is done via API calls, so there's a quick setup through Red hats, the OpenShift console that can all be done via API. All of the installations once we have OpenShift running, are almost all done through just running YAML files.01:15:49Or JSON files and using that and applying it inside of, of openshift. You can work everything, not everything. You can work a lot of it through the GUI, it's it is more designed actually for that automation.01:16:07And then for the observability piece, let me, here we go. The one last thing I wanted to show is if you really, if observability really matters to you, I I really suggest, you know, Splunk's kind of the way to go, so we can.01:16:23Absolutely look at it with the inter site metrics and I will, I can show people how to set those up real quick. But if you're using Splunk, the observability cloud, has some built in dashboards that kind of just.01:16:43Do everything for us as far as the monitoring goes. There's a Cisco AI pod built in dashboard. I'm gonna select my cluster here. We are RTP one and again this is a little busy the monitor I am sharing from is a little small, but we can see there are nine notes totally.01:16:53There are six GPUs between the two chassis in there. We can see GPU utilization, a couple of the GPUs over here are really getting kind of hammered. Most of them are saying fairly idle to be honest. And you can kind of see here in the graph, we definitely have some servers that are getting a lot more work than others.01:17:14That's just kind of the nature right now with the demo lab. What you'd like to see ideally in your environments is kind of consistent uniform utilization. And if not, you kind of want to rebalance across that. Average GPU memory utilization, that's probably not right at 200 %, but we can we'll take a look at that. And we can see the tensor corey.01:17:33Power by node. For those of you who also do the network side, this also does monitoring of the Nexus switches.01:17:53There's a pure back end on some of this, so we've also, we're also monitoring that it's kind of a one stop shop. We can see in it, you know, from the inter site there's the alerts alarms and warnings. We can see our UCS host temperature, the hotter ones are gonna be the ones with the GPUs in case anyone was wondering.01:18:01So this is what it looks like in the splunk. If you want to observe this through Intersight, the best way to do this is to set up some custom rules down the explorer. So we'll go to analyze Explorer and I built a couple of these earlier, so we'll just take a quick look.01:18:21Or like GPU memory utilization, you can set these up to five GPU. I've got this one set up to show me both maximum and average, the environment that this particular inter site instance is monitoring I don't think anybody's using the last few days we can.01:18:37I check back a little bit. Yeah, I don't, you know, small spike to about 1 % at one point. The one that we're monitoring with Splunk though is actually fairly constantly and used by a lot of people, which is why we get the spikes that go up and down.01:18:57So we we do have the ability, a couple of different ways to monitor this. Intersight will monitor it, OpenShift will monitor the DCG with the dashboard we built. And I do recall that I owe everyone putting that into the chat window before you leave today, so I will definitely do that. But that kind of wraps up what I've got jeff.01:19:14So if you want to take us home with that that last slide up.01:19:34Excellent. Great. Thank you very much, Alan. That was a very thorough demo. I know I learned a lot. Just watching through that, I hope that our audience participants did as well. And what we've got here is our last poll question that we're hiving through Slido.01:19:41And hopefully you're able to get to Slido. Thanks everybody that attempted if the Webex client doesn't pop out for the Slido, hopefully the slido.com webpage allows you to access it, log in.01:19:58And take a quick vote. So what we've got up for now is what are the biggest compute challenges regarding AI that our customers are facing today. So let me get that presented here and we'll take a look at it together.01:20:15I'll share my screen out and if everybody that's able to can go there and take a vote, what we'll see at the top of the list so far is that the vote for somewhat is, is getting the votes so far.01:20:34So we don't have anything in the very category or the not at all category. I give everybody a minute to continue the vote there. Again hopefully this will go into our.01:20:53Content that will come in the future and allow us to better customize these expert insight series sessions to be a better fit for our audience. Now there was a question earlier, hey, is this getting recorded?01:21:10And that link was shared where the recording is going to be posted. At the very top of the chat, you should see a link that I shared. This is gonna take you to registration forms where you can sign up for some future sessions there.01:21:26And if you're not able to attend those lives, they will still be posted on the recordings page there. So it looks like everybody's at somewhat. We've given everybody a chance to vote, and so what we'll do now is go back and take a look for any questions that have.01:21:43Come up.01:22:03I know that in our Webex chat, we had one or two of those were answered verbally, we had one around the 8805 being monitorable in intersight. I think the timeline was posted for that.01:22:03Also two, we're gonna take a look at.01:22:24Where we are coming up for, let's see, let me go down to reader.01:22:29Bear with me a second here. Let me get called up on that let's see.01:22:47I know these are things Alan covered during his demo and we did have that uploaded. Bear with me a second here. Let me get this re shared.01:23:07So that we can take a look at our closeout. Again, if you have any questions, just pop those in the Webex chat there. We'll do our best to respond before the end of the session. I'm gonna get.01:23:29This set up for us so that we can take a look, ok.01:23:48So I'm going to share this out. We'll take a look at it together. You should be seeing our journey there so you can continue your AI journey. We see that we've gone through our AI portfolio overview.01:24:08We've gone through a deep dive on data center networking for AI in today's presentation and demonstration was a deep dive on data center compute for AI workloads.01:24:27Now our next subject is gonna be real life use cases on Cisco infrastructures, and that might include some content that we may not have time for included today. So please catch that, that will be in one week from today.01:24:41And the final session there is agentic AI from automation to autonomous networks. Again, the registration link is shared at the top of your chat. We'll leave that open for a few more minutes so that folks can access it.01:24:56And with that, if we don't have any outstanding questions, we've come to the end of our presentation for today. This has been a deep dive on data center compute for AI workloads.01:25:16We appreciate everybody that was able to attend. You should see a poll for the attendees that is open now. I encourage you to vote. Please take a minute, let us know how we're doing. Ask any questions you didn't get a chance to ask before and give us some feedback there. That's for our benefit.01:25:33We'll leave that open for another minute, make sure everybody has the opportunity.01:25:56In the meantime.01:26:20I'm gonna post the series registration link, back in the chat just so it's closer to what everybody's probably looking at and then we'll have to scroll all the way up.01:26:22They're there at the bottom next to a couple of Allen's link to some JSON files, you should have the news.01:26:39Series registration link where you can go and register for those next sessions. So hopefully everybody's able to take a quick vote in our participation survey. We do appreciate your attendance.01:26:53To that, let's see, I'm scanning for questions here. I don't see any new questions in either Slido or our Webex client. So again, we appreciate your cooperation and.01:27:13And thank you for your time today.01:27:33