Scaling AI and High Performance Computing (HPC) with Cisco UCS

Oh Hello everyone, I'm Galbers and thank you so much for joining us today.00:00To this new session about the our expert Cisco expert inside series, ok? This is the scaling AI and HPC with UC Cisco UCS, ok? So thank you so much.00:17Once again to join us today, this is our second session about the Cisco UCS as you can read here in the, in our three part series from call to the age, ok? Today's 11 June, so we are going to go in deep in this in our UCS seri.00:34Cisco portfolio, focused in the HPC workloads, ok? And next week.00:53You will have the, the, the latest one with the about the unified edge computing with my teammates Ruc and and Mark, but today as I said, we are, we will put focus in the, in the HPC, ok? This is the flow of the session for.01:00Today.01:19The idea is to, you will see it doing in in our agenda right now. The idea is to explore the AA challenges and all of those things we are facing in our AI deployments and AI trainings. After that we will go into the.01:20Infrastructure deployment, how to scale in the AI infrastructure and so on. And finally we will.01:40Right up with a summaries of those of the of all of those things, ok? And you will have some time to make some questions and answer all of those, all of those things. But before we start this session remember that we launched Cisco launched the Cisco IQ is the new.01:47AI customer experience platform is the evolution of the CX cloud.02:08Bringing together, support analytics, automation, and, and artificial intelligence into a single unified experience, ok? So it's trying to provide the comprehensive visibility.02:15Into Cisco assets, the infrastructure health, and life cycle status. So the the idea is to identify the risk, security, vulnerabilities and configuration, ok? Before the, the, the impact operations, ok?02:30The the this is a single interface that meets what you are you were and you are, ok, personalized personalized to your environment for both professional and and support.02:48Service digital capabilities. If you are already for for existence six cloud customers, you will be able to migrate your data fully automated, ok? And for the new Cisco IQ account administrators, you need to follow this.03:03Simple process creating the new Cisco account, obviously creating the name and selecting the the primary storage and the sub guided account configuration. It's, it's pretty easy, ok? So, as I said I tried to divide this.03:23Agenda in three different in two different main sections, ok? And so we will cover the key challenges on of deployment AI and at scale the infrastructure components required for to support modern AI workloads and how Cisco AI ready.03:41Solutions help to simplify the the deployments and operations. We all we will also look into the validated AI architectures, the scalable training and inference designs and and and how e.g. in this case open red Hat open.04:01Operations across the the full AI life cycle before closing with the.04:24Key takeaways and and next step, ok? But I tried to divide the, the agenda to make it clear the two different things we will talk about, ok? And, 1st of all, we had our Slido because we will we will like to know more about your, your.04:30Or experience with, with AI, ok? This is the beauty that you can.04:50Where you can login in under the passcode. We have some questions for you and so I will give you 10 s to, to log in, ok?04:55If you are already in, I will give you some time.05:11We have some questions about your experience, about what you are thinking about the your AI trainings and and so on. We have some questions during the session, ok?05:40So we would like to ask you about which area are you most interested in learning about today. I will start the the poll, ok? We have those three options, the GPU and AI infrastructure design, we have the AS scalability and performance or maybe the Cisco.05:55UCS AI portfolio.06:15Today e.g. we today we will put focus in the, in some specific servers designed for AI, but obviously you have, we have more and you can check it out in our Cisco validated designs if you want to know more about this about the hardware, ok?06:19And our chat is enabled by the way, if you want to type some questions or specific questions for me. Okay, so we have 89 90 % in people who is interested in the GPU and AI infrastructure design. We have been D of course.06:42The scalability, the Cisco AI portfolio, that's good. That's good. Great. Okay let's start then. Oh yeah, yeah, yeah, sure. Let me go back to the slide of here.07:03Hello Jenny, by the way. Good morning. You got it? This session is being recorded.07:28So you, you will be able to check it out later, ok? If you miss something. Cool. The winner is the GPU one AI infrastructure design. So that's a good thing. Okay, let's let's start.07:47Yeah, the AI challenges that we have. So let's get started. As you know the here, as you know the AI, the AI is NO longer limited to a few technologies, technology companies, ok?08:04Organizations across the healthcare, finance, you have plenty of them, retail, transportation, and so on are actively adopting AI, ok? Each industry has its unique use cases, but similar infrastructure needs. In fact, in the same infrast in the same industry we get.08:21Can find out different use cases. So it's a technology who.08:41Needs to be implatted in every single industry in this world, ok? So, e.g., the most obvious in the healthcare may focus on diagnostics and and patching support, the financial services, we have plenty of customers in the financial.08:47Industry needs to be focused in the frag detection and risk analysis so retail e.g. personalization and customer engagement and so on and so on and so on, and so on.09:03The AI is becoming a horizontal business capability, ok? So regardless of of industry organizations are looking to improve the efficiency, the automate processes, enhance the decision making and and and so on.09:18But if we talk about the, from a operational point of view, the AI is a challenging to deliver an AI use case in in production, ok? Sometimes it's a, it's a challenge. You need to identify the and select the model that meets the the needs of the use case, then you have to create.09:37Created your organization data and customize the model.09:57And once the model is in production ready, integrate with and deliver an AI enabled application to your, to your user, either internal users or or to your customers, then scale and maintain the the application. So sometimes it's not easy to deliver this pipeline and and it's not trivial.10:00No it's not true, ok? So.10:20The three major challenges we find out with the AI projects I think the organizations face those three major challenges that you can see. The the complexity, security and lack of of clarity. In the in the case of complexity.10:23The build a AA solutions requires integrate integrating compute networking, the storage, the data pipelines, models, and and all of those things in a high performance architecture. That's not easy. Cisco, one of the most important things for Cisco in this case is to.10:40Simplify the, the, the operations and the deployment of all of all of our infrastructure from in the hardware and software side, ok? You will see it later in the in this presentation. 2nd one is the security. The the AI introduced new attack surface for the hackers through model.11:01Frameworks, applications and data pipelines, ok? So we need to secure as much as possible our eh infrastructure.11:20And finally, the lack of clarity we identify this these challenges the what this challenge in many organizations are struggling to define the right use of the AI use cases, the the business value, the, the of investment, which.11:29Allows adoption adoption sorry adoption and delays time to value, ok? So we are trying to, attack those challenges ok from with our solutions. So the AI is accelerating the technology adoption.11:47Forcing organizations to evaluate, deploy and operate operationalize the new technologies faster than than ever before. The generative AI and the llmiss are a new way of knowing innovation they are creating both opportunities, but at the same time they.12:05Infrastructure challenges for, for enterprises. So our organizations need to be flexible and scalable platforms that can adapt to so quickly of aborbing AI technologies, reducing the, the, the, the deployment complexity.12:25This is a good example in the in the financial services, we will make assuming in this, in this industry, ok? So if you are part of this industry, thanks for for joining us this session. As I said before, as we said before, one of the most important things in the, to understand the about the.12:43AI is that there is NO a single use case.13:03For single industry driving adoption. So, if we take the financial services as this example, the AI opportunities span across multiple line of business in in customer service, the AI port assistance can provide.13:06Intelligent questions answering and faster support in the line of business of sales, e.g., the AI can identify the renewal and and upsell opportunities through the data analytics line of business, ok?13:24We have plenty of them in marketing in in in finance department, all of those, all of those needs to develop his own its own use case, ok? As I said, let's make a zoom in in a, in a.13:41In this range of AI workloads and.14:01If we were to distill this, use cases down to AI or or machine learning workloads that enterprise IT and infrastructure teams needs to support, it comes down to to this. You have multiple use cases spanning multiple line of.14:04The business that you have to provide infrastructure for, that's.14:23Our job of, of course, and and you can build isolated environments for this or you might, include them as a part of larger data center infrastructure where you already run missing critical applications and and services.14:28So let's make a zoom in in this in this part, and in every AI initiative is following a life cycle, which is the most important every every AI initiative has its its pathway, starting from a business need moving through the the.14:45Projects and and use case definition and then progressing through model training, the fine tuning and and obviously the inference, ok? So our succeeds depends on more than the model itself. The, the many organizations must address data preparation.15:06The app application integration, the user experience, the security, and the operational processes across the entire AI life cycle, ok? And finally the the AI environments are hybrid by nature, it are hybrid and often spanning on premise and cloud infrastructure.15:24Sure Recruiting cons consistent management, governance and and and the life cycle control for both model and that, ok?15:44So in this pipeline is that the AI is a pipeline in a continuous life cycle that start with the data preparation, as you can read here, moves through the model training and serving and ends with the application integration and and business value.15:55Delivery in this in this slide we are trying to communicate that Cisco supporting supports you in every stage of the, of this process in, in every stage of this life cycle with a scalable architecture, you will see during this presentation how to design and and make it scalable in the future.16:15If you are growing up, the automation with inter site, you will be able to deploy all of this infrastructure fully automated, the monitoring.16:35And the end to end security to to accelerate the AI deployment and and the and the and the efficiency, ok? So that's the the thing for today. Let's start with the, with 2nd poll. As I said.16:44So once said this about the challenge we are facing, we will ask you to what is your organization main focus for the AI today? Probably you are in different stage of the process. Don't be shy. So we have these five different options.17:01The AI model training, we have the AI inference and production AI, the traditional HPC workloads, data analytics, we're still exploring right now the winner is the.17:21We are still exploring AI and HPC. That's a good thing. You are, you are trying to find out your use case and that's that's a good thing. We can help you to define this use case and the better thing to define the design and and deployment.17:41Model training. We will talk about the model training, the model training later from scratch or using a customized AI model. It depends of you. Good. Yeah, the winner is the that we are still supporting the.18:09AI, as I said that that's a good thing. We need to study 1st our use case, define what we need, and then work in the in the design and and explore. Okay, that's it, great. Let's continue it.18:29Let's move to the infrastru infrastructure considerations, ok? We said we defined the the the challenges we are facing off with this new technology, but when we are planning the AI infrastructure, one of the most critical considerations is.18:47The memory capacity for the for the training and and fine tuning we need enough CPA GPU memory not only for the model itself but also for activations gradients and optimizer states for plenty of things, ok? For the.19:04E.g., for the inference, the focus shift towards service and efficiency, the concurrent user, the con the context windows size and latency charges, ok? So the capacity planning as we did in the past with VMware, e.g., for virtualization environments, the capacity.19:23Planning should start with the understanding of the model size.19:43The workloads character characteristics, the expert demand to which allows us to estimate the GPU memory and the compute recurrence for our AI infrastructure. Okay, so the here you can see the two different pillars of the.19:48Predictive and and generative AI that can be divided in this in these two pillars, the predictive.20:08AI is is using the historical data to forecast outcomes and drive decisions supporting use cases such as e.g. fraud detection detection, forecasting and and recommendations. Meanwhile, the, the AI goes a step further, I think.20:15By creating a new content, a new content including text.20:35Images code using using NL sorry, using LLM llm's foundation while both delivers business values and and and typical requires more infrastructure resources due to its larger model sizes and computational demands.20:40Okay, so that's important to clarify in this in those two pillars.21:00And let's move on to the training LMs for for scratch from scratch and sometimes it's too much. We need to admit it, ok? This slide is trying to highlight it, why the training and LLM from scratch is beyond the reach of many organizations, ok?21:05As models have grown from ten or billions to trillion of parameters, parameters they require data sets, GPUs resources, training times and training times have increased dramatically, ok? So this data is too much sometimes, this.21:24Load is too much sometimes, so the modern.21:44Foundation models can require thousands of of GPUs running for weeks or months or years, making a training extremely expensive. Sometimes it's it occurs and we cannot deny that as a result of this thing, more most enterprises achieve faster.21:48Attempt to value by leveraging pre trained models and and focusing on fine tuning, e.g., rack on and and inference instead of building their own foundation model from from scratch from from the ground up, ok?22:09Yeah, you can see the the.22:28What we need with GP GPT four or for Lama three is a huge amount of of GPUs. Okay, so the the the foundation of the once it is after the previous slide, most organizations are not training model from scratch. We we fill it.22:30We we we saw it in the past. Instead, they are starting pre with pre trained foundation models such as GBT or or Lama.22:50These models, can be used as is or customized through techniques like fine tuning e.g. the the prome engineering and to to address these specific business requirements, ok? And the final step obviously is to integrate the model.23:00You can see here on the right hand side the final step is integrating them the model.23:20Into business applications, so we we are trying to enable in the use cases such as chatbots, the document analysis, the computer vision copilots, et cetera, and plenty of them. And the the.23:27Those, as you can see we have plenty of fine tuned model foundation models can be adapted to many AI domains as you can see in this in this image, including the the NLP, then the the natural language processing.23:42The computer vision, multimodal AI speeds and audio, plenty of them a single pre trained model can be fine tuned for a specific task such as questions answering, summarization, translation, image classification.23:57Etc et cetera. The the those companies that NO longer build separate models for for every task. Instead, they start with a foundation model and customize it for their business use case. So we are trying to.24:16Dramatically accelerating the AI adopting and and reducing the development effort. That's one of them that's one of the most important challenges that we have, ok? So the idea is the question is why customize our model? The th those foundation model are.24:36Are very, very powerful but generic and they need to be trained on public datasets and often a lack of domain specific knowledge required for enterprise use cases, so those the using a model as is in.24:56Introduced risk including the the 1st oneizations limited sources traceability plenty of them the the difficult to control the updating specific knowledge with the within the model, ok? Within the board.25:16So the customization improves the accuracy and business relevance.25:36Incorporating in enterprise data, the the processes and and and and context such as fine tuning or or rack, ok? Customize then another way would be to customize the the LLM so.25:43There are several, we have several approaches to customize the the llmiss, the the the fine tuning and parameter efficient techniques like like Laura, e.g. modified the the model itself.25:59To specialize specialize for a particular domain or or task. Alternatively organization can use their rack and from engineering to provide additional context and and at inference time without change.26:14In the model without change in the model. In practice in practice rack, e.g., is often the the preferred enterprise approach because it keeps their response.26:35Grounded in, in current business that are well avoiding the cost on complexity of retraining the, the, the large models, ok? We have some a couple of slides about how to estimate the the memory in this case for, from five.26:47The tuning when.27:06And when we are trying to estimate the memory for the file fine tuning, we need to consider much more than the the the model, ok? The the VRAM that's why I highlighted here the VRAM must also accommodate gradients, the optimizer states, the active.27:07Patients and and operational overhead. So those factors such as model position, the batch size, the sequence length, length, and, and e.g. con context window.27:27Can signifi significantly impacting the memory consumers so that this data is this is why accurate capacity planning is critical before selecting the the GPUs and and design.27:40You will see during this presentation the the the portfolio of GPUs that we have ok and the and the servers but it's important to double check this information with your team, with your teammates, with your teams.27:57And you can contact with us to know more about the design, to know about more about how to create this capacity planning if you need some help, ok? You you you will be able to contact with me. Okay, in this case the GPU estimation the after estimating the, the, the.28:13The memory footprint for of the model, the next step would be to select the appropriate GPU platform and you already have plenty of nvidia, Intel AMD GPUs. And the GPUs differs not only in the me.28:34For capacity, but also in the memory bandwidth and the and the AI compute performance. So.28:54The H100 and the A 100 e.g., are typically use it for the large scale training. Meanwhile, the L 40 years and the L 40 are more provided cost efficient cost efficiency options for.29:00Enterprise fine tuning and and inference. The goal, the the idea, the goal is too much the the workloads requirements with the right balance of performance and and and cost. And so.29:15Select the GPU is extremely important for you to create this capacity planning. Okay, here, e.g., you don't need to remember the all of this data. Of course, this is just for your information, so not in your mind. This slide is trying to highlight it.29:32The the slide high highlights the whole different fine tuning approach impact infrastru infrastructure requirements, ok? So the full fine tuning delivery maximum customized but deliver.29:52But request substantial GPU resources, ok? Especially for for large models language models. Yeah. Maybe it's it's and maybe I read it wrote wrong, so sorry ma'am.30:09It's for this. Okay, so for most enterprise AI projects, the the parameter, the parameter efficient fine tuning provides the best balance between accuracy cost and and scalabilities. Thank you, Lastlo, by the way. I will fix it later, sorry.30:29Let's move to the, to the inference, the how to estimate the the memory in the in the inference Eh request the inference memory requirements are different from training eh of course. So during the inference, you don't need to graduate or optimize the state. The.30:48The memory.31:07Is primarily consumed by, by model weights, activations and the KB becomes a major factor for the, for the llmiss, ok? And the inference sizing depends of workloads character characteristics, sorry, the th this.31:08Model size, precisions, second lens, batch size, all of those things.31:28And the user concurrency or all all of those things influencing the GPU memory requirements, ok? So the small changes in the context length can have a substantial impact on on the of the of your GPUs, ok?31:34This is the GPU estimation inference depends heavily in the on the model size, the precision, the precision and memory requirements and as as model scale from.31:55It be to 400 B 405 B The parameters that requires GPU memory and the number of GPUs increase significantly, ok? Making the precision.32:15Optimization critical for, for efficient deployment. For the for the large, e.g., for the large inference workloads such Lama three, our AI infrastructure infrastructure can leverage high memory GPUs like nvidia eight 100 or two.32:30Hundred six M you will see it in this presentation where.32:50Higher GPU memory capacity directly reduce the total GPU count required for the, for the deployment, ok? So that's important once again for the for our estimations.32:56For the retrieval augmented generation enhances the the AI response by combining LLMM sorry with enterprise data storage in the vector databases. So enable.33:21Following the more accurate and context word answers, ok? So the process ingests and beats enterprise documents into a vector database.33:38Meanwhile, the during the inference, the, the, the retriever dynamically access relevant information to generate generate in intelligent real time response, ok? So that's that's important about the data ingestion and the and the inf.33:51Difference.34:11In this case with the video conferencing print widely used in, in Cisco, e.g., the the modern be those modern video conferencing, not only Webex of course combines the traditional CPU basic collaboration services with GPU accelerator.34:13AAI capabilities.34:32To deliver, let's say richer and more intelligent user user experience, more important impact directly to the to the user experience in internally, ok? So meanwhile the CPUs handle a real time communication workloads such as speed.34:34Video, the audio chat, recording like in this case the the TPUs accelerate the AI driven features like meet summarization, the live transcription and the real time translation, real time translation. Okay, so in the in the flexible acceleration.34:54The AI infrastructure must support a widely range of of workloads plenty of them, many different of them. From data preparation and edge inferencing, to large scale model trainings each of them requires different level of computer.35:13And our Cisco AI solutions.35:34We are trying to provide the flexible scaling across CPUs and GPUs enabling our customers to efficiently support anything from, let's say light lightweight inference and, and at the edge to massive super massive foundation model trainings environment without.35:37Of of GPUs, ok? So, this, all of those things lead us to the how to create our building blocks Beyond the memory, the compute is 2nd key dimension in the AI capacity planning.35:57The amount of the amount of GPUs compute required depends of the depends of the model size, the workload type, throughput targets and the time to results requirements. We need to understand these factors to determine the the number or or or and the time.36:14Type of the of the GPUs needed for our infrastructure to meet the the performance.36:34Objectives let's say efficiently, ok? We need to make efficiency in this thing. So our AI ready infrastructure delivers the the this an end to end platform combining the compute.36:40Combining the storage, the networking, of course, the AI software security observability we have to support the entire that entire entire AI life cycle, ok? So it's complimented by violated architectures, services, automation, the the.36:57In my, in my opinion, e.g., the automation is critical for this kind of deployments and the the the partnerships that help organizations to deploy the AI faster security the security at this at this scale, ok? So let's move to the, to the compute part, my favorite part.37:17Just a second. I don't have it here this might.37:37Here, what is my blocks? I have something to hold on, please.37:59So in our compute platforms, we have plenty of them, ok, from UCS dense GPU servers to the the typical UCS blades with GPU expansion with et cetera etc. Plenty of them. We are trying to cover.38:15The.38:36The that widely range of, of, of servers, ok? So in this case I tried to highlight it in the, here in our in our in our slide with those different servers that we have.38:37And today I would like to introduce the, the, those reference architectures that we have with the the reference architectures of of Nvidia, ok? We have HG HGX and the modular GPU architecture and the the HDX, the.38:58Hyperscale GPU accelerator is it's a maximum perf it's focused in the maximum performance for, for the model training for the monitoring and different demanding use cases, ok?39:18So we have those three different key features that, our customers wants, ok? So we have had eight fabric eight wave fabric attached GPUs with MB links, with MB suites of of MBDA, the infinity fabric for, for AMD.39:31We have the the GPUs and and all of those things. And the the MGX is the the scalable performance for for many of use cases, as you can see that we have those key features that our customers needs or wants for the PCIe GP GPUs.39:49You can.40:09Install it in the in the GPUs in the using the the traditional PCIe slots flexible configuration. It's more focused in the in the model GPU architecture, ok? So we have the 1st one, the 1st of our heroes is the UCS 880.40:10Five, is a is our high density GPU rack server.40:30The the propose of this big guy is, it's for GPU intensive use cases. You you can here we have some our latest data because the MI, the AMD MI 300 is end of sale, so we now.40:35We support this new AMD MI 350, ok? That's important to note. We have AMD four or five gym epic processor ok.40:55And the obviously the for the for the MBD. This is the physical layout, the the cooling and hot swap design of this server, it's fantastic. As you pro as you know the the the.41:11How to evacuate the the and how to reduce the temperature of this server is one of the challenges because the GPUs generates.41:31A super huge temperature in the in our data center and that's important to note that Cisco is trying to reduce as much as possible this temperature and the consumer of the, of those of those servers, ok? So in this case you can see the physical layout and how have been.41:40Using a modular design for for our service, ok? You can see here the 880, the physical layout. This is just for your information with the with our PCA with our PSUs, the CPU was led to.42:00Make it easier to introduce and extract the the server, ok? The differences between is slightly different in the networking. Here you have the, oh, we can go back to bogo.42:20This one, the previous one maybe yeah the differences between those physical layouts is the 81 808.42:37Five and eight 800 eh 80.42:57This slightly different let's move on. This is just for your, for your understanding about how this server has been have been builded and designed. Here in the network connectivity we have the three different.43:00From the backend and to the front end we have, we can segregate or separate the the the traffic and doesn't matter if we scale up in the future from the node one to node N in the in in the future.43:20With eight with eight 400 gb other net ok with the front end, the storage network, we will talk about the storage partnership that partnerships partnership that we have with with the storage companies the management network, all of those things, the, the.43:36All of this traffic is fully separated and designed in in the the same way, ok?43:57So here you can see the the the GPU connectivity in the 885 with four MV MV switches often video with the SX XXS XXM five tensor core.44:03Each NVNB switch is fulling and blocking and connecting the the to all of those the the eight GPUs, ok? So we it's allowing to allow us to the eight 200 e.g., to talk to talk with.44:19Any other 200 eh concurrently, ok, with the MB switch.44:39Okay, as you can see here, in this case we are talking about the the 880 GPU connectivity. Each of those B 300 GPUs have 18, up to 18 physical and billing.44:44Connect connections to, to the fabric, to full on blocking MV link switches connected to the, to, to all of those eight GPUs which provides bi directional bandwidth between any pair of, of.45:04Of the GPUs, the black world GPUs in the same, in the same node, ok? So we are talking the same server.45:24Oh Nicolette is asking how much are we looking in or in terms of cost the.45:35Yeah, I cannot give you a, I cannot give you a figure a number but yeah you need to, you need to double check with your with your icon manager or we can check it after.45:52After this session. I I don't know how this this number on my mind.46:12But yeah, the the idea is to get the this possibility for all of our customers with AMD or or intel. So it depends on it depends of your of your infrastructure, sorry.46:20So, yeah, let's move to the sorry let's move to the 818 hundred 45, ok, our, our young guy for for it's our rack server, the the youngest guy of this, of this team is it's extremely.46:37Flexible which I love it by the way, the the modular design for many organizations that are looking to the yeah start from small, NO one delivers there are NO there are NO too much too many organizations who deliver 100 servers.46:57From scratch with plenty millions of thousands of, of GPUs. So, yeah, I think start small from, from.47:17From small, I think it's it's a really, good practice, ok? So we are talking about a dense GPUs server two four or or eight configuration with the Intel with AMD, with a, with our GPUs often vidia.47:29You can deliver, of course, Cisco hyper fabric and is based in the Nvidia MTX architecture for the artificial for the AI ok, obviously have been designed for medium sized training, but you can scale up in the future with.47:44No issues at all. You can select it for the HPC for the traditional workloads like like the visualization and for for data analytics, I think it's a really good start point for many companies, this server. Here you can see the physical layout.48:02With plenty of data, you don't need to remember it with our PCIe is west options that you have here, the different GPUs supported by the, by the server.48:21Double check this information, please, if you want to know more about this server. And here you have one of the most important slides in this, in this presentation. The portfolio of Nvidia is.48:38Is huge. We support plenty of them, the H100, the most typical, the the widely used GPU used by our customers, the eight 200 the L forties, all of those with different use cases, let's say or or use case for.48:55E for each piece and finally the the AMD and E and A 210, ok, for the deep learning training, we support both. Once again the the the.49:14How to select the GPU is, is a key thing for you in the design of your infrastructures, ok? So double check after the the the the characteristics of the of all of those NVIDIA after after this session, ok? To select the the the perfect.49:30Choice for you, ok? In in this case we are showing up the the whole how would be the configuration of the of the network connectivity in this specific server, it's pretty similar to the to the big compared with the big guys you can divide the front end, the.49:48Storage the backend and the management network in the, in the same server and how easily is to scale up in the future. So that's important in a server like this as I said, starting from, from small.50:08And here we are trying to aware our customers about the how to populate the the GPUs. In the right hand side you can see it's not recommended. It's supported but not recommended. The idea is to use the GPU from the slot from the S five for.50:23The slot five to the SS8, ok? So we are using the same this the we are.50:43Using the the yeah checking the throughput of this traffic to the same of this data to the same CPU. It's supported but not recommended, ok? In this case with four GPUs, by the way, you can see on top of the, on top of the.50:55Of the slide. And here, I tried to, I added these slides with the MV MV link to show up how to configure your GPUs with the MV link too of of Nvidia.51:11How those TPUs are connected using the the MV MV link of of breach on video in this in the right hand side you have the non rec not recommended because we are using two different PCA switches, so.51:26Cisco is trying to communicate to your customers, to the customers that you must populate those the traffic in this in one specific PCI switch, ok? So in this case you have you can feel the difference between those.51:44Images, ok?52:03Here we are using the H1H 200. Okay, in the in two in this case we have eight GPUs with two MV link breach and we are using we have the same, the, the most recommended or the the the best pract the best practice would be to.52:05Populate the GPUs in one MV link that we have here. In the right hand side is not recommended but is supported again, ok? But the idea especially when you are growing, when you are growing up your infrastructure is to use the same MV link envy link for.52:25Or with those four GPUs, ok? So not combining. I think more or less is, is clear.52:45All of this deployment can be done with Cisco intercite. It's our management infrastructure infrastructure management platform. You you can automate all of this deployment, including obviously operating systems virtualization environments, and so on, can be done.52:53By by inter site. You will be able to automate the deployment configure those servers as one so you are, we are trying to avoid the the the the human error configuration.53:12If something is going wrong, the configuration is the same, the same for all of them. So, it's the widely used optimized platform to operate our infrastructure, doesn't matter if we are talking about the X series.53:27Exeries with GPUs that our Cisco air ready solutions and so on and so on and so on, including the rack and blade servers, ok? So we can run any workloads from a very normal.53:47A Windows server to a very complex AI infrastructure, ok? So check it out our inter site after the session. Let's move a little bit to the, to the network side. Obviously the the data center portfolio of.54:01Of Cisco is the best of the best. We have the high performance switches switching fabrics for the workloads. Our Nexus 400 gb is fantast are fantastic. And in the right hand side you can see the 100 gb or two for the.54:16Cloud scale switches, ok?54:37And the different design options that you have. Obviously with Nexus dashboard, you can control your Nexus 909000, of course. You probably already know Nexus dashboard and right now with Cisco hyper fabric, the Cisco you can control.54:40And simplify your management and operations with those networking infrastructure for the, for the AI, ok? Here in the designing of the, of the AI infrastructure in terms of networking, ok? The the we have the.55:00Backend and the and the front end, the backend is based in the MPGP.55:20VXLAN architecture for dedicated training workloads, of course, and the front end we have the the VXLAN EVPN for ACI front end fabrics. We are trying to.55:25Let's say provide a connect connectivity across top of the rack switches that connected to the compute and storage domains, is the next section, the the storage part out here.55:40We have plenty of partnerships with our friends of netapp's, NetApp, e.g. pure storage, Hitachi, Bast, for the, for the machine learning infrastructure, ok? For the local storage, we trust in the software defined solutions like Red Hat openshift.55:57Nutanix, we have a new partnership with with Nutanix and and obviously you can trust on the local storage of our Cisco UCS with nbme storage, e.g., that's fantastic for the for the for the for our AI infrastructure.56:17Here you can see the different, by the way, you can check it out all of this configuration with in this case with past, with NetApp, with pure storage now over pure. Please check it out the Cisco validated designs, the cbts after the session.56:35Because those are the most powerful documents of the designing in Cisco. You can trust in those designs because they will.56:55Did you or they will give you all information needed and recommended and fully supported by the way, not only recommended but fully supported by Cisco to deploy your infrastructure including MML, but I'm talking about other different workloads.57:07With VMware, e.g., with Oracle rag, et cetera etc. And you can find out those documents in the Cisco validated design website where you can, where you can trust us in these designs, ok?57:27With bast, with NetApp, with everpure, you will be able to deploy your workloads, doesn't matter the the use case. I would like to talk about the the scale out, the scale unit definition. We have three different, let's say three plus one.57:42Types of scaling in your AI infrastructure. Here, e.g., we have.58:0332 seven where you can scale up up to 64 using the Cisco Nexus 9000 of course with the UCS. In this case we are using the 885. Okay ok we are talking about a very huge.58:13Architecture of of Cisco, of course, but this is the best way to show you how to scale up in the future using this configuration with two Cisco Nexus 9000, ok?58:30You can grow up up to 16 notes in the right hand side with 128 GPUs using the same scale unit, let's say, and how to grow up. It's fantastic and pretty easy.58:48Okay and pretty to the and pretty easy to deploy with inter site and pretty easy to control the your infrastructure, your network infrastructure with Nexus dashboard. So as I said before, we are trying to be with you in the life cycle of this deployment.59:04Here you can see the, the, the type one auto scale auto 128 GPUs from is with a with a training fabric, ok? 32 plus 32.59:23With the same configuration. This is another way to grow up your infrastructure using the, this, those spine those Cisco nexus, and 9300 yeah with different modules, let's say modules, ok? With 32 each of them.59:38It depends of you depends of your infrastructure infrastructure and how to and how when you want to grow up in the future, your infrastru infrastructure, but this is the this is another way. The type two would be in this case with the 885.59:58We are growing up from 64 using those spine and and leaf switches and in both sides 9000 and we are growing up 64 plus 64 plus 64. Okay, so this is the best way.01:00:14To avoid problems in the infra infrastructure design because every model is the same one and you already know who to deploy 2nd one, the 3rd one and the 4th one. Okay, this is the the AI ports that we can provide.01:00:30And here in the types in the type three, we are scaling up up to 256 CPUs. Obviously this is a super huge deployment. We are not doing it every single day, of course, but this is the best way to, the th the.01:00:48The best design to grow up to 256 GPUs ok so this is a a total of of 128 plus 128, ok?01:01:08I think it's, is not a secret but we need to define from scratch how we need to create this grow up in the future. And this is a, this is just an example about the AI training design and how this is just an example repeat. We have this scale unit.01:01:20Type two with two with Cisco UCS 885 and eight, ok, and how.01:01:40In this case, we have a dedicated for the training fabric in the back end and the enterprise data center, we have a front end em so we are scaling up from eight GPUs to 64 GPUs, ok? So this is a scale out. It's non blocking by the way back in fabric so that's.01:01:49That's important, not to be blocked.01:02:09And each of those UCS connect to the fabric front end fabric for the storage, for the, for the storage access and and the management. As you can see here on top we have our Cisco inter site to deploy to create this deployment on the next dashboard, ok? In both cases, we.01:02:11We can.01:02:31Control or manage this both infrastructures, ok? Here, do we have our hybrid design, all of these are examples of course you don't need to follow up these specific examples, but this here we are creating a hybrid design is non blocking for training.01:02:35And and fine and fine tuning, we have our converged fabric for training and fine tuning inference using this.01:02:53Servers that we have, the the 845, we try to explain all of those examples with different types of servers for your own understanding and trying to cover the, the, the whole use cases that you have and and obviously the budget that you have for those.01:03:02Servers, each of those UCS are connected between all of them. So.01:03:22It's the best way to get this hybrid design, ok? This is slightly different. Here you have a different example of the AI inferencing design with with UCSX series and and C series, the, the, the, the, the rack servers, ok?01:03:29So we have, it's we are using the same the same design, the same architecture with different servers, with different servers, but we are still getting the same AI inferencing design for, for both of them.01:03:46In this, in this session, we try to explain you how to, in this case we are putting focus in the red Hat openshift, solution to define.01:04:03Or to deploy your machine learning infrastru infrastructure, ok? Maybe in the future we will use another another in another session we will use another solution, but in this case we try to put focus in the Red Hat openshift. We already have a Cisco validated design with Red Hat openshift. In the past, if I'm not wrong, we have another.01:04:17Decision try to find out on Google.01:04:37Because we already have another session about how to deploy with UCS. In this case we will put focus in the machine learning infrastructure, ok? So we are is the best along with Intersight is the best way to deploy.01:04:42Your, your Kubernetes ecosystem, ok, to embrace these containers and and be portable easy and and and how to auto scale capabilities, ok? So we have the operator operators which provide the the the.01:05:00Framework to, to add those new capabilities including obviously the mbda GPUs and the storage, and we can use it for the for the virtualization.01:05:20Layer the the nvidia, will provide this virtual GPU capability which is important in in the enabled those capabilities is important are important in the.01:05:32Individual GPUs when you are using the virtualization. And finally, if I'm not wrong, this is the 3rd question for you guys. What is your biggest challenge when you are, if you explore the how to scale in the AI infrastructure, we have different.01:05:45Things. For those who answered the before that we are exploring the AI, this is a good question because this is where you need to put focus and in your.01:06:05In your investigation, let's say or exploration about the AIs, ok? Infrastructure complexities is a challenge. That's the way that's why we are pushing why we are putting focus in the.01:06:21In the, in how Cisco is trying to simplify the operations and the deployment in in our infrastructures, ok? Because budget, it's important indeed the GPUs are not.01:06:38Cheaper and the poor and cooling indeed, good. Let's continue with the ML ops. As you can see the the this is once again this is a life cycle.01:06:53Of the machine learning operations with the streamlined from automation diversion control, how to accelerate and and simplify this life cycle is one of the most important and you need to follow up this life cycle in the future if you want to.01:07:13Create your machine learning and operations and the machine learning operations, ok?01:07:34So here the the the the platforms from with Red Hats OpenShift and OpenShift and machine learning operations they we are trying to provide the scalable foundation for the AI efforts, ok? So we are we have some examples like.01:07:40For ML flow ML ML flow, ok? So we are providing the providing the the the the futures and tools to different status of of a machine learning life cycle.01:08:00From the, from the data management and monitoring to deployment and and and the monitoring, all of those features are extremely important in the, in our machine learning operations, ok? Here we have the the idea of the Red Hat OpenShift AI should.01:08:12Solution is to simply, like we said, is simplify the the the process from moving to models from.01:08:32From scratch to production environment to this growing up trouble the idea is to simplify this, this trouble, the flexibility to choose the the the and how to adapt to the latest innovation of the AI technologies.01:08:42And unified the, the, the operational environment as you can see across the the on premise cloud and and edge, ok? We are using a single platform for the application and and and deliver this, this model.01:09:01The idea of OpenShift is the in with AI is the is that you can is to give you the chance to to run anywhere OpenShift, ok, where you where doesn't matter in in which platform, ok?01:09:18And those GPUs will be provisioned and and managed by, by OpenShift. All of these all of these services will be managed by by openshift. So we are trying to deliver the enterprise class application platform, ok, deploying, as an operator multitask.01:09:38In a multi tenant environment with security policies and and those resources available off to to OpenShift AI, ok?01:09:57Here you have the the the the different goals that we have depending of the of the profile of if we are the the business leader or the machine learning engineer, we have different stage, let's say stage or experience to provide with, with Red Hat openshift.01:10:08Okay.01:10:28Here you have the different ecosystem that we have with openshift, with the obviously with the model development training and tuning with the pie charts with kuda. And at the bottom you have the different operators of of OpenShift with the pilot.01:10:31Times with etc et cetera, including including Prometheus by the way.01:10:51Here you have the the how to train and serve the, how to integrate the easy the the this deployment with plus nvidia. So here you have the different how to accelerate the the training time with rapid soul and and.01:10:58That openshift, the tensor flow of course to leverage the the GPU optimizer deep deep learning, ok, and standard frameworks.01:11:18With the capabilities to support the the model execution with the new models, ok? So you have this all of this ecosystem of OpenShift with the with a with an AI solution in a AI solution, ok?01:11:30And let the the the red Hat openshift, AI ready infrastructure is trying to cover those pillars we said before the the predictive and and generative AI to support those those use cases at the scale.01:11:49And make it easy the the the operation and try to be consistent in the operation. Here we are including the Cisco inter site for the infrastructure and GPU deployment and then the open shift will be in charge of the of.01:12:07The Kubernetes deployment, the GPU operator.01:12:27And the machine learning operators. So we have Cisco in the infrastructure part and OpenShift for the Kubernetes and machine learning operations. Okay, so finally for the wrap up, I would like to summarize all of these things we have been.01:12:30That we have been talking today.01:12:50And obviously the challenging of of to deploy the an AI solution is not easy. We know about this thing. As I said, you can contact with us in the future if you need to know more about how to design or to define your use case and how to solve all of those challenges.01:12:52And the idea is to accelerate all of those things with the AI ready infrastructure of Cisco along with, with or without openshift, ok? And how to define how to define your design to select and your the best capacity planning for you.01:13:11Simplified the operations of this deployment, it's critical in in our opinion with inter site with with you and with our partnership you can accelerate this deployment and this training for your AI models.01:13:27Okay, so as you can see here, you, we.01:13:47As I said before, we are trying to cover the full life cycle, the most important here is our AI solutions and CBDs you can find out on, on cisco.com to know more about these designs and how these documents can help you in the in the.01:13:51Those designs, ok? And as I said, we are trying to cover the, this, the, all of this spectrum, ok? In the Cisco AI portfolio. Here you can, here you can find out the CBDs for the.01:14:11For the, for your infrastructure the the data center design zone, you can find out that that the the all of these guides for for these deployments, ok? And the GitHub report that we have also.01:14:30Oh, here. So if you need more information, you can contact with me in in this email, ok? Then we can help you in this, in this design, in this design with AI.01:14:47Okay, so thank you very much for joining us today, for the future you will have the next the as I said before, the next week we have the Cisco UCS another Cisco UCS session. So thank you very much for joining us today.01:15:02