Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0

 

Published: May 2025

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: http://www.cisco.com/go/designzone

Executive Summary

Cisco Validated Designs (CVDs) consist of systems and solutions that are designed, tested, and documented to facilitate and improve customer deployments. These designs incorporate a wide range of technologies and products into a portfolio of solutions that have been developed to address the business needs of our customers.

Generative Artificial Intelligence (Generative AI) stands as a transformative force across every industry, driving innovation in multiple use cases. Despite opportunities, integrating generative AI into enterprise settings poses unique challenges. Leveraging internal data effectively is critical for on-premises AI solutions. Building the right infrastructure with appropriate computational resources is critical. Visibility and monitoring of the entire stack is important from the operations point of view.

The Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 solution takes the complexity out of adopting generative AI by providing prescriptive steps for deploying the underlying infrastructure for Nutanix GPT-in-a- Box 2.0. This solution combines Cisco® servers and SaaS operations with Nutanix software, utilizing the most popular Large Language Models (LLMs) to produce a fully validated AI-ready platform that can simplify and jumpstart your AI initiatives from the data center to the edge.

This document explains the Cisco Validated Design and Deployment for GPT-in-a-Box 2.0 on Cisco Compute Hyperconverged with Nutanix to deploy on-premises Generative AI applications. This solution outlines the design that supports the deployment of an innovative, flexible, and secure generative pretrained transformer (GPT) solution for Generative AI to privately run and manage organization’s choice of AI large language models (LLMs) and applications leveraging it.

Solution Overview

This chapter contains the following:

●     Introduction

●     Purpose of this document

●     New in this release

●     Audience

●     Solution Summary

Introduction

Generative AI is reshaping industries, from dynamic marketing content to interactive virtual assistants and chatbots. However, quick  provisioning of models, management, and monitoring of secure LLM end points across several teams within an organization remains a challenge. It is critical to effectively import and deploy certified LLM models and use the internal data of the organization in these applications.  A robust infrastructure, observability across the stack, optimized model deployment and serving, high availability, and scaling are some of the other challenges which IT Administrators are facing today.

The solution highlights how enterprises can design and deploy Generative Pretrained Transformer (GPT) solution for Generative AI to privately run and manage organization’s choice of AI large language models (LLMs) and applications leveraging it. The solution effectively focusses to import, generate secure LLM end points and monitor the inference end points provisioned across the organization.

The hardware and software components are integrated so that customers can deploy the solution quickly and economically while eliminating many of the risks associated with researching, designing, building, and deploying similar solutions from the ground up.

Purpose of this document

This document explains the Cisco Validated Design and Deployment details for GPT-in-a-Box 2.0 solution on Cisco Compute Hyperconverged with Nutanix. The solution presented in this document will address design, reference architecture, and deployment  and validates Nutanix Enterprise AI stack providing ease of importing models and generating secure inference end points through NAI dashboard. 

This Cisco Validated Design is just one example of a supported GPT configuration. You can design and build a GPT solution in many ways, and you can deviate from this specific configuration while still following CVD best practices.

New in this release

GPT-in-a-Box 2.0 provides several new features through the latest offerings from Nutanix Cloud Platform and validated on the industry’s first hyperconverged solution using a modular server architecture:

●     An enterprise Grade inference endpoint provided by Nutanix Enterprise AI which provides an easy way to deploy your choice of LLMs (large language models) from leading LLM providers and create and manage secure APIs to connect your GenAI applications.

●     An enterprise-grade Kubernetes solution Nutanix Kubernetes Platform Integrating seamlessly with Nutanix's hyperconverged infrastructure, NKP is ideal for businesses aiming to accelerate application modernization while maintaining operational consistency and scalability.

●     Validated on Cisco Compute Hyperconverged X-Series System combines the operational simplicity of the Nutanix Cloud Platform with the flexibility and efficiency of the award-winning Cisco UCS X-Series Modular System, enabling organizations to easily deploy, scale, and upgrade hyperconverged clusters with a more sustainable, future-ready solution.

●     The solution supports Cisco UCS X-Series Direct to address all your edge, retail, and small and remote-office use cases.

Audience

The intended audience of this document includes IT decision makers like CTOs and CIOs, IT architects and customers who are working on or interested in design, deployment, and life cycle management of generative AI systems and applications.

Solution Summary

GPT-in-a-Box 2.0 on is a new turnkey solution that includes everything needed to build AI-ready infrastructure. AI applications can be easily deployed on top of GPT-in-a-Box 2.0.

Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 solution is a reference architecture that combines:

●     Nutanix GPT-in-a-Box 2.0 software-defined solution

●     Cisco Compute Hyperconverged X-Series modular system

●     Cisco UCS X210c M7 All-NVMe server with Cisco UCS X440p PCIE node configured with 2x NVIDIA L40S GPU configured and managed through Cisco Intersight

●     Nutanix Prism Central for deploying and managing solution software components

●     A range of the Nutanix certified large language models which can be easily imported through Hugging Face or Nvidia NGC Catalog

GPT-in-a-Box 2.0 solution software component includes:

●     Uses NKP as Kubernetes Platform and Nutanix Unified Storage (NUS)

●     Leverages Nutanix Enterprise AI

●     Integrates with Nutanix Unified Storage

Some highlights of the solution include the following:

●     AI-ready platform to enable customers to quickly design, size, and deploy an on-premises AI solution

●     Quickly deploy off-the-shelf AI applications or empower developers to build their own

●     A single-cloud operating model using Nutanix Cloud Platform hyperconverged infrastructure with graphic processing units (GPUs)

●     Nutanix Unified Storage (NUS) for total data management, security, privacy, and resilience

●     Generate secure inference end points with imported LLM models deployed on GPU pods

●     Support for popular AI/ML frameworks

●     LLM freedom of choice through custom imports enabled though Nutanix Enterprise AI

Generative AI: Concepts and Components

This chapter contains the following:

●     What is Generative AI?

●     What is Generative AI Inferencing?

●     Large Language Models

●     Generative AI Workflow

This chapter explains various concepts of Generative AI, including model development workflow, inferencing challenges and use cases.

What is Generative AI?

Generative AI is a powerful branch of artificial intelligence that holds immense potential for addressing various challenges faced by enterprises. With generative AI, users and applications can quickly generate new content based on a variety of inputs; inputs and outputs to these models can include text, images, sounds, animation, 3D models, or other types of data. Due to the versatility of generative AI models, applications leveraging them can perform multiple tasks based on available data and inputs, increasing functionality beyond just text and image generation or chat-based Q&A.

How Does Generative AI Compare to Traditional AI?

Generative AI can create new content, chat responses, designs, synthetic data, and more. Traditional AI, on the other hand, is focused on detecting patterns, making decisions, honing analytics, classifying data, and detecting fraud.

As more organizations recognize the value of using AI to create new content, they’re now exploring large language models (LLMs) and other generator models. Since pretrained LLMs are available, known as foundation models, adopting generative AI requires less upfront training compared with traditional AI models. This results in significant cost and time savings when developing, running, and maintaining AI applications in production.

While 2023 has been the year of Generative AI with the introduction of ChatGPT and models like Stable Diffusion, the technology has been in development for some time. NVIDIA and other companies have been researching and innovating in this space for years, which has helped lead us to where we are today. Examples include StyleGAN (2018), which creates realistic images of people, and GauGAN (2019), which allows you to create fingerpaint-style images that instantly become realistic landscapes. NVIDIA has released an app based on this research called Canvas, and these technologies have been used broadly by ecosystem partners.

What is Generative AI Inferencing?

Generative AI inferencing refers to the process of using a trained generative AI model (large language models and non-large language models) to generate new data or content based on input or contextual cues. During inferencing, the model applies its learned knowledge to produce outputs that are not direct repetitions of the training data but are rather novel creations generated by the model.

The inferencing process is crucial for leveraging the generative capabilities of the models in practical applications. It allows users to obtain novel outputs by providing inputs or guiding the model's behavior based on specific requirements or constraints. The generated content can be used for various creative purposes, prototyping, or as a tool for exploration in different domains.

The term "inferencing" in the context of generative AI is associated with generating content like:

●     Text Generation

◦     Storytelling: Generative models can create fictional stories, narratives, or even entire chapters of books.

◦     Poetry and Prose: AI models can generate poetic verses, prose, or creative writing.

◦     Dialogues: Conversational agents powered by generative models can produce human-like dialogues.

●     Image Generation

◦     Artistic Creations: Generative Adversarial Networks (GANs) can generate visually appealing and artistic images.

◦     Style Transfer: Models can transform images into different artistic styles.

◦     Face Synthesis: GANs can create realistic faces of non-existent individuals.

●     Music Composition

◦     Melody Generation: AI models can compose original melodies and music.

◦     Genre-specific Music: Generative models can create music in specific genres, mimicking different styles.

●     Code Generation

◦     Source Code: AI models can generate code snippets or even entire programs based on a given task or description.

●     Language Translation

◦     Multilingual Text: Models like OpenAI's GPT can generate text in multiple languages.

◦     Translation: AI models can translate text from one language to another while preserving context.

●     Content Summarization

◦     Text Summaries: Generative models can summarize large blocks of text into concise and coherent summaries.

●     Content Completion

◦     Sentence Completion: AI models can complete sentences or paragraphs in a way that fits the context.

◦     Text Expansion: Generative models can expand on given ideas or concepts.

●     Product Descriptions

◦     E-commerce Descriptions: AI models can generate product descriptions for e-commerce websites.

●     Scientific Writing

◦     Research Abstracts: Models can generate abstracts or summaries of scientific research papers.

●     Conversational Agents

◦     Chatbot Responses: AI-powered chatbots can generate responses in natural language during conversations.

Large Language Models

Generative AI is a broad category that includes models designed to generate new and original content. This content can be in various forms, such as images, text, audio, or even video. Large language models are a specific subset of generative AI designed to understand and generate human language. They are primarily focused on natural language processing tasks.

Large language models (LLMs) are a class of natural language processing models which uses deep learning methodologies to comprehend and generate human language. These models are trained in vast amounts of textual data to learn the patterns, structures, and nuances of language.

One of the notable examples of LLMs is the GPT (Generative Pre-trained Transformer) series developed by OpenAI.

Key features of large language models include:

●     Scale: LLMs are characterized by their large number of parameters, often ranging from tens of millions to billions. The scale of these models allows them to capture complex linguistic patterns and generate diverse and contextually relevant text.

●     Pre-training: LLMs are typically pre-trained on a massive corpus of text data before being fine-tuned for specific tasks. During pre-training, the model learns to predict the next word in a sentence or fill in missing words, which helps it acquire a broad understanding of language.

●     Transformer Architecture: LLMs, including GPT, are built on the Transformer architecture, which enables efficient processing of sequential data. Transformers use self-attention mechanisms to capture relationships between words in a sentence, facilitating better context understanding.

●     Transfer Learning: LLMs leverage transfer learning, where the knowledge gained during pre-training on a general language understanding task is transferred to specific tasks with minimal additional training. This approach allows these models to excel in a variety of natural language processing (NLP) applications.

●     Fine-tuning: After pre-training, LLMs can be fine-tuned for specific tasks, such as text classification, language translation, summarization, and more. This fine-tuning process adapts the model to the nuances of the target application.

●     Diverse Applications: Large Language Models find applications in a wide range of tasks, including but not limited to natural language understanding, text generation, sentiment analysis, machine translation, question answering, and chatbot development.

The development of Large Language Models has significantly advanced the field of natural language processing, enabling the creation of sophisticated AI systems capable of understanding, and generating human-like text across various domains. However, ethical considerations, biases in training data, and potential misuse are important considerations associated with the deployment of these models.

Model Parameters

Model parameters are the internal variables or weights that the model learns during the training process. Weights are the coefficients that scale the input features in a neural network. In the context of LLMs, these weights determine the strength of connections between neurons in different layers. For example, in a transformer model, weights are associated with the attention mechanisms and transformations applied to input sequences.

LLMs often consist of multiple layers, each with its set of weights and biases. In transformer architectures, these layers may include self-attention mechanisms and feedforward neural networks. The parameters of each layer capture different aspects of the input data.

The total number of parameters in an LLM is a critical factor in its capacity to capture complex language patterns and nuances.

Generative AI Workflow

Typical Generative AI workflow starts with aligning to business objectives while maintaining a concise and accurate technical focus in every stage.

Business Strategy and Use Case Definition: Define generative AI objectives aligning with business goals.

●     Key Tasks

◦     Identify use cases.

◦     Clearly define the generative task, whether it's image generation, text generation, style transfer, etc.

◦     Establish goals and success metrics.

Data Preparation and Curation: Ensure high-quality, well-managed dataset availability.

●     Key Tasks

◦     Gather a diverse and representative dataset for training the generative model.

◦     Data cleansing and labeling.

◦     Data aggregation and preprocessing.

◦     Increase the diversity of the training data through techniques like rotation, scaling, or flipping.

◦     Anonymization or synthetic data generation if required.

◦     Leveraging MLOps platforms for efficient data management.

Model Training: Utilize accelerated infrastructure for efficient training.

●     Key Tasks

◦     Training from scratch or selecting pretrained models.

◦     Allocating heavy computational resources.

◦     Optimizing performance with validated, high-performance infrastructure.

Model Customization: Fine-tuning, prompt learning (including prompt tuning and P-tuning), transfer learning, reinforcement learning.

●     Key Tasks

◦     Adapt pretrained models to specific business needs.

◦     Implement customization methods based on requirements.

Inferencing: Deploy and operate trained models for ongoing generation.

●     Key Tasks

◦     Scale computing resources (scaling up or out) based on demand.

◦     Iterate on inferencing based on new data and customization opportunities.

◦     Continuous monitoring of inferencing performance.

◦     Identification and optimization of opportunities for further customization and fine-tuning.

This workflow emphasizes technical aspects, highlighting the importance of infrastructure efficiency, model customization techniques, and ongoing optimization in the inferencing phase.

Technology Overview

This chapter contains the following:

●     Cisco Intersight Platform

●     Cisco Compute Hyperconverged X9508 Chassis

●     Cisco Compute Hyperconverged with Nutanix

●     NVIDIA GPUs

Cisco Intersight Platform

As applications and data become more distributed from core data center and edge locations to public clouds, a centralized management platform is essential. IT agility will be a struggle without a consolidated view of the infrastructure resources and centralized operations. Cisco Intersight provides a cloud-hosted, management and analytics platform for all Cisco Compute for Hyperconverged, Cisco UCS, and other supported third-party infrastructure deployed across the globe. It provides an efficient way of deploying, managing, and upgrading infrastructure in the data center, ROBO, edge, and co-location environments.

Cisco Intersight provides:

●     No Impact Transition: Embedded connector allows you to start consuming benefits without forklift upgrade.

●     SaaS/Subscription Model: SaaS model provides centralized, cloud-scale management and operations across hundreds of sites around the globe without the administrative overhead of managing the platform.

●     Enhanced Support Experience: A hosted platform allows Cisco to address issues platform-wide with the experience extending into TAC supported platforms.

●     Unified Management: Single pane of glass, consistent operations model, and experience for managing all systems and solutions.

●     Programmability: End to end programmability with native API, SDK’s and popular DevOps toolsets will enable you to deploy and manage the infrastructure quickly and easily.

●     Single point of automation: Automation using Ansible, Terraform, and other tools can be done through Intersight for all systems it manages.

●     Recommendation Engine: Our approach of visibility, insight and action powered by machine intelligence and analytics provide real-time recommendations with agility and scale. Embedded recommendation platform with insights sourced from across Cisco install base and tailored to each customer.

For more information, go to the Cisco Intersight product page on cisco.com.

Cisco Intersight Virtual Appliance and Private Virtual Appliance

In addition to the SaaS deployment model running on Intersight.com, you can deploy on-premises options separately. The Cisco Intersight virtual appliance and Cisco Intersight private virtual appliance are available for organizations that have additional data locality or security requirements for managing systems. The Cisco Intersight virtual appliance delivers the management features of the Cisco Intersight platform in an easy-to-deploy VMware Open Virtualization Appliance (OVA), Microsoft Hyper-V Server or a Nutanix AHV virtual machine, which allows you to control the system details that leave your premises. The Cisco Intersight private virtual appliance is provided in a form factor designed specifically for users who operate in disconnected (air gap) environments. The private virtual appliance requires no connection to public networks or to Cisco network.

Licensing Requirements

The Cisco Intersight platform uses a subscription-based license with two different tiers. You can purchase a subscription duration of 1, 3, or 5 years and choose the required Cisco UCS server volume tier for the selected subscription duration. You can purchase any of the following higher-tier Cisco Intersight licenses using the Cisco ordering tool:

●     Cisco Intersight Essentials: Essentials includes all the functions of the Base license plus additional features, including Cisco UCS Central software and Cisco Integrated Management Controller (IMC) supervisor entitlement, policy-based configuration with server profiles, firmware management, and evaluation of compatibility with the Cisco Hardware Compatibility List (HCL).

●     Cisco Intersight Advantage: Advantage offers all the features and functions of the Base and Essentials tiers. It also includes storage widgets and cross-domain inventory correlation across compute, storage, and virtual environments (VMware ESXi). OS installation for supported Cisco UCS platforms is also included.

Servers in the Cisco Intersight Managed Mode require at least the Essentials license. For more information about the features provided in the various licensing tiers, go to: https://www.intersight.com/help/saas/getting_started/licensing_requirements/lic_intro

Cisco Compute Hyperconverged X9508 Chassis

Cisco and Nutanix have partnered to introduce the industry’s first hyperconverged solution using a modular server architecture. The Cisco Compute Hyperconverged X-Series System combines the operational simplicity of the Nutanix Cloud Platform with the flexibility and efficiency of the award-winning Cisco UCS X-Series Modular System, enabling organizations to easily deploy, scale, and upgrade hyperconverged clusters with a more sustainable, future-ready solution. There is also support for Cisco UCS X-Series Direct to address all your edge, retail, and small and remote-office use cases.

The Cisco Compute Hyperconverged X-Series System with Nutanix combines the operational simplicity of Nutanix Cloud Platform (NCP) with the efficiency, flexibility, and sustainability of the Cisco UCS X-Series Modular System. The X-Series system comprises modular components that can be assembled into systems through the Cisco Intersight cloud-operations platform.

The Cisco Compute Hyperconverged X-Series System is engineered to be adaptable and future-ready. With a midplane-free design, the system achieves I/O connectivity, using frontloading, vertically oriented compute and accelerator nodes that intersect with horizontally oriented I/O connectivity modules in the rear of the chassis. In the front of the chassis, Cisco Compute Hyperconverged X210c M7 All NVMe Nodes with Intel® Xeon® Scalable Processors offer compute density and storage capacity in a single form factor and the Cisco UCS X440p PCIe Node supports adding GPUs with PCIe Gen 4.0 with the Cisco UCS X9416 X-Fabric Module.

In the rear of the chassis, a unified Ethernet fabric is supplied with Cisco UCS 9108 100G Intelligent Fabric Modules and the Cisco UCS X9416 X-Fabric Module, part of the Cisco UCS X-Fabric Technology, supply PCIe Gen 4 industry-standard protocols for GPU accelerators. Interconnections can be easily updated with new modules supporting faster Ethernet, PCIe Gen 5, and CXL.

Cisco Compute Hyperconverged X-Series Direct is a self-contained system with a pair of integrated fabric interconnects and can be used if you do not want top-of-rack fabric interconnects and need to support edge, retail, and small or remote-office use cases.

The X-Series system was designed with sustainability in mind and is equipped with power-delivery and cooling innovations designed to reduce material waste and energy consumption.

Features and Benefits

●     7-Rack-Unit (7RU) chassis has 8x front-facing flexible slots. These can house a combination of hyperconverged nodes, compute nodes, and a pool of future I/O resources that may include GPU accelerators, disk storage, and nonvolatile memory.

◦     2x 9108 25G Intelligent Fabric Modules (IFMs) at the top of the chassis that connect the chassis to upstream 6400 Series Fabric Interconnects or 6500 Series fabric interconnects. Each IFM features:

◦     Up to 100 Gbps of unified fabric connectivity per compute node.

◦     8x 25-Gbps SFP28 uplink ports / 8x 100-Gbps SFP28 uplink ports. The unified fabric carries management traffic to the Cisco Intersight cloud-operations platform, Fibre Channel over Ethernet (FCoE) traffic, and production Ethernet traffic to the fabric interconnects.

●     At the bottom are slots ready to house future I/O modules that can flexibly connect the compute modules with I/O devices. The connectivity is named Cisco UCS X-Fabric technology because the “X” is a variable that can evolve with new technology developments.

●     Six 2800W Power Supply Units (PSUs) provide 54 V power to the chassis with N, N+1, and N+N redundancy. A higher voltage allows efficient power delivery with less copper and reduced power loss.

●     Efficient, 4x100mm, dual counter-rotating fans deliver industry-leading airflow and power efficiency. Optimized thermal algorithms enable different cooling modes to best support the network environment. Cooling is modular so that future enhancements can potentially handle open-or closed-loop liquid cooling to support even higher-power processors.

Benefits

Since we first delivered the Cisco Unified Computing System™ (Cisco UCS) in 2009, our goal has been to simplify the data center. We pulled management out of servers and into the network. We simplified multiple networks into a single unified fabric. And we eliminated network layers in favor of a flat topology wrapped into a single unified system. With the Cisco Compute Hyperconverged X-Series System, we take that simplicity to the next level:

●     Simplified operations with a solution that combines the operational simplicity of hyperconverged software with the efficiency and flexibility of a modular system.

●     Increased agility and response to the dynamic needs of your business with a solution that is inherently easy to scale and includes support for future generations of processors, storage, accelerators, networking technologies, and SaaS innovations.

●     Improved sustainability with a solution that is engineered to be more energy efficient and can be easily upgraded and reused, lowering the consumption of power and raw materials when compared to traditional rack servers.

Cisco UCS X210c M7 All-NVMe/All-Flash Servers

The Cisco Compute Hyperconverged X210c M7 All NVMe Node delivers performance, flexibility, and optimization for deployments in data centers, in the cloud, and at remote sites. This enterprise-class server offers market-leading performance, versatility, and density without compromise for workloads. Up to eight hyperconverged nodes can reside in the 7-Rack-Unit (7RU) Cisco Compute Hyperconverged X9508 Chassis, offering one of the highest densities of compute, I/O, and storage per rack unit in the industry.

For more information, see the Cisco Compute Hyperconverged X210c M7 All NVMe Node Data Sheet.

Cisco UCS X440p PCIe Node

The Cisco UCS X440p PCIe Node is the first PCIe resource node to integrate into the Cisco UCS X-Series Modular System. The Cisco UCS X9508 Chassis has eight node slots, up to four of which can be X440p PCIe nodes when paired with a Cisco UCS X210c M6 Compute Node. The Cisco UCS X440p PCIe Node supports two x16 full-height, full-length dual slot PCIe cards, or four x8 full-height, full-length single slot PCIe cards and requires both Cisco UCS 9416 X-Fabric modules for PCIe connectivity. This provides up to 16 GPUs per chassis to accelerate your applications with the Cisco UCS X440p Nodes. If your application needs even more GPU acceleration, up to two additional GPUs can be added on each Cisco UCS X210c compute node.

Cisco UCS X440p supports the following GPU options:

●     NVIDIA H100 NVL, 400W, 94GB, 2-slot FHFL GPU

●     NVIDIA H100: 350W, 80GB, 2-slot FHFL GPU

●     NVIDIA L40S: 350W, 48GB, 2-slot FHFL GPU

●     NVIDIA L4 Tensor Core, 70W, 24GB

●     NVIDIA A16 PCIE 250W 4X16GB

For more information, see the Cisco Compute Hyperconverged 440p PCIe Node.

Cisco Compute Hyperconverged with Nutanix

Cisco and Nutanix have come together to offer the industry’s most simple, comprehensive HCI solution. The Cisco Compute Hyperconverged with Nutanix solution combines the Cisco Unified Computing System (UCS) innovative server, networking, and SaaS management with Nutanix’s leading HCI foundation, offering a fully integrated and validated system with flexible deployment options and a unified, enhanced support model backed by two world-class organizations.

The solution offers the following key benefits:

●     Complete simplicity: The solution offers both SaaS and on-premises management options and includes day-0 through day-N operations, including server profiles for compute, storage, and networking customized for Nutanix to help simplify and accelerate cluster deployment and deliver better performance and resiliency. This also includes preinstalled software configured with a choice of hypervisor for a faster, easier start. Cisco and Nutanix combined and complementary cloud operating models provide control, visibility, and consistency across highly distributed IT environments, including fully integrated cluster installation, expansion, and end-to-end software and firmware upgrades. To simplify the buying experience, the complete solution can be ordered and delivered from Cisco.

●     Complete flexibility: The Cisco Compute Hyperconverged with Nutanix solution addresses modern applications and use cases, offering multiple choices in UCS server deployment options, the latest accelerator and drive technologies, and SaaS innovations from two industry powerhouses, including integrations with the leading public cloud providers. Additionally, the solution incorporates Cisco's best-in-class networking technology, including Cisco ACI integrations, to enhance performance and resiliency for data-intensive workloads in hybrid cloud environments.

●     Complete resiliency: The joint solution utilizes only enterprise-grade components and offers augmented system protection with a collaborative support model and proactive, automated resilience and security capabilities. This includes integrated support systems and case notes for faster triage. Any time log files are uploaded, or case notes are generated, that information is shared between Cisco TAC and Nutanix support, enabling enhanced collaboration among support teams to resolve issues faster and provide an improved customer experience.

●     Our policy-based approach minimizes human error and configuration drift, resulting in consistent, reliable cluster deployments.

●     It also enforces overall security posture through centralized authorizations, preventing tampering of configurations.

NVIDIA GPUs

This solution provides a reference architecture for GPT-In-Box in the enterprises using  NVIDIA L40S GPUs.

NVIDIA L40S GPU

The NVIDIA L40S GPU is the most powerful universal GPU for the data center, delivering end-to-end acceleration for the next generation of AI-enabled applications—from gen AI, LLM inference, small-model training and fine-tuning to 3D graphics, rendering, and video applications.

The L40S GPU is optimized for 24/7 enterprise data center operations and designed, built, tested, and supported by NVIDIA to ensure maximum performance, durability, and uptime. The L40S GPU meets the latest data center standards, is Network Equipment-Building System (NEBS) Level 3 ready, and features secure boot with root of trust technology, providing an additional layer of security for data centers.

Table 1.        NVIDIA L40S Tensor Core GPU Specifications

Solution Design

This chapter contains the following:

●     Solution Overview

●     Infrastructure Design

●     Network Design

●     Cluster Design

●     Storage Design

●     Nutanix Files and Objects Design

●     Management Design

●     Security and Compliance

●     Kubernetes Cluster Design

●     Large Language Model Design

●     Backup and Disaster Recovery

Solution Overview

This solution provides a foundational reference architecture for AI-ready platform to enable you to quickly design, size, and deploy an on-premises AI solution. The key design components configured in this solution are elaborated in Figure 5.

This design includes the following hardware and software components:

●     Single availability zones (AZs) in a single region in an on-premises Cisco Compute Hyperconverged (CCHC) with Nutanix cluster.

◦     The CCHC with Nutanix (AHV) cluster deployed on Cisco X-Series chassis with 4x X210C All NVMe compute node each paired with UCS X440p PCIe node each equipped with 2x NVIDIA L40S: 350W, 48GB GPUs.

●     The CCHC with Nutanix cluster hosts the following key Nutanix services:

◦     Nutanix Unified Storage (NUS).

◦     Nutanix Kubernetes Platform designed to simplify deployment, scaling, and management of containerized applications across hybrid and multicloud environments, integrating seamlessly with Nutanix's hyperconverged infrastructure.

◦     An enterprise Grade inference endpoint provided by Nutanix Enterprise AI which provides an easy way to deploy your choice of LLMs (large language models) from leading LLM providers and create and manage secure APIs to connect your GenAI applications.

Infrastructure Design

The deployment architecture for Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 is detailed in the figure below. The entire Day 0 deployment is managed through workflow defined in Nutanix Foundation Central.

CCHC with Nutanix cluster for GPT-in-a-box 2.0 is managed through Prism Central deployed on the same Nutanix AHV cluster. Prism Central can also be deployed on a separate AHV based Nutanix Cluster.

The Cisco UCS X-Series system is connected to a pair of Cisco UCS 6536 Fabric Interconnect managed through Cisco Intersight.

Each X210c M7 All-NVMe server is configured with:

●     2x Intel I6442Y processor (2.6GHz/225W 24C/60MB).

●     1 TB DDR5 memory (32x 32GB DDR5-4800 RDIMM).

●     2x 240GB M.2 card managed through M.2 RAID controller.

●     6x 3.8 TB NVMe.

●     1x Cisco VIC 15420  enabling up to 50 Gbps of unified fabric connectivity to each of the chassis Intelligent Fabric Modules (IFMs) for 100 Gbps connectivity per server.

●     2x NVIDIA L40S: 350W, 48GB GPUs.

Go to the Bill of Materials (BoM) section for the complete specifications of the infrastructure deployed to validate this solution.

Figure 6 illustrates the hardware deployment architecture for Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box.

The solution features a single Nutanix cluster that hosts the following services, among others:

●     Nutanix management components, including Prism Central

●     NUS to provide NFS storage and S3-compatible storage capabilities

●     NKP cluster

●     Nutanix Enterprise AI

Software Revisions

Table 2 lists the software revisions for various components of the solution.

Table 2.        Software Revisions

Network Design

Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 utilizes 100GbE end-to-end network.

Figure 8 illustrates the network design used in this solution.

The key aspects of network design are as follows:

●     Each of the 2x Cisco UCS Intelligent Fabric Module (IFM) are connected to 8x 25G on Cisco UCS Fabric Interconnects.

●     Each node is equipped with 1x Cisco UCS VIC 15420 2x 50G mLOM.

●     The network ports in the hypervisor are configured in Active-Passive Mode. These are configured automatically during automated cluster installation.

●     The deployment in Intersight Managed Mode and does not support bond mode 4 (LACP).

●     Fabric Interconnect ports are separate from 100G Uplink ports and can be connected via the core LAN or through dedicated management network switch. 

VLAN Configuration

Table 3 lists the VLANs configured for this solution.

Table 3.        VLAN Usage

Table 4 lists the IP Address Assignment for this solution.

Table 4.        Virtual Machines

Cluster Design

The design incorporates a single GPU-enabled Nutanix cluster dedicated to GPT-in-a-Box 2.0 workloads that offers access to large language models (LLMs). Supporting management applications like Prism Central and file and object storage are hosted on this cluster.

Size the Nutanix GPT-in-a-Box 2.0 cluster Controller VM (CVM) with 16 vCPU and 64 GB of memory.

This design uses one region with a single AZ that hosts the GPT-in-a-Box 2.0 cluster. This solution doesn't use any replication targets to protect the workloads because the focus is to host the GPT workloads in a single cluster.

Table 5.        Cluster Design Decisions

Cluster Resilience

VM high availability ensures that VMs restart on another AHV host in the AHV cluster when a host becomes unavailable, either because the original AHV host has a complete failure or becomes network partitioned or because of an AHV host management process failure. When the AHV host where a VM is running becomes unavailable, the VM turns off; therefore, from the perspective of the VM operating system, VM high availability involves a full VM start cycle.

Table 6.        High Availability Configuration

Storage Design

Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 utilizes a distributed, shared-nothing architecture for storage.

Figure 10 illustrates the storage design used in this solution.

When creating the Nutanix AHV cluster, the following storage containers are automatically created:

●     NutanixManagementShare: Used for Nutanix features like Files and Objects and other internal storage needs; doesn't store workload vDisks

●     SelfServiceContainer: Used by the NCM Self-Service Portal to provision VMs

●     Default-Container-XXXX: Used by VMs to store vDisks for user VMs and applications

In addition to the automatically created storage containers, the following additional storage containers are created during the Nutanix Files and Objects deployment:

●     NTNX_<fileserver_name>_ctr: Provides storage for the file server instances, which provide NFS storage for the application tier

●     objectsd<uniqueidentifier>: Data container for Nutanix Objects

●     objectsm<uniqueidentifier>: Metadata container for Nutanix Objects

The Default-Container stores VMs and their vDisks. The additional containers for Nutanix Files and Objects are created throughout the deployment process of the respective components.

Data Reduction and Resilience Options

To increase the effective capacity of the cluster, the design enables inline compression on all storage containers. It doesn't use additional functionalities such as deduplication or erasure coding. Replication factor 2 protects against the loss of a single component in case of failure or maintenance.

Table 7.        Data Reduction Settings

Table 8 lists the information about the storage decisions made for this design.

Table 8.        Storage Design Decisions

Nutanix Files and Objects Design

Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 utilizes Nutanix Files providing high-performance, shared storage to applications using the NFS protocol. Nutanix Files temporarily stores the LLMs and makes them available across VMs and Kubernetes services.

Nutanix Objects provides S3-compatible storage capabilities to the application and enables users to upload new data models.

Nutanix Files and Objects run on the same Nutanix cluster as the GPT-in-a-Box 2.0 workloads.

 

Figure 11 illustrates the Nutanix Files and Objects design. The VM displayed in Figure 11 is an end user or jump host used to mount the NFS share and store the LLMs.

Nutanix Files and Objects have the following network requirements:

●     For maximum performance, the storage network for Nutanix Files uses the same subnet as the CVMs.

●     To maximize security, the client network connects to a separate subnet.

●     The GPT-in-a-Box 2.0 cluster that provides a single file server instance requires three storage network IP addresses and four client network IP addresses for its three file server VMs (FSVMs).

Nutanix Objects runs as a containerized service on a Kubernetes microservices platform, which provides benefits such as increased velocity of new features. Several of the required storage IP addresses are for functions related to the underlying microservices platform. You must also manage these networks. Nutanix Objects, with three worker nodes, requires seven storage network IP addresses and two client network IP addresses.

Each cluster provisioned by NKP has minimum IP address requirements. A Kubernetes cluster in a production-level layout with three worker nodes requires one static IP address and eight or more IPAM addresses.

The client network provides all IP addresses. You might need additional IPAM addresses for additional worker nodes.

For more information, go to: Nutanix Files, Nutanix Objects, and Nutanix Kubernetes Engine Network Design.

Table 9 lists information about the Nutanix Files and Objects decisions made for this design.

Table 9.        GPT-in-a-Box 2.0 Files Design Decisions

For more information, go to: Nutanix Files and Nutanix Objects Design Decisions.

Table 10.     GPT-in-a-Box 2.0 Objects Design Decisions

Management Design

Management components such as Cisco Intersight (cloud managed) Prism Central, Active Directory, DNS, and NTP are critical services that must be highly available. Prism Central is the global control plane for Nutanix, responsible for VM management, application orchestration, micro segmentation, and other monitoring and analytics functions.

This solution utilizes two key management plane:

●     Cisco Intersight (cloud managed) supports the entire Cisco UCS X-Series modular system. It enables server, fabric, and storage provisioning as well as device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection.

●     Prism Central was deployed as single VM on a separate Nutanix AHV cluster. Prism Central manages the following:

◦     Nutanix Files and Objects

◦     VMs

◦     RBAC

◦     Monitoring, observability, and auditing for the core Nutanix Services

●     Nutanix Kubernetes Platform is managed through commander dashboard

●     Nutanix Enterprise AI is managed through NAI dashboard

DNS Management

Name resolution and SSL certificate management are critical to deploying and managing Kubernetes clusters. In this solution, DNS root domain is hosted locally on a windows DNS server. Subdomains map the DNS and route the traffic to the Kubernetes clusters.

Monitoring

Monitoring in the solution falls into two categories: event monitoring and performance monitoring. Each category addresses different needs and issues.

In a highly available environment, you must monitor events to maintain high service levels. When faults occur, the system must raise alerts on time so that administrators can take remediation actions as soon as possible. This solution configures the Nutanix platform's built-in ability to generate alerts in case of failure.

In addition to keeping the platform healthy, maintaining a healthy level of resource usage is also essential to the delivery of a high-performing environment. Performance monitoring continuously captures and stores metrics that are essential when you need to troubleshoot application performance. A comprehensive monitoring approach tracks the following areas:

●     Application and database metrics

●     Operating system metrics

●     Hyperconverged platform metrics

●     Network environment metrics

●     Physical environment metrics

By tracking a variety of metrics in these areas, the Nutanix platform can also provide capacity monitoring across the stack. Most enterprise environments inevitably grow, so you need to understand resource usage and the rate of expansion to anticipate changing capacity demands and avoid any business impact caused by a lack of resources.

Monitor the Conceptual Design

In this design, Prism Central performs event monitoring for the Nutanix core infrastructure. This NVD uses syslog for log collection; for more information, see the Security and Compliance section. SMTP-based email alerts serve as the channel for notifications in this design. To cover situations where Prism Central might be unavailable, each Nutanix cluster in this NVD sends out notifications using SMTP as well. The individual Nutanix clusters send alerts to a different receiving mailbox that's only monitored when Prism Central isn't available.

Prism Element transmits the data to Prism Central using an API. Prism Central then transmits the log to Nutanix Pulse (using the Secure Sockets Layer) and the syslog server using the User Datagram Protocol (UDP). All components (PC, NUS and NKP) send the alerts to the monitoring system using SMTP.

Prism Central monitors cluster performance in key areas such as CPU, memory, network, and storage usage, and captures these metrics by default. When a Prism Central instance manages a cluster, Prism Central transmits all Nutanix Pulse data, so it doesn't originate from individual clusters. When you enable Nutanix Pulse, it detects known issues affecting cluster stability and automatically opens support cases.

The network switches that connect the cluster also play an important role in cluster performance. A separate monitoring tool that's compatible with the deployed switches can capture switch performance metrics. For example, an SNMP-based tool can regularly poll counters from the switches.

Table 11 lists the descriptions of the monitoring design decisions.

Table 11.     Monitoring  Design Decisions

Security and Compliance

Nutanix recommends a defense-in-depth strategy for layering security throughout any enterprise datacenter solution. This design section focuses on validating the layers that Nutanix can directly oversee at the control and data-plane levels.

Security Domains

Isolate Nutanix cluster management and out-of-band interfaces from the rest of the network using firewalls and only allow direct access to them from the management security domain. In addition, Nutanix recommends separating out-of-band management and cluster management interfaces onto a dedicated VLAN away from the application traffic.

Syslog

For each control plane endpoint (Prism Central), system-level internal logging goes to a centralized third-party syslog server that runs in the existing customer landscape. The system is configured to send logs for all available modules when they reach the syslog Error severity level.

This design assumes that the centralized syslog servers are highly available and redundant, so you can inspect the log files in case the primary log system is unavailable.

Certificates

SSL endpoints serve all Nutanix control plane web pages. In the deployment and validation, self-signed certificates are utilized. The existing solution can replace the default self-signed certificates with certificates signed by an internal certificate authority from a Microsoft public key infrastructure (PKI). Any client endpoints that interact with the control plane should have the trusted certificate authority chain preloaded to prevent browser security errors.

 

Table 12.     Security Design Decisions

Table 13.     Security Configuration References

Kubernetes Cluster Design

This conceptual design outlines a robust, highly available Kubernetes environment on the Nutanix platform, using Nutanix Kubernetes Platform (NKP) and integrating with essential Kubernetes tools (many of which come pre-canned with NKP) and practices for efficient and secure operations.

Table 14.     GPT-in-a-Box 2.0 Cluster with NKP License Design Decisions

In this design, a single NKP Ultimate Management Cluster is used to manage and push out workloads and configuration to multiple Workload Clusters via NKP Application Catalogs Apps and NKP Continuous Delivery (CD) mechanism. The NKP Management cluster is used for management only. NAI and its supporting components run on one NKP workload cluster with GPU node pools. GenAI Apps and supporting components run on another workload cluster(s). The data resides locally within these clusters or in a S3 bucket (Nutanix Objects) or NFS share (Nutanix Files). All production Kubernetes clusters are set up as production-level clusters using multiple control plane and worker nodes distributed across different physical hosts to ensure high availability.

When a NKP cluster is built, the following preconfigured Cloud Native tools are deployed automatically for NKP's internal use:

●     Certificate Management - Cert-Manager

●     Ingress Controller - Traffic

●     Authentication - Dex

●     CD - FluxCD

●     Kubernetes Dashboard

●     Policy Admission Webhook - Gatekeeper

●     Cost Management - Kubecost

Table 15.     GPT-in-a-Box 2.0 Cluster with NKP Workspace & Projects Design Decisions

The following NKP workspaces are created and the NKP workload cluster(s) are deployed to:

●     NAI Workspace: One NAI Workload Cluster deployed to this workspace

●     GenAI Workspace: One or more GenAI Workload Clusters deployed to this workspace

The Common NKP Workspace Applications explicitly enabled from the NKP Application catalog in both Workspaces (NAI and GenAI) are:

●     Logging - Logging Operator, Grafana Logging, Grafana Loki, Fluent Bit

●     Monitoring - Prometheus Monitoring, Prometheus Adapter

●     Backup & Recovery - Velero with Nutanix Objects as the S3 Compatible Object Store used for storage

●     Object Store - Rook Ceph, Rook Ceph Cluster

The NKP Workspace Applications explicitly enabled only on the NAI Workspace are:

●     Istio Service Mesh

●     NVIDIA GPU Operator

●     Knative

●     Nutanix Enterprise AI

The GenAI App Project is created in the GenAI Workspace and used to deploy the GenAI app leveraging its Continuous Delivery (CD) mechanism. The cluster(s) in the GenAI Workspace is/are explicitly attached to this project.

Table 16.     GPT-in-a-Box 2.0 Cluster with NKP Scalability Design Decisions

The workloads are divided into NKP Management, GPT-in-a-Box 2.0 NAI, and Production/Development environments. Since there is only a single NKP management cluster, it is named <custom-prefix>-nkp-mgmt. Nutanix Enterprise AI is running in a dedicated cluster in the NAI workspace <custom-prefix>-nai. The following naming convention is used for workload clusters: <custom-prefix>-<cluster_type>-<environment_id>-wl-<app_id>-<optional_app_index_number>.

In this design, the NKP Management or Workload cluster's worker nodes can be scaled to accommodate different workload sizes. Nutanix recommends scaling out. The total number of workers in the GPU node pools is constrained by the number of physically installed GPUs.

All GPUs are licensed to run NAI, but this design will keep some GPU in spare for dev/experimental workloads:

●     NKP Management Cluster:

◦     cvd-nkp-mgmt

●     GPT-in-a-Box 2.0  NAI Environment

◦     cvd-nai (2 GPU nodes with 2 L40s)

●     Prod-Environment:

◦     cvd-nai-wl-01

Kubernetes Resilience

NKP clusters are production-level clusters that provide a resilient control plane by running multiple nodes for the control plane and etcd. To ensure high availability, Kubernetes deployments use multiple replica pods and implement pod anti-affinity rules. This approach helps maintain service availability, even in the event of a worker node update or failure.

Kubernetes services and ingress controllers perform the essential service of load balancing by evenly distributing network traffic across all available pods, enhancing service reliability and system performance.

Kubernetes Networking

Each deployed cluster uses a base configuration for networking:

●     kube-vip: Load-balancing service

●     nginx-ingress: Ingress service for L4 and L7 network traffic

●     cert-manager: Service that creates valid SSL certificates for TLS application services

●     Local DNS with self-signed certificate

Kubernetes Monitoring

NKP comes with a fully integrated Observability stack out-of-the-box. The logging stack is disabled by default and needs to be enabled explicitly.

The Monitoring stack uses:

●     Prometheus for metric collection and alert generation. It uses Prometheus Operator and allows modifying the configuration using CRDs such as PrometheusRules, ServiceMonitors, and so on.

●     Prometheus Adapter for serving Kubernetes metric API

●     AlertManager for alerting

●     Grafana for visualization of metrics with a rich set of pre-created custom dashboards

Additionally, for centralized monitoring, the NKP Management cluster runs Thanos for metric aggregation from all attached clusters. A centralized Grafana instance hosts dashboards to convert these metrics to meaningful information. Similarly, Karma provides a multi-cluster dashboard for Alerts.

The Logging stack uses:

●     Fluentbit for host, host_kernel, audit log collection

●     Logging-Operator for container log collection and forwarding

●     Loki for log indexing and storage (these are stored by default in a locally deployed rook-ceph Object Store)

●     Grafana Logging for logs and audit dashboard visualization.

All clusters run their local logging and monitoring stack. A Thanos instance running on the management cluster aggregates metrics from all Prometheus instances across NKP clusters. The management cluster runs an instance of Karma, which pulls alerts from workload clusters and shows them in its dashboard.

Kubernetes and application-level monitoring is based on the core infrastructure monitoring concept. The observability stack uses OpenTelemetry to collect and move the following data between the Kubernetes clusters:

●     Kubelet metrics

●     Host metrics

●     Kubernetes cluster metrics

●     Kubernetes events

●     Pod logs

●     Prometheus metrics from service monitors

●     Application-specific data from instrumentalization

Kubernetes Backup

The default storage class installed by NKP during deployment provides persistent storage for the management and workload clusters. The persistent volumes and application data in the clusters can be protected by a Velero backup schedule and stored in a S3 bucket provided by Nutanix Objects. A bucket in the local Nutanix Object store can be configured as the Backup Storage Location and replicated to an external location.

Large Language Model Design

This reference design presents a robust architecture for running LLM applications around the Kubernetes-based Nutanix AI inference endpoint, using NKP as the orchestration platform. It provides a comprehensive, scalable, and efficient framework tailored for building RAG pipeline applications. This framework capitalizes on the latest LLM technologies and supporting tools, ensuring ease of development, deployment, and monitoring.

Large Language Model Logical Design

At the core of this architecture are the LLM inference endpoints, which are provided through the Nutanix Enterprise AI platform. This essential component plays a critical role in deploying and managing machine learning models, particularly for addressing real-time inference requirements. The integration with the Nutanix GPT-in-a-Box 2.0 Design guarantees scalability, reliable accessibility, and consistent performance.

The modular architecture enables independent scaling and updating of each component, providing flexibility for system optimization. Its compatibility with the RAG framework allows the LLM to query and retrieve data from the Milvus vector database, significantly improving the accuracy and relevance of generated outputs.

The data ingestion process is designed for versatility, supporting both batch processing and event-driven workflows using Kafka. This dual approach allows the system to handle large-scale periodic batch uploads as well as continuous real-time data streams seamlessly. Event processing is implemented using serverless functions built on Knative, which are triggered by Nutanix Objects event notifications relayed through Kafka. This architecture ensures efficient, scalable, and highly responsive handling of incoming data streams. Intel AMX features are discovered automatically by embedding function to allow optimized processing of the data.

Once the domain-specific documents are ingested, the data is then vectorized using LangChain, where the embedding model encodes its data into vector representations. These vectors are then stored into Milvus, a scalable and high-performance vector database. Nutanix Objects provides the robust back-end storage necessary to meet the large-scale data demands of Milvus.

During the inference process, its integration with the RAG pipeline architecture allows the LLMs running on Nutanix Enterprise AI to dynamically query and retrieve relevant information from the Milvus vector database, significantly improving the accuracy and relevance of generated outputs. This approach enhances the system's ability to provide accurate, domain-specific answers by reducing reliance on the LLM's internal knowledge, which might be insufficient. Instead, the RAG workflow ensures that responses are informed by up-to-date and highly relevant data, improving both accuracy and contextual relevance for user queries.

In this architecture, NAI leverages NKP’s observability stack to enable seamless monitoring and logging for LLM workloads and GPU performance. Upon deploying the NVIDIA GPU Operator from the NKP catalog, NVIDIA Data Center GPU Manager (DCGM) metrics are automatically configured for Prometheus, along with default Grafana dashboards that detail GPU utilization, memory bandwidth, and thermal performance. Metrics are aggregated with Thanos and visualized in Grafana, while Fluentbit and Loki handle log collection and indexing, resulting in an advanced monitoring solution that goes beyond conventional GPU observability.

Large Language Model Research Workflow

The LLM uses the following research workflow:

1.     Ask a question: The interaction begins when the end user poses a question through the UI or chatbot interface.

2.     Create a query embedding: The embedding model transforms the user's query into a vector representation. This process is known as vectorization.

3.     Search and retrieve similar context: The vector database, which is specifically designed for similarity searches, stores the document embeddings generated by the model. It can efficiently search for and retrieve items based on these embeddings, which encapsulate the semantic meaning of the texts.

4.     Send the prompt: The workflow augments the user's query with relevant contextual information retrieved from the database, then sends this enriched query as a prompt to the LLM endpoint. The LLM processes the enriched query and generates a response.

5.     Get an answer: The UI or chatbot interface presents the LLM's response as the answer to the user's query.

Backup and Disaster Recovery

The scope of this solution is a single standalone GPT-in-a-Box 2.0 cluster, and you must back up the application data on the S3 buckets in the Nutanix Objects store to an external environment.

The persistent volumes and application data in the clusters can be protected by a Velero backup schedule and stored in a S3 bucket provided by Nutanix Objects.

You can use the streaming replication mechanism built into Nutanix Objects to replicate the data at the bucket level to a different S3 object store outside the GPT-in-a-Box 2.0 cluster. You can also use the existing backup solution to back up the persistent application data and store it outside the GPT-in-a-Box 2.0 cluster.

Solution Deployment

This chapter contains the following:

●     Infrastructure Deployment

●     Deploy GPT-in-a-Box 2.0

Infrastructure Deployment

This section details the prerequisites for installing the GPT-in-a-Box 2.0 solution and divided into the following key sections and procedures:

1.     Install AHV-based CCHC with Nutanix Cluster.

2.     Install the NVIDIA Grid Driver.

3.     Enable and Configure Nutanix Files.

Procedure 1.     Install AHV-based CCHC with Nutanix Cluster

This solution requires a minimum of four (4) Cisco Compute Hyperconverged X210c M7 All NVMe Nodes. Each of the cluster node is enabled with Intel® Xeon® Scalable Processors offering compute density and storage capacity in a single form factor and the Cisco UCS X440p PCIe Nodes supporting 2x NVIDIA L40S GPUs. For detailed specifications on the specification of server nodes, go to the Solution Design chapter.

Step 1.          Ensure the Fabric Interconnect Firmware is 4.3(.240066) or above.

Step 2.          From Intersight Dashboard, go to Operate > Chassis  and ensure Domain Profile is deployed successfully.

Step 3.          Go to Operate > Servers and ensure the X210C compute nodes firmware is 5.2 2.40074 or above.

Step 4.          Go to Operate > Chassis > <Chassis Name> and ensure the X210C compute nodes and X440P PCIe nodes are displayed correctly. The compute nodes are placed in Slot 1, 3, 5, 7 and PCIe nodes on Slot 2, 4, 6, 8.

Step 5.          Go to Chassis > Inventory tab and verify the IFM (Intelligent Fabric Module), X-Fabric Module,  and the X210C compute nodes are displayed on the dashboard.

Step 6.          Go to Servers and select one of the servers and view the inventory of the server. Ensure the nodes are attached to UCS X440P PCIe node which is equipped with 2x L40S or H100 GPUs.

Step 7.          Install the Nutanix cluster with AOS 6.7.1. and go to the Field Guide for information to successfully install the cluster.

The screenshot below shows four (4) nodes:

Step 8.          Provision a Prism Central Instance deployed on the Nutanix cluster and register the existing cluster to Prism Central. You have a choice to either deploy Prism Central on the GPT-in-a-Box cluster or utilize an existing Prism Central Instance.

Procedure 2.     Install NVIDIA Grid Driver

The following steps detail the process to deploy the NVIDIA driver on cluster nodes configured with L40S GPUs.

Step 1.          Download the GPU driver as per the AOS build installed on the cluster. This can be retrieved from the Compatibility and Interoperability matrix on the Nutanix portal. The existing cluster with AOS 6.10 is compatible with NVIDIA GRID for AHV 20230302.102001.

Step 2.          Ensure the NVIDIA GPU is available as a pci device. Log into AHV (node Host IP)  with root / nutanix/4u or the password set by the administrator.

Step 3.          Execute lspci | grep -I nvidia and ensure the GPU is installed on the host.

Step 4.          Log into the Nutanix node CVM (Nutanix/Nutanix/4u) or with the password set by the administrator and copy the tar file to /home/Nutanix.

Step 5.          Install the driver install_host_package -r lcm_nvidia_aie_20230302.102001_550.127.06.tar. This driver is installed across all nodes through the rolling restart process.

Step 6.          Log into AHV (node Host IP) with root / nutanix/4u or password set by the administrator.

Step 7.          Run nvidia-smi on the host and inspect the output for a table containing the driver version and detected GPU resources.

Procedure 3.     Enable and Configure Nutanix Files

This procedure provides high-level steps to enable Nutanix Files from Prism Central and configure from Prism Element Nutanix Files runs on the same cluster as GPT-in-a-Box 2.0 cluster. For Nutanix Files architecture details, go to Files on the Nutanix Portal.

Step 1.          Prior to creating the File Server, create the IP Address Management (IPAM) network.

Step 2.          Log into Prism Central and ensure the Name Server and NTP Server are updated.

Step 3.          From Prism Central, go to Infrastructure, select Network & Security > Subnet.

Step 4.          Enter the Name, Type=VLAN, select the Cluster (the GPT-in-a-Box 2.0 cluster), select the VLAN ID = <VLAN-ID> and Virtual Switch, enable Nutanix IP Address Management and give a range of available IPs, specify the Domain Settings, (Domain Server, Domain Search and Domain Name ). Click Create.

Step 5.          Go to Prism Central, click Admin Central, choose LCM, and click Perform Inventory.

Step 6.          Choose Marketplace and search for Files.

Step 7.          From Files click Get. This will enable File Services.

Step 8.          Log into Prism Element and click File Server. Click + File Server. Continue with selecting the latest File Server available. File Server Version 5.0.0.3 was the latest version available during the solution validation.

Step 9.          Enter the File Server details (Name, DNS Domain, and File Server Storage - Size 5TB). Choose the IPAM created in the previous step. Ensure the DNS and NTP Server are specified as per the environment. Click Next.

Step 10.       Click Client Network. The present configuration has same the Client and Storage network through a single IPAM.

 

Step 11.       Click Storage Network and the same IPAM.

Step 12.       Click Directory Services and then click Use NFS Protocol.

Step 13.       Verify the Summary and click Create.

Step 14.       Ensure the File Server name entries are updated in DNS.

In Prism Central, the File Server managed through Prism Element is shown below:

Deploy GPT-in-a-Box 2.0

This section details the steps to configure and deploy the GPT-in-a-Box 2.0 solution.

1.     Create and configure Jump Host VM.

2.     Install NKP and create the Base Image.

3.     Create a NKP management cluster.

4.     Install and Configure the NKP workload cluster and NAI.

Procedure 1.     Create and configure Jump Host VM

This procedure provides the high-level steps to configure Ubuntu jump host VM on Nutanix Cluster. This VM resides in the same Nutanix Cluster used for solution deployment. This VM would be used to deploy Nutanix Kubernetes  Cluster and the entire setup for GPT-in-a-Box 2.0 solution. The detailed steps to install the Jump Host VM are here: https://nai.howntnx.win/infra/infra_jumphost_tofu/#create-jump-host-vm

Step 1.          Install OpenTofu cli and visual studio code (VSCode) from your windows or Mac workstation. Reference https://nai.howntnx.win/infra/workstation/#install-opentofu.

Step 2.          In VSCode, create the workspace tofu-workspace.

Step 3.          In VSCode, create a jumphost-vm folder and add cloud-init.yaml with the details as shown below.

Step 4.          Edit the hostname and generate the RSA public key:

Step 5.          In VSCode, within the jumphost-vm folder, create a new file jumphostvm_config.yaml and provide your Nutanix Cluster details:

Step 6.          In the jumphost-vm folder, create a opentofu manifest file and name it jumphostvm.tf:

Step 7.          Open a new terminal in VSCode, initial and validate the tofu code:

Step 8.          Apply the tofu code to the create jump VM:

Step 9.          Get the jump VM IP details, as output:

Step 10.       Execute the Terraform state list command to verify what resources have been created:

Step 11.       Verify the jumphost-vm is accessible through VSCode > Terminal:

Step 12.       Connect to the jump host through VSCode and install utilities.

Step 13.       Install devbox with defaults:

Step 14.       Setup docker on jumphost VM:

Procedure 2.     Install NKP and create the Base Image

This procedure details the process to install Nutanix Kubernetes (NKP) binaries and create Base Image for NKP. The base image for NKP is a minimal image that contains the required packages and tools to run the Kubernetes cluster. The base image is used to create the worker node VMs and the control plane VMs.

In this procedure, you will go through creating a base image for all the control plane and worker node VMs on Nutanix. Use the Ubuntu 22.04 image as the base image since you will need GPU support for AI applications.

 

Step 1.          Access jumphost VM previously created.

Step 2.          Download NKP binaries from Nutanix portal and install NKP. You can change the NKP binary as per the version available on the Nutanix Portal.

Step 3.          Install NKP:

Step 4.          Create a NKP folder in $Home directory and go to the NKP folder.

Step 5.          Run the following command to generate an new RSA key pair on the jumphost VM. This SSH key pair will be used for authentication between the jumphost and NKP K8S cluster nodes.

Step 6.          Accept the default file location as ~/.ssh/id_rsa. SSH key pair is stored in the following location:

Step 7.          Create a .env file with the following details. Edit the environment variables as per your environment:

Step 8.          Load the environment variables and its values:

Step 9.          Create the base image and upload it to Prism Central using the following command:

Step 10.       Populate the .env file with the base image:

Procedure 3.     Create a NKP management cluster

In this procedure you will go through the CLI process of creating a NKP management cluster. When the management cluster is created, you can create the GPT-in-a-Box 2.0 workload cluster from NKP commander dashboard.

Step 1.          Access the jumphost VM previously created.

Step 2.          Append the environment variable to $HOME/nkp/.env to create NKP bootstrap cluster. The bootstrap cluster is named cvd-nkp-mgmt:

Step 3.          Execute the .env file to load environment variables:

Step 4.          Create a NKP bootstrap cluster:

Step 5.          You can monitor the cluster creation by exporting KUBECONFIG=~/cvd-nkp-mgmt-bootsrap.conf and executing the following commands:

Step 6.          From Prism Central you can monitor VMs created on the Nutanix Cluster:

Step 7.          During the creation of the cvd-nkp-mgmt cluster, verify the creation of nutanixmachines and describe the cvd-nkp-mgmt cluster:

Step 8.          When the core k8s cluster components are created, verify the creation of CAPI components. Ensure you export KUBECONFIG=~/cvd-nkp-mgmt.conf:

Step 9.          During the creation of the cvd-nkp-mgmt cluster, verify the pods in creation state:

Step 10.       During the creation of the NKP cluster, verify the created helm releases:

Step 11.       Verify the successful creation of the NKP management cluster:

Step 12.       Get the NKP Dashboard access details:

Step 13.       Access the NKP Global Dashboard:

Step 14.       Activate the NKP Ultimate license to continue creating the workload cluster:

Procedure 4.     Install and Configure the NKP workload cluster and NAI

This procedure provides the steps to create Nutanix Enterprise AI (NAI) and Nutanix workload cluster which enables GPU resources.

Workload clusters have a dedicated node pool specifically for GPU resources. This node pool hosts pods that require GPU capabilities, such as nodes with machine learning or data processing workload.

Step 1.          Connect to the jump host vm and execute source ~/nkp/.env.

Step 2.          Connect to the NKP management cluster KUBECONFIG=~/cvd-nkp-mgmt.conf.

Step 3.          Create the NAI workspace:

Step 4.          Add the Nutanix catalog to the nai workspace:

Step 5.          Create knative-override.yaml:

Step 6.          Create gpu-override.yaml:

Step 7.          Create nai-override.yaml. Ensure you have the login and password to access the NAI repository:

Step 8.          Enable catalog apps in the workspace:

Step 9.          Go to the NKP dashboard and create a workload cluster with the following configurations:

The workload cluster configuration (cvd-nai-wl-01) is displayed below:

Step 10.       Add the GPU node pool to the workload cluster (cvd-nai-wl-01):

Below are the workload cluster details with GPU pool from the CLI.

Step 11.       Execute the .env file to load the environment variables and connect to the workload cluster:

Step 12.       In the exiting deployment (recommended for PoC), use the ingress IP associated with the istio ingress load balancer service to auto-generate the certificate. Execute the following snippet to auto-generate certificate:

Step 13.       Ensure the nai-cert is created:

Step 14.       To use a certificate with a custom hostname and the corresponding certificate, use the following snippet. This is recommended for production environments.

Step 15.       Apply the following patch to knative-serving:

Step 16.       Log into Prism Element and go to the previously created File Server (fileserver002) and create a REST API access user for NAI. This is created under FileServer > Configuration > Manage Roles.

Step 17.       Add a user with username= csi and password <password>.

Step 18.       Apply the following configuration to the create storage class and NFS volume:

Step 19.       Identify the istio-ingressgateway IP for NAI to access the NAI dashboard:

Step 20.       Open a web browser and go to istio-ingressgateway IP (10.1082.93). This open the Nutanix Enterprise AI (NAI) portal. Log in with admin/Nutanix.123. Update the Password in the next window.

The following screenshot displays the NAI dashboard:

Solution Validation

This chapter contains the following:

●     Summary of Validated Models

●     Validate Nutanix Enterprise AI

●     Visibility and Monitoring

●     Sizing Considerations

Summary of Validated Models

Nutanix Enterprise AI allows downloaded pre-validated models from Hugging Face, NVIDIA NGC Catalog. You can also download custom models.

Validate Nutanix Enterprise AI

Nutanix Enterprise AI (NAI) allows you to download models and create secure inference end points which can be leveraged by different teams either to implement chatbots or Retrieval Augmented Generation (RAG) pipelines.

1.     Import Models with NAI.

2.     Deploy Secure Endpoint.

Procedure 1.     Import Models with NAI

This procedure details the NAI workflow to import validated models either through Hugging Face or NGC. NAI also supports downloading custom models. Please refer to the Nutanix YouTube link to download custom models.

Step 1.          Log into the NAI dashboard.

Step 2.          Go to Models in the navigation pane. Click Import Models.

Step 3.          Before you import models, you need to add the Hugging Face Model Hub token or the NVIDIA NGC key, using the Settings option in NAI.

Step 4.          From the import Model, select From Hugging Face Hub. You’ll see a list of validated models which can be imported to NAI.

Step 5.          Select a model and click Import.

Step 6.          Before importing a model, ensure on Hugging Face you have granted permission for gated model.

Step 7.          Enter a model instance name and click Import.

Step 8.          Monitor the import model process on workload k8s cluster in nai-admin namespace. You can monitor the k8s pod created to import the model and a new persistent volume created which stores the model locally on NFS volume:

When the model is imported successfully, the status of the model becomes Active:

The pod on the workload cluster is deleted with the size of pvc equal to the size of the model:

Procedure 2.     Deploy Secure Endpoint

Step 1.          From the NAI dashboard, click the Endpoints tab and click Create Endpoint. 

Step 2.          Define the Endpoint name, select the model already imported in NAI, choose the GPUs and number of instances. Choose the Endpoint access key. You can create multiple key assignment which can be leveraged to monitor the number of request per api key to the inference Endpoint. Click Create.

Step 3.          View the container creation on the workload cluster in nai-admin name space:

Step 4.          Monitor the inference service creation in the nai-admin namespace:

The status for the end point creation should be marked as Active in the NAI dashboard:

Step 5.          View the sample request and test the created Endpoint:

Visibility and Monitoring

It is critical to gain complete visibility into the entire stack, including Physical infrastructure (Compute, Storage and Network), Virtualized infrastructure, NKP clusters and the Nutanix Enterprise AI. It helps to gain insight into infrastructure bottlenecks and factors that increase costs and ensures the performance for model inferencing and applications.

This section provides some ways of gaining visibility of the application stack.

Infrastructure Insights

The physical and virtual Infrastructure components can be monitored through multiple dashboards

The physical infrastructure deployed in Cisco Compute Hyperconverged for Nutanix can be monitored through Cisco Intersight.

Intersight offers a wide array of options to monitor your data center environment. The devices connected to Intersight report inventory information and alerts back to Intersight, which you can then view. For more information, see Server Inventory View, Chassis Inventory View, and Fabric Interconnects Inventory View.

The information about the connected environment is also enriched with more information by Cisco:

●     Hardware Compatibility List (HCL)—Checks if your server and firmware combination is validated by Cisco

●     Advisories—Lists known vulnerabilities and other information for your specific environment.

●     Service Contract—Checks if your device has an active service contract with Cisco.

Additionally, Intersight collects metrics from Fabric Interconnects, Chassis, and Servers that are managed as Intersight Managed Mode (IMM) or UCMS Managed Mode (UMM) domains in Cisco Intersight. These IMM or UMM domains capture metrics for the various devices and components in a domain locally. These metrics are then aggregated and sent to Intersight in regular intervals. An example of this would be calculating the minimum, average, and maximum value. These metrics are stored in Intersight, and you can query the metrics.

For more information about Cisco Intersight monitoring metrics, see the Cisco Intersight Monitoring Overview page.

The physical infrastructure deployed in Cisco Compute Hyperconverged for Nutanix can be monitored through Cisco Intersight.

The virtual infrastructure such as VMs and File Services can be monitored through Prism Central.

Procedure 1.     Cisco Intersight monitoring

Step 1.          Log into Cisco Intersight. The main dashboard provides an insight into the power and energy metrics of the physical servers deployed for Nutanix Cluster.

Cisco Intersight enables integration with Nutanix providing monitoring of Cisco UCS servers deployed as Nutanix Clusters.

Step 2.          View the Security Advisories, Field Notices, and End of Life Advisories through Cisco Intersight platform.

Procedure 2.     Prism Central monitoring

The Prism Central dashboard provides monitoring of key metrics for Nutanix Cluster. Details of the resource utilization of Nutanix Cluster connected to Prism Central is shown below:

Step 1.          Go to the File Server tab and view the resource utilization of File Server deployed for GPT-in-a-Box 2.0 solution.

NKP Cluster Dashboard

The Nutanix Kubernetes kommander dashboard provides resource monitoring of both management and workload clusters deployed for the solution.

Procedure 1.     NKP Kommander Dashboard

Step 1.          Identify the NKP dashboard login details:

Step 2.          Log into the NKP dashboard and view the management and workload cluster for the present solution:

Step 3.          Access the management cluster dashboard and view the services deployed and monitor the resource utilization.

Step 4.          Access the workload cluster dashboard to view the monitoring of resources deployed in the workload cluster.

NAI Dashboard

The Nutanix Enterprise AI dashboard provides monitoring of GPU resource and details the number of request and request latency of secure end points. You can create different access keys to monitor end points per user group.

The request latency and request summary of the secure inference Endpoints deployed with NAI is shown below:

Sizing Considerations

This section provides an overview and  general guidance for sizing GPU memory required to deploy LLM models. Nutanix provides minimum sizing requirements to install Nutanix Enterprise AI and deploy pre-validated models. For information about the minimum requirements to successfully deploy NAI, go to: Nutanix Enterprise AI Requirements

Table 17 lists the pre-validated LLMs and the minimum number of supported GPUs required to deploy these LLMs.

Table 17.     Minimum GPU requirements

Key Terms

Some of the key terms with respect to LLM inferencing are:

●     Batch Size: Batch size refers to the number of samples that are processed together in a single inference run. The batch size can significantly impact the latency and throughput of the inference process. For example, increasing the batch size can increase throughput but at the cost of increased latency.

●     Precision: The size in bytes used for each parameter in the model.

●     Context Size: Represents the maximum number of tokens the model process for the prompt + response. (Input Tokens Length + Output Tokens Length)

●     Key and Value Cache (KV Cache): The amount of memory consumed by a single token based on the model dimensions and layers.

●     Activations: When a token is being processed within the model, it is called an Activation. Full activation memory is calculated from the KV Cache size and context size.

LLM Inferencing Phases

LLMs generate text in a two-step process:

●     Prefill: In the first phase, the model ingests your prompt tokens in parallel, populating the key-value (KV) cache. Prefill generates the key and value cache (KV Cache) for future decoding.

●     Decoding Phase: In the second phase, we leverage our current state (stored in the KV cache) to sample and decode the next token. We pay a small price in storage to not recalculate the cache for every single new token. Without the KV cache, every successive token would take longer to sample because we would have to pass all previously seen tokens through the model.

The input sequence from the user is first tokenized in the same way the model's training data was tokenized. Then the tokens are fed to the trained model.

As the first step of model execution, the input sequence is converted into an embedding vector in the embedding layer of the trained model. This vector essentially translates the token into a high-dimensional space where similar tokens have similar vectors.

Using the token embeddings, queries, keys, and values for each token are computed through a process known as linear projection. Here, each token's embedding vector is multiplied by separate weight matrices learned during the training to produce the query, key, and value vectors.

The concept of using queries, keys, and values is directly inspired by how databases work. Each database storage has its data values indexed by keys, and users can retrieve the data by making a query and comparing if the key value matches the query. In the LLM case, the model generates the queries itself. The key values are not directly compared to the query, but the relevance of each key to the query is computed using a compatibility function to generate a weight vector.

An attention output is computed for each query as the weighted sum of the "values" using the weight vector previously computed.

This attention output goes through a prediction (or decode) layer, which assigns probabilities to each token in the entire vocabulary of the model, indicating the likelihood of that token being the next one.

One can sample from the predicted probability distribution to choose the next token probabilistically or select the token with the highest probability as the predicted next token.

Cluster Sizing

This CVD describes the design for a single GPT-in-a-Box 2.0 cluster with four or more nodes. If one of the scaling factors (such as the total number of VMs, Kubernetes applications, or GPU workloads) exceeds the maximum specified for the solution, extend the cluster with the required capacity (CPU, memory, storage, GPU). After reaching the maximum cluster size, consider building a new cluster to support the demand for further growth.

Infrastructure Considerations

The following are the key factors for sizing infrastructure for Generative AI inferencing:

●     Model Specifications

◦     Model Architecture: Understand the architecture of the language model, including the number of layers, attention heads, and parameters.

◦     Token Embedding Size: Larger embedding sizes can significantly impact memory requirements.

●     Hardware Acceleration

◦     Mixed Precision: Explore mixed-precision training and inference to leverage hardware capabilities efficiently.

●     Memory Requirements

◦     Model Size: Large language models can have substantial memory requirements. Ensure sufficient GPU memory for both the model and input sequences.

◦     Sequence Length: Consider the maximum sequence length the model can handle and its impact on memory usage.

●     Batching and Parallelization

◦     Batch Size: Experiment with batch sizes. Larger batch sizes can improve throughput but may increase memory requirements.

◦     Data Parallelism: Implement data parallelism to distribute inference across multiple devices, if necessary.

●     Latency and Throughput

◦     Latency Requirements: Language models often have real-time constraints, especially in interactive applications. Minimize latency based on use case.

◦     Throughput Targets: Determine the required throughput in terms of processed tokens or sequences per second.

●     Scalability

◦     Model Parallelism: Consider model parallelism if the model size exceeds available GPU memory, distributing parts of the model across multiple GPUs.

◦     Infrastructure Scaling: Design for horizontal scalability to handle increased demand.

●     Redundancy and High Availability

◦     Checkpointing: Implement regular model checkpointing to recover from failures without losing training progress.

◦     Replication: Use redundant systems to ensure high availability during inference.

●     Network Considerations

◦     Bandwidth: Assess the network bandwidth required for transferring large model parameters between devices.

◦     Inter-Device Communication: Optimize communication patterns between devices to minimize latency.

●     Storage Requirements

◦     Model Storage: Choose storage solutions capable of efficiently loading large model parameters.

◦     Data Storage: Assess storage needs for input data and any intermediate results.

●     Containerization and Orchestration

◦     Containerization: Deploy the language model within containers for easier management and consistency across different environments.

◦     Orchestration: Use container orchestration tools like Kubernetes for managing and scaling the deployment efficiently.

●     Middleware and Serving Frameworks

◦     Serving Framework: Choose a serving framework optimized for deploying large language models, such as TensorFlow Serving, Triton Inference Server, or others.

◦     Middleware: Implement middleware for handling communication between clients and the deployed model, ensuring compatibility with your application's requirements.

●     Monitoring and Optimization

◦     Resource Monitoring: Employ monitoring tools to track GPU utilization, memory usage, and other relevant metrics.

◦     Dynamic Optimization: Optimize parameters dynamically based on real-time performance metrics.

●     Security

◦     Data Protection: Implement measures to secure input and output data, especially if it involves sensitive information.

◦     Model Security: Protect large language models from adversarial attacks and unauthorized access.

When selecting the GPU for this solution, there are several crucial factors to consider:

●     Processing Power: Look for GPUs with sufficient computational power to handle the complex neural network computations required for LLM inference.

●     Memory Capacity: LLM models often have large memory requirements. Ensure that the GPU has enough VRAM (Video RAM) to accommodate the model size and batch sizes.

●     Tensor Cores: Tensor cores accelerate matrix operations, which are essential for LLM inference. GPUs with tensor cores (such as NVIDIA's RTX series) can significantly improve performance.

●     Compatibility: Check if the GPU is compatible with the Cisco UCS and the deep learning framework you plan to use  and if (e.g., TensorFlow, PyTorch).

●     Power Consumption: Consider the GPU's power draw and ensure it aligns with your system's power supply capacity.

Memory Calculations for LLM Inferencing

Total Memory required is the sum of model memory size and the KV cache.

Calculations for the required total memory is provided below:

Model Memory Size = Model Parameters * Precision

KV Cache Size = 2 x Batch Size x Context Size  x Number of Layers x Model Dimensions x Precision

Total Memory Requirements (GB) = Model Memory Size (GB) + KV Cache Size (GB)

For some models, model dimension data might not be available. In that case, model dimension can be calculated as:

Model Dimensions = Attention Head Size X Number of Attention Heads

Model Parameters, Precision, Number of layers, Model Dimension are specific to models, and it can be found in the Model card for the model.

Context Size and batch size are input from users.

The following is an example memory calculation for Llama 2:

For the Llama 2 model:

Total model parameters:         6.74B Parameters.

Precision:                                 FP16. (2 Bytes)

Number of layers:               32

Model Dimension:                4096

The model memory is calculated is shown below:

Model Memory Size = Model Parameters * Precision

 

Model Memory Size for Llama 2   = 6,740,000,000 * 2 Bytes/Parameter

                                = 13,480,000,000 Bytes

                                 = 13.48 Giga Bytes                                             

Considering an example of maximum Input Tokens Length of 1024, Maximum Output Tokens Length of 1024,  and the Batch size of 8, below are the calculations for KV Cache Size:

KV Cache Size         = 2 x Batch Size x Context Size  x Number of Layers x Model Dimensions x Precision

KV Cache Size         = 2 x 8 x (1024+1024) x 32 x 4096 x 2 Bytes/Parameter

                      = 8,589,934,592 Bytes

                      =  8.59 Giga Bytes

Therefore, Llama2 with maximum Input Tokens Length of 1024, Maximum Output Tokens Length of 1024, and the Batch size of 8, the total memory required is as shown below:

Total Memory Requirements (GB) = Model Memory Size (GB) + KV Cache Size (GB)

                                 = 13.48 + 8.59 Giga Bytes

                                 = 22.07 Giga Bytes

Performance Calculations

The performance benchmark can be run on the model.

Based on the performance requirement, number of users, number of input and output tokens, latency and throughput required, you can choose the appropriate Large Language Model, Inferencing backend, GPUs, and compute infrastructure.

The performance of the model depends on the prefill and decode phases. These two phases have different impacts on the performance of the LLM. While the prefill phase effectively saturates GPU compute at small batch sizes, the decode phase results in low compute utilization as it generates one token at a time per request.

The prefill phase is compute-bound, while the decode phase is memory-bound. So, the following factors need to be considered and measured:

●     Prefill Latency

●     Prefill Throughput

●     Decode Total Latency

●     Decode Token Latency

●     Decode Throughput

The performance benchmark can be run with different sizes (1,2,4,8,10,25,250, 100 and so on). Also, separate tests can be run focused on performance comparison between 2 different models.

Conclusion

This validated solution is a valuable resource for navigating the complexities of Generative AI application deployment in real-world enterprise environments leveraging Nutanix Enterprise AI to deploy your choice of LLMs (large language models) from leading LLM providers and create and manage secure APIs to connect your GenAI applications.

This Cisco Validated Design for Cisco Compute Hyperconverged with Nutanix GPT-in-a-Box 2.0 provides a foundational reference architecture for deployment of an innovative, flexible, and secure generative pretrained transformer (GPT) solution for Generative AI to privately run and manage organization’s choice of AI large language models (LLMs) and applications in which it leverages.

In combination of Cisco UCS and Nutanix, this solution intends to enable an AI inferencing turnkey solution which can be quickly deployed as an on-premises solution. The infrastructure is designed using the Cisco Intersight Managed X-Series modular system with  Cisco UCS X210c M7 All-NVMe servers configured with 2x NVIDIA L40S GPUs on each Cisco UCS X440p PCIe node, which leverages Software-defined Nutanix Cloud Infrastructure supporting GPU-enabled server nodes for seamless scaling of virtualized compute, storage, networking supporting Kubernetes-orchestrated containers and Nutanix Unified Storage.

Appendix

This appendix contains the following:

●     Appendix A – Bill of Materials

●     Appendix B – References used in this guide

Appendix A – Bill of Materials

Table 18 list of the Bill of Materials used in this solution design and the deployment and validation described in this document.

Table 18.     Bill of Materials

Appendix B – References used in this guide

GPT-in-a-Box 2.0 NVD

https://portal.nutanix.com/page/documents/details?targetId=Nutanix-Enterprise-AI-v2_0:Nutanix-Enterprise-AI-v2_0

NAI LMM Guide

https://nai.howntnx.win/iep/

Field Guide to deploy Cisco Compute Hyperconverged with Nutanix in IMM mode

https://community.cisco.com/t5/unified-computing-system-knowledge-base/cisco-compute-hyperconverged-with-nutanix-on-x-series-field/ta-p/5219852

Nutanix Kubernetes Platform Guide

https://portal.nutanix.com/page/documents/details?targetId=Nutanix-Kubernetes-Platform-v2_13:Nutanix-Kubernetes-Platform-v2_13

Cisco Compute Hyperconverged with Nutanix

https://www.cisco.com/c/en/us/products/hyperconverged-infrastructure/compute-hyperconverged/index.html

Cisco Intersight

https://www.cisco.com/c/en/us/products/servers-unified-computing/intersight/index.html

Cisco UCS X210c All-NVMe/All-Flash Server

https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-x-series-modular-system/x210cm7-specsheet.pdf

Nutanix Reference Documentation

https://portal.nutanix.com/

About the Authors

Anil Dhiman, Technical Marketing Engineer, Cisco Systems, Inc.

Anil Dhiman has over 20 years of experience specializing in data center solutions on Cisco UCS servers, and performance engineering of large-scale enterprise applications. Over the past 14 years, Anil has authored several Cisco Validated Designs for enterprise solutions on Cisco data center technologies. Currently, Anil's focus is on Cisco’s portfolio of hyperconverged infrastructure and data protection solutions.

Paniraja Koppa, Technical Marketing Engineer, Cisco Systems, Inc.

Paniraja Koppa is a member of the Cisco Unified Computing System (Cisco UCS) solutions team.  He has over 15 years of experience designing, implementing, and operating solutions in the data center. In his current role, he works on design and development, best practices, optimization, automation and technical content creation of compute and hybrid cloud solutions. He also worked as a technical consulting engineer in the data center virtualization space. Paniraja holds a master’s degree in computer science. He has presented several papers at international conferences and speaker at events like Cisco Live US and Europe, Open Infrastructure Summit, and other partner events. Paniraja’s current focus is on Generative AI solutions.

Wolfgang Huse, Sr. Staff Solution Architect – Cloud Native & AI, Nutanix, Inc.

Wolfgang Huse is the technical lead for Cloud Native in Solutions and Performance Engineering team at Nutanix and is based out of Germany. In this role, he and his team are primarily responsible for the development of cloud native solutions and evangelizing the benefits of adopting cloud native architectures within Nutanix Cloud Platform. Wolfgang and his team have been instrumental in building many key solution artifacts such as the various Nutanix tech notes, best practice guides and validated designs to support solutions such as RedHat OpenShift, Rancher and most recently Nutanix GPT in-a-Box. Prior to this role, he worked in sales engineering roles at Nutanix, directly providing assistance on designing and implementing complex solutions, globally.

Arvind Bhoj, Senior Solutions Architect - Cloud Native & AI, Nutanix, Inc.

Arvind is a Sr. Solutions Architect for Cloud Native Technology at Nutanix. Arvind has over 20 years of experience in the tech industry and is extremely passionate about Kubernetes and Cloud & IT infrastructure automation/integration. He has extensive experience in product engineering as well as working directly with customers in the field with large organizations across the globe to help them in their Cloud Native and Container Orchestration journey.

Jesse Gonzalez, Staff Solution Architect – Cloud Native & AI, Nutanix, Inc.

Jesse Gonzalez is a Cloud Native Solutions Architect on the Solutions and Performance Engineering team at Nutanix.  With over 20+ years of experience in IT, Jesse has had the privilege of working closely with many organizations of all sizes to overcome their challenges in enabling cloud-native (and more recently Generative AI) solutions on the Nutanix platform.  Prior to this role, Jesse has worked in roles within both services and sales engineering at Nutanix.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

●     Chris O'Brien, Senior Director, Cisco Systems, Inc.

Feedback

For comments and suggestions about this guide and related guides, join the discussion on Cisco Community here: https://cs.co/en-cvds.

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series, Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_P5)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)