FlashStack for Enterprise RAG Pipeline with NVIDIA NIM, NIM Operator, and RAG Blueprint

 

Published: May 2025

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: http://www.cisco.com/go/designzone.

Executive Summary

The Generative AI revolution necessitates secure integration with unique enterprise data to realize its full potential. Retrieval Augmented Generation (RAG) is the pivotal technology for this, yet deploying robust, enterprise-grade RAG pipelines has traditionally presented significant complexity, prolonged timelines, and considerable integration challenges.

This document introduces a Cisco Validated Design (CVD), specifically engineered to simplify, and accelerate the deployment of advanced RAG solutions. The design synergistically integrates the best-in-class FlashStack converged infrastructure—a high-performance foundation of Cisco UCS X-Series compute, Nexus networking, and Pure Storage FlashArray and FlashBlade—with NVIDIA's cutting-edge AI platform. This platform features the NVIDIA RAG Blueprint, providing a foundational structure with extensive customization freedom for tailoring pipelines to specific enterprise needs, and utilizes NVIDIA Inference Microservices (NIM) ensuring maximally optimized execution of AI inference tasks on the underlying NVIDIA GPUs. Orchestrated on Red Hat OpenShift Container Platform and managed via Cisco Intersight, this integrated system constitutes a dedicated AI enablement platform.

Lengthy integration cycles and associated deployment uncertainties are substantially mitigated by this approach. The validated blueprint provides a fast track for deploying powerful, private RAG pipelines, constructed upon meticulously tested, enterprise-grade components. This solution empowers enterprises to unlock their data's potential, establish trustworthy AI applications, and achieve a competitive advantage with unprecedented speed and operational confidence. This guide provides the detailed RAG design, while the FlashStack with Red Hat OpenShift Container and Virtualization Platform using Cisco UCS X-Series CVD offers the core infrastructure specifications.

If you are interested in understanding the FlashStack design and deployment details, including configuration of various elements of design and associated best practices, refer to the Cisco Validated Designs for FlashStack here: https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/data-center-design-guides-all.html#FlashStack.

Solution Overview

This chapter contains the following:

●     Introduction

●     Audience

●     New in this release

●     Solution Summary

Introduction

As enterprises seek to harness Generative AI with their proprietary data, Retrieval Augmented Generation (RAG) emerges as a critical technique. RAG enhances Large Language Models (LLMs) by grounding them in specific enterprise information, enabling accurate, context-aware responses for use cases like internal knowledge base querying. However, deploying robust, scalable, and secure RAG pipelines presents significant infrastructure and integration challenges.

This document introduces a Cisco validated solution addressing these challenges: Enterprise RAG Pipeline on FlashStack, leveraging NVIDIA RAG Blueprint and NVIDIA NIMs. This collaborative solution, jointly engineered by Cisco, Pure Storage and NVIDIA, provides a pre-integrated and optimized stack for deploying demanding GenAI workloads.

The foundation is FlashStack converged infrastructure. Layered upon this infrastructure is the NVIDIA AI software stack. The solution utilizes the NVIDIA RAG Blueprint, a reference architecture that accelerates the development of customizable RAG pipelines. This blueprint leverages NVIDIA Inference Microservices (NIM) – optimized, self-hosted endpoints for LLM inferencing, text embedding, and reranking – ensuring data privacy and low latency. NIMs are deployed and managed using the NIM Operator on Red Hat OpenShift Container Platform, orchestrated across NVIDIA GPU-accelerated Cisco UCS servers.

By integrating these best-in-class components into a Cisco Validated Design (CVD), this solution offers enterprises a streamlined path to deploying powerful, private RAG capabilities, reducing operational complexity, and accelerating time-to-value for GenAI initiatives.

Audience

The intended audience of this document includes IT decision makers such as CTOs and CIOs, IT architects, sales engineers, field consultants, professional services, IT managers, partner engineers, and customers. It is designed for those interested in the design, deployment, and life cycle management of Retrieval Augmented Generation pipelines with Cisco AI ready datacenter.

Purpose of this document

This document provides design and deployment guidance set up Retrieval Augmented Generation pipeline that uses NVIDIA NIM and GPU-accelerated components on FlashStack Datacenter.

New in this release

The following design elements are built on the CVD FlashStack with Red Hat OCP Bare Metal Manual Configuration with Cisco UCS X-Series Direct Deployment Guide to implement an end-to-end Retrieval Augmented Generation pipeline using NVIDIA RAG Blueprint and  NVIDIA NIMs.

●     Installation and configuration of RAG pipeline components on FlashStack:

◦     NVIDIA GPUs

◦     NVIDIA AI Enterprise (NVAIE)

◦     NVIDIA NIM Operator

◦     NVIDIA NIM microservices - NVIDIA NIMs for LLM Inferencing, text embedding and text reranking.

◦     NVIDIA AI Blueprint version 1.0.0 for RAG

◦     RAG Evaluation

◦     NIM for LLM Benchmarking

◦     Vector Database Benchmarking

Solution Summary

This solution provides a validated, foundational architecture for deploying enterprise-grade Retrieval Augmented Generation pipelines. It leverages a Cisco UCS X-Series based FlashStack Datacenter as the high-performance converged infrastructure foundation. Key NVIDIA AI technologies, including the RAG Blueprint reference implementation and NVIDIA Inference Microservices (NIM) running on NVIDIA GPUs, power the AI workload. NVIDIA NIM Operator manages the life cycle of the NVIDIA NIM for LLMs, Text Embedding NIM and Reranking NIMs.

 The entire stack is deployed on a bare-metal Red Hat OpenShift cluster, utilizing Portworx by Pure Storage for container-native persistent storage. This pre-tested integration of validated models, GPUs, and backend components, accessible via NIM's industry-standard APIs, delivers an optimized and AI-ready platform, significantly simplifying the deployment and management of secure, private RAG applications.

This architecture represents a rigorously tested and validated integration across all layers – compute, networking, storage, NVIDIA GPUs, AI models, and the software stack (OCP, Portworx, NVIDIA AI Enterprise). NVIDIA NIM facilitates streamlined application development and integration via industry-standard APIs (including OpenAI compatibility). The resultant platform is an optimized, secure, and AI-ready infrastructure solution designed to mitigate deployment complexities, reduce operational overhead, and accelerate the delivery of high-performance, private RAG capabilities within the enterprise.

Figure 1 illustrates the solution overview.

The FlashStack solution as a platform for Retrieval Augmented Generation offers the following key benefits:

●     The ability to implement a Retrieval Augmented Generation pipeline quickly and easily on a powerful platform with high-speed persistent storage.

●     Blueprint leverages NVIDIA NIM microservices deployed on-premise to meet specific data governance and latency requirements.

●     Evaluation of performance of platform components.

●     Simplified cloud-based management of solution components using policy-driven modular design.

●     Cooperative support model and Cisco Solution Support.

●     Easy to deploy, consume, and manage architecture, which saves time and resources required to research, procure, and integrate off-the-shelf components.

●     Support for component monitoring, solution automation and orchestration, and workload optimization.

Like all other FlashStack solution designs,  FlashStack for Enterprise RAG Pipeline with NVIDIA NIM, NIM Operator and RAG Blueprint is configurable according to demand and usage. You can purchase exactly the infrastructure you need for your current application requirements and then scale-up/scale-out to meet future needs.

Technology Overview

This chapter contains the following:

●     Retrieval Augmented Generation

●     NVIDIA AI Blueprint for RAG

●     NVIDIA AI Enterprise

●     NVIDIA Inference Microservices

●     NVIDIA NIM for Large Language Models

●     NeMo Text Retriever NIM

●     FlashStack Components

●     Benefits of Portworx Enterprise with OpenShift Container Platform

●     Benefits of Portworx Enterprise with Pure Storage FlashArray

●     Benefits of Pure Storage FlashBlade

Retrieval Augmented Generation

Retrieval Augmented Generation (RAG) is an enterprise application of Generative AI. RAG represents a category of large language model (LLM) applications that enhance the LLM's context by incorporating external data. It overcomes the limitation of knowledge cutoff date (events occurring after the model’s training). LLMs lack access to an organization's internal data or services. This absence of up-to-date and domain-specific or organization-specific information prevents their effective use in enterprise applications.

RAG Pipeline

Figure 2 illustrates the RAG Pipeline overview.

In this pipeline, when you enter a prompt/query, document chunks relevant to the prompt are searched and fetched to the system. The retrieved relevant information is augmented to the prompt as context. LLM is asked to generate a response to the prompt in the context and the user receives the response.

RAG Architecture

RAG is an end-to-end architecture that combines information retrieval component with a response generator.

RAG can be broken into two process flows; document ingestion and inferencing.

Figure 4 illustrates the document ingestion pipeline.

Figure 5 illustrates the inferencing pipeline.

The process for the inference serving pipeline is as follows:

1.     A prompt is passed to the LLM orchestrator.

2.     The orchestrator sends a search query to the retriever.

3.     The retriever fetches relevant information from the Vector Database.

4.     The retriever returns the retrieved information to the orchestrator.

5.     The orchestrator augments the original prompt with the context and sends it to the LLM.

6.     The LLM responds with generated text/ response and presents it to the user.

NVIDIA AI Blueprint for RAG

The NVIDIA AI Blueprint for RAG gives developers a foundational starting point for building scalable, customizable retrieval pipelines that deliver both high accuracy and throughput. Use this blueprint to build RAG applications that provide context-aware responses by connecting LLMs to extensive multimodal enterprise data—including text, tables, charts, and infographics from millions of PDFs. With 15x faster multimodal PDF data extraction and 50 percent fewer incorrect answers, enterprises can unlock actionable insights from data and drive productivity at scale.

This blueprint can be used as-is, combined with other NVIDIA Blueprints, such as the Digital Human blueprint or the AI Assistant for customer service blueprint, or integrated with an agent to support more advanced use cases. Get started with this reference architecture to ground AI-driven decisions in relevant enterprise data - wherever it resides.

NVIDIA AI Enterprise

The NVIDIA AI Enterprise (NVAIE) platform was deployed on Red Hat OpenShift as the foundation for the RAG pipeline. NVIDIA AI Enterprise simplifies the development and deployment of generative AI workloads, including Retrieval Augmented Generation, at scale.

NVIDIA Inference Microservices

NVIDIA Inference Microservice (NIM), a component of NVIDIA AI Enterprise, offers an efficient route for creating AI-driven enterprise applications and deploying AI models in production environments. NIM consists of microservices that accelerate and simplify the deployment of generative AI models via automation using prebuilt containers, Helm charts, optimized models, and industry-standard APIs.

NIM simplifies the process for IT and DevOps teams to self-host large language models (LLMs) within their own managed environments. It provides developers with industry-standard APIs, enabling them to create applications such as copilots, chatbots, and AI assistants that can revolutionize their business operations. Content Generation, Sentiment Analysis, and Language Translation services are just a few additional examples of applications that can be rapidly deployed to meet various use cases. NIM ensures the quickest path to inference with unmatched performance.

NIMs are distributed as container images tailored to specific models or model families. Each NIM is encapsulated in its own container and includes an optimized model. These containers come with a runtime compatible with any NVIDIA GPU that has adequate GPU memory, with certain model/GPU combinations being optimized for better performance. One or more GPUs can be passed through to containers via the NVIDIA Container Toolkit to provide the horsepower needed for any workload. NIM automatically retrieves the model from NGC (NVIDIA GPU Cloud), utilizing a local filesystem cache if available. Since all NIMs are constructed from a common base, once a NIM has been downloaded, acquiring additional NIMs becomes significantly faster. The NIM catalog currently offers nearly 150 models and agent blueprints.

Utilizing domain specific models, NIM caters to the demand for specialized solutions and enhanced performance through a range of pivotal features. It incorporates NVIDIA CUDA (Compute Unified Device Architecture) libraries and customized code designed for distinct fields like language, speech, video processing, healthcare, retail, and others. This method ensures that applications are precise and pertinent to their particular use cases. Think of it like a custom toolkit for each profession; just as a carpenter has specialized tools for woodworking, NIM provides tailored resources to meet the unique needs of various domains.

NIM is designed with a production-ready base container that offers a robust foundation for enterprise AI applications. It includes feature branches, thorough validation processes, enterprise support with service-level agreements (SLAs), and frequent security vulnerability updates. This optimized framework makes NIM an essential tool for deploying efficient, scalable, and tailored AI applications in production environments. Think of NIM as the bedrock of a skyscraper; just as a solid foundation is crucial for supporting the entire structure, NIM provides the necessary stability and resources for building scalable and reliable portable enterprise AI solutions.

NVIDIA NIM for Large Language Models

NVIDIA NIM for Large Language Models (LLMs) (NVIDIA NIM for LLMs) brings the power of state-of-the-art large language models (LLMs) to enterprise applications, providing unmatched natural language processing (NLP) and understanding capabilities.

Whether developing chatbots, content analyzers, or any application that needs to understand and generate human language — NVIDIA NIM for LLMs is the fastest path to inference. Built on the NVIDIA software platform, NVIDIA NIM brings state of the art GPU accelerated large language model serving.

High Performance Features

NVIDIA NIM for LLMs abstracts away model inference internals such as execution engine and runtime operations. NVIDIA NIM for LLMs provides the most performant option available whether it be with TensorRT, vLLM or LLM others.

●     Scalable Deployment: NVIDIA NIM for LLMs is performant and can easily and seamlessly scale from a few users to millions.

●     Advanced Language Models: Built on cutting-edge LLM architectures, NVIDIA NIM for LLMs provides optimized and pre-generated engines for a variety of popular models. NVIDIA NIM for LLMs includes tooling to help create GPU optimized models.

●     Flexible Integration: Easily incorporate the microservice into existing workflows and applications. NVIDIA NIM for LLMs provides an OpenAI API compatible programming model and custom NVIDIA extensions for additional functionality.

●     Enterprise-Grade Security: Data privacy is paramount. NVIDIA NIM for LLMs emphasizes security by using safetensors, constantly monitoring and patching CVEs in our stack and conducting internal penetration tests.

Applications

The potential applications of NVIDIA NIM for LLMs are vast, spanning across various industries and use cases:

●     Chatbots & Virtual Assistants: Empower bots with human-like language understanding and responsiveness.

●     Content Generation & Summarization: Generate high-quality content or distill lengthy articles into concise summaries with ease.

●     Sentiment Analysis: Understand user sentiments in real-time, driving better business decisions.

●     Language Translation: Break language barriers with efficient and accurate translation services.

Architecture

NVIDIA NIM for LLMs is one of what will become many NIMs. Each NIM has its own Docker container with a model, such as meta/llama3-8b-instruct. These containers include the runtime capable of running the model on any NVIDIA GPU. The NIM automatically downloads the model from NGC, leveraging a local filesystem cache if available. Each NIM is built from a common base, so once a NIM has been downloaded, downloading additional NIMs is extremely fast.

When a NIM is first deployed, NIM inspects the local hardware configuration, and the available optimized model in the model registry, and then automatically chooses the best version of the model for the available hardware. For a subset of NVIDIA GPUs, see: Support Matrix, NIM downloads the optimized TRT (TensorRT) engine and runs an inference using the TRT-LLM library. For all other NVIDIA GPUs, NIM downloads a non-optimized model and runs it using the vLLM library.

NIMs are distributed as NGC container images through the NVIDIA NGC Catalog. A security scan report is available for each container within the NGC catalog, which provides a security rating of that image, breakdown of CVE severity by package, and links to detailed information on CVEs.

Deployment Lifecycle

Figure 9 illustrates the deployment lifecycle.

NeMo Text Retriever NIM

NeMo Text Retriever NIM APIs facilitate access to optimized embedding models — essential components for RAG applications that deliver precise and faithful answers. By using NVIDIA software (including CUDA, TensorRT, and Triton Inference Server), the Text Retriever NIM provides the tools needed by developers to create ready-to-use, GPU-accelerated applications. NeMo Retriever Text Embedding NIM enhances the performance of text-based question-answering retrieval by generating optimized embeddings. For this RAG CVD, the Snowflake Arctic-Embed-L embedding model was harnessed to encode domain-specific content which was then stored in a vector database. The NIM combines that data with an embedded version of the user’s query to deliver a relevant response.

Figure 10 shows how the Text Retriever NIM APIs can help a question-answering RAG application find the most relevant data in an enterprise setting.

Enterprise-Ready Features

Text Embedding NIM comes with enterprise-ready features, such as a high-performance inference server, flexible integration, and enterprise-grade security.

●     High Performance: Text Embedding NIM is optimized for high-performance deep learning inference with NVIDIA TensorRT and NVIDIA Triton Inference Server.

●     Scalable Deployment: Text Embedding NIM seamlessly scales from a few users to millions.

●     Flexible Integration: Text Embedding NIM can be easily incorporated into existing data pipelines and applications. Developers are provided with an OpenAI-compatible API in addition to custom NVIDIA extensions.

●     Enterprise-Grade Security: Text Embedding NIM comes with security features such as the use of safetensors, continuous patching of CVEs, and constant monitoring with our internal penetration tests.

FlashStack Components

The FlashStack architecture was jointly developed by Cisco and Pure Storage. All FlashStack components are integrated, allowing customers to deploy the solution quickly and economically while eliminating many of the risks associated with researching, designing, building, and deploying similar solutions from the foundation. One of the main benefits of FlashStack is its ability to maintain consistency at scale. Figure 11 illustrates the series of hardware components used for building the FlashStack architectures.

All FlashStack components are integrated, so you can deploy the solution quickly and economically while eliminating many of the risks associated with researching, designing, building, and deploying similar solutions from the foundation. One of the main benefits of FlashStack is its ability to maintain consistency at scale. Each of the component families shown in Figure 11 (Cisco UCS, Cisco Nexus, Cisco MDS, Portworx by Pure Storage, Pure Storage FlashArray and FlashBlade systems) offers platform and resource options to scale up or scale out the infrastructure while supporting the same features and functions.

Benefits of Portworx Enterprise with OpenShift Container Platform

Portworx Enterprise is a multi-cloud solution, providing cloud-native storage for workloads running anywhere, from on-prem to cloud to hybrid/multi-cloud environments. Portworx Enterprise is deployed on Red Hat OpenShift cluster using the Red Hat certified Portworx Enterprise operator, available on Red Hat’s OperatorHub. Portworx with Red Hat OpenShift Container Platform enhances data management for container workloads by offering integrated, enterprise-grade storage. It includes simplified storage operations through Kubernetes, high availability and resiliency across environments, advanced disaster recovery options, and automated scaling capabilities. This integration supports a unified infrastructure where traditional and modern workloads coexist, providing flexibility in deployment across diverse infrastructures and ensuring robust data security.

Portworx and Stork offer a combined solution for managing persistent data and stateful applications within Red Hat OpenShift (OCP), providing features like storage orchestration, disaster recovery, and data protection. Portworx provides a software-defined storage solution that works natively within Kubernetes, including OpenShift, while Stork acts as a storage scheduler and orchestrator, enhancing the functionality of Portworx.

Benefits of Portworx Enterprise with Pure Storage FlashArray

Portworx on FlashArray offers flexible storage deployment options for Kubernetes. Using FlashArray as cloud drives enables automatic volume provisioning, cluster expansion, and supports PX Backup and Autopilot. Direct Access volumes allow for efficient on-premises storage management, offering file system operations, IOPS, and snapshot capabilities. Multi-tenancy features isolate storage access per user, enhancing security in shared environments.

Portworx on FlashArray enhances Kubernetes environments with robust data reduction, resiliency, simplicity, and support. It lowers storage costs through deduplication, compression, and thin provisioning, providing 2-10x data reduction. FlashArray’s reliable infrastructure ensures high availability, reducing server-side rebuilds. Portworx simplifies Kubernetes deployment with minimal configuration and end-to-end visibility via Pure1. Additionally, unified support, powered by Pure1 telemetry, offers centralized, proactive assistance for both storage hardware and Kubernetes services, creating an efficient and scalable solution for enterprise needs.

Benefits of Pure Storage FlashBlade

Pure Storage FlashBlade//S is a scale-out storage system that is designed to grow with your unstructured data needs for AI/ML, analytics, high performance computing (HPC), and other data-driven file and object use cases in areas of healthcare, genomics, financial services, and more. FlashBlade//S provides a simple, high-performance solution for Unified Fast File and Object (UFFO) storage with an all-QLC based, distributed architecture that can support NFS, SMB, and S3 protocol access. The cloud-based Pure1^® data management platform provides a single view to monitor, analyze, and optimize storage from a centralized location.

FlashBlade//S is the ideal data storage platform for AI, as it was purpose-built from the ground up for modern, unstructured workloads and accelerates AI processes with the most efficient storage platform at every step of your data pipeline. A centralized data platform in a deep learning architecture increases the productivity of AI engineers and data scientists and makes scaling and operations simpler and more agile for the data architect.

An AI project that uses a single-chassis system during early model development can expand non-disruptively as data requirements grow during training and continue to expand as more live data is accumulated during production.

Table 1.        FlashBlade//S Specifications

Solution Design

This chapter contains the following:

●     Design Considerations

●     FlashStack Topology

●     Red Hat OpenShift Container Platform on Bare Metal Server Configuration

●     RAG Design Summary

Design Considerations

The FlashStack Datacenter with Cisco UCS and Cisco Intersight meets the following general design requirements:

●     Resilient design across all the layers of infrastructure with no single point of failure

●     Scalable design with the flexibility to add compute capacity, storage, or network bandwidth as needed

●     Modular design that can be replicated to expand and grow as the needs of the business grow

●     Flexible design that can support different models of various components with ease

●     Simplified design with the ability to integrate and automate with external automation tools

●     Cloud-enabled design which can be configured, managed, and orchestrated from the cloud using GUI or APIs

●     Repeatable design for accelerating the provisioning of end-to-end Retrieval Augmented Generation pipeline

●     Provide a testing methodology to evaluate the performance of the solution

●     Provide an example implementation of Cisco Webex Chat Bot integration with RAG

To deliver a solution which meets all these design requirements, various solution components are connected and configured as explained in the following sections.

FlashStack Topology

The FlashStack for Accelerated RAG Pipeline with NVIDIA NIM and NIVIDIA Blueprint is built using the following reference hardware components:

●     One Cisco UCS X9508 chassis, equipped with a pair of Cisco UCS X9108 100G IFMs, contains six Cisco UCS X210c M7 compute nodes and two Cisco UCS X440p PCIe nodes each with two NVIDIA L40S GPUs. Other configurations of servers with and without GPUs are also supported. Each compute node is equipped with fifth-generation Cisco VIC card 15231 providing 100-G ethernet connectivity on each side of the fabric. A pair of Fabric Modules installed at the rear side of the chassis enables connectivity between the X440p PCIe nodes and X210c M7 nodes.

●     Cisco fifth-generation 6536 fabric interconnects are used to provide connectivity to the compute nodes installed in the chassis.

●     High-speed Cisco NXOS-based Nexus C93600CD-GX switching design to support up to 100 and 400-GE connectivity.

●     FlashBlade//S200 is used as S3 compatible object storage to persist Milvus Vector database large-scale files, such as index files and binary logs. It is directly exposed to Milvus vector database pods hosted on the OpenShift cluster. The OpenShift cluster is configured to route the object storage traffic from Milvus Pods to FlashBlade via worker node’s object storage network interface. Pure Storage FlashBlade is a unified and scale out storage platform providing native file and object storage, offering a modular architecture for unstructured data workloads, enabling independent scaling of compute and capacity.

●     FlashArray//XL170 is used as high performing back end storage for Portworx Enterprise which provides cloud native persistent storage with enterprise grade features for containers and other workloads running on the OpenShift cluster.

Figure 12 shows the physical topology and network connections used for this Ethernet-based FlashStack design.

The software components consist of:

●     Cisco Intersight platform to deploy, maintain, and support the FlashStack components.

●     Cisco Intersight Assist virtual appliance to help connect the Pure Storage FlashArray and Cisco Nexus Switches with  the Cisco Intersight platform to enable visibility into these platforms from Intersight.

●     Red Hat OpenShift Container Platform for providing a consistent hybrid cloud foundation for building and scaling containerized and virtualized applications.

●     Portworx by Pure Storage (Portworx Enterprise) data platform for providing enterprise grade storage for containerized workloads hosted on OpenShift platform.

●     Pure Storage Pure1 is a cloud-based, AI-driven SaaS platform that simplifies and optimizes data storage management for Pure Storage arrays, offering features like proactive monitoring, predictive analytics, and automated tasks.

FlashStack Cabling

The information in this section is provided as a reference for cabling the physical equipment in a FlashStack environment.

Compute Infrastructure Design

The compute infrastructure in FlashStack solution consists of the following:

●     Cisco UCS X210c M7 Compute Nodes

●     Cisco UCS X-Series chassis (Cisco UCSX-9508) with Intelligent Fabric Modules (Cisco UCSX-I-9108-25G)

●     Cisco UCS Fabric Interconnects (Cisco UCS-FI-6536)

Compute System Connectivity

The Cisco UCS X9508 Chassis is equipped with the Cisco UCSX 9108-100G intelligent fabric modules (IFMs). The Cisco UCS X9508 Chassis connects to each Cisco UCS 6536 FI using four 100GE ports, as shown in Figure 13. If you require more bandwidth, all eight ports on the IFMs can be connected to each FI.

Compute UCS Fabric Interconnect 6536 Ethernet Connectivity

Cisco UCS 6536 FIs are connected to Cisco Nexus 93600CD-GX switches using 100GE connections configured as virtual port channels. Each FI is connected to both Cisco Nexus switches using a 100G connections; additional links can easily be added to the port channel to increase the bandwidth as needed. Below figure illustrates the physical connectivity details.

Pure Storage FlashArray//XL170 Ethernet Connectivity

Pure Storage FlashArray controllers are connected to Cisco Nexus 93600CD-GX switches using redundant 100-GE. Figure 15 illustrates the physical connectivity details.

Pure Storage FlashBlade//S200 Ethernet Connectivity

Pure Storage FlashBlade uplink ports (2x 100GbE from each FIOM) are connected to Cisco Nexus 93600CD-GX switches as shown in Figure 16. Additional links can easily be added (up to 8x 100GbE on each FIOM) to the port channel to increase the bandwidth as needed. Following figure illustrates the physical connectivity details.

Red Hat OpenShift Container Platform on Bare Metal Server Configuration

A simple Red Hat OpenShift cluster consists of at least five servers – 3 Master or Control Plane Nodes and 2 or more Worker or compute Nodes where applications and VMs are run. In this lab validation 3 Worker Nodes were utilized. Based on published Red Hat requirements, the three Master Nodes were configured with 64GB RAM, and the three Worker Nodes were configured with 1024GB memory to handle containerized applications and Virtual Machines. Each Node was booted from RAID1 disk created using two M.2 SSD drives. Also, the servers paired with X440p PCIe Nodes were configured as Workers. From a networking perspective, both the Masters and the Workers were configured with a single vNIC with UCS Fabric Failover in the Bare Metal or Management VLAN. The Workers were configured with extra NICs (vNICs) to allow storage attachment to the Workers.

Each worker node is configured with two additional vNICs with the iSCSI A and B VLANs tagged as native to allow iSCSI persistent storage attachment from FlashArray//XL170 and future iSCSI boot. Finally, each worker is also configured with one additional vNIC with the OCP Object-Storage VLAN tagged as native VLAN to provide object persistent storage from FlashBlade//S200.

Worker Node Network Configuration

The worker node is configured with four vNICs. The first three vNICs are used by worker node for OpenShift cluster management traffic (eno5) and storage traffic using iSCSI protocol (eno6 and eno7). While the last vNIC (eno8) is used for object storage traffic over ethernet. vNICs eno5 and eno8 vNICs are configured with Fabric-Failover option while the vNICs eno6 and eno7 are used as independent interfaces for iSCSI storage traffic via Fabric-A and B, respectively. The following figure illustrates the vNIC configuration of OpenShift worker nodes.

The control plane (master node) will just have one vNIC eno5 with Fabric-Failover option enabled. This vNIC is used to carry the worker management and OCP cluster traffic.

VLAN Configuration

Table 2 lists the VLANs configured for setting up the FlashStack environment along with their usage.

Table 2.        VLAN Usage

Table 3 lists the infrastructure services running on either virtual machines or bar mental servers required for deployment as outlined in the document. All these services are hosted on pre-existing infrastructure with in the FlashStack

Table 3.        Infrastructure services

Software Revisions

The FlashStack Solution with Red Hat OpenShift on Bare Metal infrastructure configuration is built using the following components.

Table 4 lists the required software revisions for various components of the solution.

Table 4.        Software Revisions

RAG Design Summary

This solution implements a validated Retrieval Augmented Generation (RAG) pipeline architected to enhance Large Language Model (LLM) capabilities with real-time access to enterprise-specific data, thereby increasing user trust and mitigating hallucinations. The design adheres to core RAG principles aligning with the robust methodologies.

 

This entire RAG pipeline is deployed upon a validated FlashStack architecture, specifically configured following the best practices outlined here: FlashStack with Red Hat OpenShift Container and Virtualization Platform using Cisco UCS X-Series Design and Deployment Guide.

The layered infrastructure, running on Red Hat OpenShift Container Platform with persistent storage managed by Portworx Enterprise, provides a high-performance, secure, and scalable environment. The use of locally deployed NVIDIA NIMs within this secure infrastructure ensures data privacy and control.

This design leverages the NVIDIA RAG Blueprint as a foundational framework, implemented with optimized NVIDIA NIMs. It directly addresses the goals of RAG by providing up-to-date, proprietary information to the LLM securely. The underlying FlashStack platform offers significant performance and reliability, while also being extensible for future AI initiatives such as model fine-tuning, training, or other inferencing use cases, contingent on appropriate resource allocation.

Network Switch Configuration

This chapter contains the following:

●     Physical Connectivity

●     Cisco Nexus Switch Manual Configuration

●     Claim Cisco Nexus Switches into Cisco Intersight

Physical Connectivity

Physical cabling should be completed by following the diagram and table references in section FlashStack Cabling.

The following procedures describe how to configure the Cisco Nexus 93600CD-GX switches for use in a FlashStack environment. This procedure assumes the use of Cisco Nexus 9000 10.1(2), the Cisco suggested Nexus switch release at the time of this validation.

The procedure includes the setup of NTP distribution on both the mgmt0 port and the in-band management VLAN. The interface-vlan feature and ntp commands are used to set this up. This procedure also assumes that the default VRF is used to route the in-band management VLAN.

This document assumes that initial day-0 switch configuration is already done using switch console ports and ready to use the switches using their management IPs.

Cisco Nexus Switch Manual Configuration

Procedure 1.     Enable features on Cisco Nexus A and Cisco Nexus B

Step 1.          Log into both Nexus switches as admin using ssh.

Step 2.          Enable the switch features as described below:

Procedure 2.     Set Global Configurations on Cisco Nexus A and Cisco Nexus B

Step 1.          Log into both Nexus switches as admin using ssh.

Step 2.          Run the following commands to set the global configurations:

Sample clock commands for the United States Eastern timezone are:

clock timezone EST -5 0

clock summer-time EDT 2 Sunday March 02:00 1 Sunday November 02:00 60

Procedure 3.     Create VLANs on Cisco Nexus A and Cisco Nexus B

Step 1.          From the global configuration mode, run the following commands:

Procedure 4.     Add NTP Distribution Interface

Cisco Nexus - A

Step 1.          From the global configuration mode, run the following commands:

Cisco Nexus - B

Step 1.          From the global configuration mode, run the following commands:

Procedure 5.     Define Port Channels on Cisco Nexus A and Cisco Nexus B

Cisco Nexus – A and B

Step 1.          From the global configuration mode, run the following commands:

Procedure 6.     Configure Virtual Port Channel Domain on Nexus A and Cisco Nexus B

Cisco Nexus - A

Step 1.          From the global configuration mode, run the following commands:

Cisco Nexus - B

Step 1.          From the global configuration mode, run the following commands:

Procedure 7.     Configure individual Interfaces

Cisco Nexus-A

Step 1.          From the global configuration mode, run the following commands:

Cisco Nexus-B

Step 1.          From the global configuration mode, run the following commands:

Procedure 8.     Update the port channels

Cisco Nexus-A and B

Step 1.          From the global configuration mode, run the following commands:

Step 2.          To check for correct switch configuration, run the following commands:

Cisco Nexus Configuration for Storage Traffic

Procedure 1.     Configure Interfaces for Pure Storage on Cisco Nexus and Cisco Nexus B

Cisco Nexus - A

Step 1.          From the global configuration mode, run the following commands:

Cisco Nexus - B

Step 1.          From the global configuration mode, run the following commands:

Claim Cisco Nexus Switches into Cisco Intersight

Cisco Nexus switches can be claimed into the Cisco Intersight either using Cisco Intersight Assist or Direct claim using Device ID and Claim Codes.

This section provides the steps to claim the Cisco Nexus switches using Cisco Intersight Assist.

Procedure 1.     Claim Cisco Nexus Switches into Cisco Intersight using Cisco Intersight Assist

Cisco Nexus - A

Step 1.          Log into Nexus Switches and confirm the nxapi feature is enabled:

Step 2.          Log into Cisco Intersight with your login credentials. From the drop-down list select System.

Step 3.          Under Admin, click Target then click Claim a New Target. Under Categories, select Network, click Cisco Nexus Switch, and then click Start.

Step 4.          Select the Cisco Assist name which is already deployed and configured. Provide the Cisco Nexus Switch management IP address, username and password details and click Claim.

Step 5.          Repeat steps 1 through 4 to claim the remaining Switch B.

Step 6.          When the storage is successfully claimed, from the drop-down list, select Infrastructure Services. Under Operate, click Networking tab. On the right you will find the newly claimed Cisco Nexus switch details and browse through the Switches for viewing the inventory details.

The L2 neighbors of the Cisco Nexus Switch-A is shown below:

Cisco Intersight Managed Mode Configuration for Cisco UCS

The chapter contains the following:

●     Fabric Interconnect Domain Profile and Policies

●     Server Profile Templates and Policies

●     Create Pools

●     vNIC Templates and vNICs

●     Ethernet Adapter Policy for Storage Traffic

●     Storage Policy

●     Compute Configuration Policies

●     Management Configuration Policies

The procedures in this chapter describe how to configure a Cisco UCS domain for use in a base FlashStack environment. A Cisco UCS domain is defined as a pair for Cisco UCS FIs and all the servers connected to it. These can be managed using two methods: UCSM and IMM. The procedures detailed below are for Cisco UCS Fabric Interconnects running in Intersight managed mode (IMM).

The Cisco Intersight platform is a management solution delivered as a service with embedded analytics for Cisco and third-party IT infrastructures. The Cisco Intersight Managed Mode (also referred to as Cisco IMM or Intersight Managed Mode) is an architecture that manages Cisco Unified Computing System (Cisco UCS) fabric interconnect–attached systems through a Redfish-based standard model. Cisco Intersight managed mode standardizes both policy and operation management for Cisco UCS C-Series M7 and Cisco UCS X210c M7 compute nodes used in this deployment guide.

 

Procedure 1.     Fabric Interconnect Domain Profile and Policies

Step 1.          Log into the Intersight portal and select Infrastructure  Service. On the left select Profiles then under Profiles select UCS Domain Profiles.

Step 2.          Click Create UCS Domain Profile to create a new domain profile for Fabric Interconnects. Under the General tab, select the Default Organization, enter name and descriptions of the profile.

Step 3.          Click Next to go to UCS Domain Assignment. Click Assign Later.

Step 4.          Click Next to go to VLAN & VSAN Configuration.

Step 5.          Under VLAN & VSAN Configuration > VLAN Configuration, click Select Policy then click Create New.

Step 6.          On the Create VLAN page, go to the General tab, enter a name (AA06-FI-VLANs), and click Next to go to Policy Details.

Step 7.          To add a VLAN, click Add VLANs.

Step 8.          For the Prefix, enter the VLAN name as OOB-Mgmt-VLAN. For the VLAN ID, enter the VLAN ID 1061. Leave Auto Allow on Uplinks enabled and Enable VLAN Sharing disabled.

Step 9.          Under Multicast Policy, click Select Policy and select Create New to create a Multicast policy.

Step 10.       On the Create Multicast Policy page, enter the name (AA06-FI-MultiCast) of the policy and click Next to go to Policy Details. Leave the Snooping State and Source IP Proxy state checked/enabled and click Create. Select the newly created Multicast policy.

Step 11.       Repeat steps 1 through 10 to add all the required VLANs to the VLAN policy.

Step 12.       After adding all the VLANs, click Set Native VLAN ID and enter the native VLANs (for example 2) and click Create. The VLANs used for this solution are shown below:

Step 13.       Select the newly created VLAN policy for both Fabric Interconnects A and B. Click Next to go to Port Configuration.

Step 14.       Enter the name of the policy (AA06-FI-PortConfig) and click Next then click Next again to go to Port Roles Page.

Step 15.       In the right pane, under ports, select port 1 and 2 and click Configure.

Step 16.       Set Role as Server and leave Auto Negotiation enabled and click Save.

Step 17.       In the right pane click the Port Channel tab and click Create Port Channel.

Step 18.       For the Role, select Ethernet Uplink Port Channel. Enter 201 as Port Channel ID. Set Admin speed as 100Gbps and FEC as Cl91.

Step 19.       Under Link Control, create a new link control policy with the following options. Once created, select the policy.

Table 5.        UDLD policy

Step 20.       For the Uplink Port Channel select Ports 1 and 2 and click Create to complete the Port Roles policy.

Step 21.       Click Next to go to UCS Domain Configuration page.

Table 6 lists the Management and Network related policies that are created and used.

Table 6.        NTP policy

Table 7.        Network Connectivity Policy

Table 8.        SNMP Policy

Table 9.        QoS Policy

Step 22.       When the UCS Domain profile is created with the above mentioned policies, edit the policy and assign it to the Fabric Interconnects.

Intersight will go through the discovery process and discover all the Cisco UCS C and X -Series compute nodes attached to the Fabric Interconnects.

Procedure 2.     Server Profile Templates and Policies

In the Cisco Intersight platform, a server profile enables resource management by simplifying policy alignment and server configuration. The server profiles are derived from a server profile template. A Server profile template and its associated policies can be created using the server profile template wizard. After creating the server profile template, you can derive multiple consistent server profiles from the template.

The server profile templates captured in this deployment guide supports Cisco UCS X210c M7 compute nodes with 5th Generation VICs and can be modified to support other Cisco UCS blades and rack mount servers.

Create Pools

The following pools need to be created before proceeding with server profile template creation.

MAC Pools

Table 10 lists the two MAC pools for the vNICs that will be configured in the templates.

Table 10.     MAC Pool Names and Address Ranges

UUID pool

Table 11 lists the settings for the UUID pools.

Table 11.     UUID Pool Names and Settings

Out-Of-Band (OOB) Management IP Pool

A OOB management IP pool (AA06-OCP-OOB-MGMT-IPPool) is created with following settings:

vNIC Templates and vNICs

In this deployment, separate server profile templates are created for Worker and Master Nodes where Worker Nodes have storage network interfaces to support workloads, but Master Nodes do not. The vNIC layout is explained below. While most of the policies are common across various templates, the LAN connectivity policies are unique and use the information in the tables below.

The following vNIC templates are used for deriving the vNICs for OpenShift worker nodes for host management, VM management and iSCSI storage traffics.

Table 12.     vNIC Templates for Ethernet Traffic

Ethernet Adapter Policy for Storage Traffic

The ethernet adapter policy is used to set the interrupts, send and receive queues, and queue ring size. The values are set according to the best-practices guidance for the operating system in use. Cisco Intersight provides a default Linux Ethernet Adapter policy for typical Linux deployments.

You can optionally configure a tweaked ethernet adapter policy for additional hardware receive queues handled by multiple CPUs in scenarios where there is a lot of traffic and multiple flows. In this deployment, a modified ethernet adapter policy, AA06-EthAdapter-16RXQs-5G, is created and attached to storage vNICs. Non-storage vNICs will use the default Linux-v2 Ethernet Adapter policy. Table 13 lists the settings that are changed from defaults in the Adapter policy used for the iSCSI traffic. The remaining settings are left at defaults.

Table 13.     Settings and Values

Using the templates listed in Table 12, separate LAN connectivity policies are created for control and worker nodes.

Control nodes are configured with one vNIC which is derived from the AA06-OCP-Mgmt-vNIC template. The following screenshot shows the LAN connectivity policy (AA06-OCP-master-LANCon) created with one vNIC for control node.

 

Worker nodes are configured with four vNICs which are derived from the templates discussed above. Following screenshot shows the LAN connectivity policy (AA06-OCP-Worker-LANConn) created with four vNICs for worker nodes.

Storage Policy

For this solution, Cisco UCS X210c nodes are configured to boot from local M.2 SSD disks. Two M.2 disks are used and configured with RAID-1 configuration. Boot from SAN option will be supported in the next releases. The following screenshot shows the storage policy (AA06-OCP-Storage-M2R1), and the settings used for configuring the M.2 disks in RAID-1 mode.

Compute Configuration Policies

Boot Policy

To facilitate the automatic boot from the Red Hat CoreOS Discovery ISO image, CIMC Mapped DVD boot option is used. The following boot policy is used for both controller and workers nodes.

Local Disk boot option being at the top ensures that the nodes always boot from the M.2 disks once after CoreOS installed. The CIMC Mapped DVC option at the second is used to install the CoreOS using Discovery ISO which is mapped using a Virtual Media policy (CIMCMap-ISO). KVM Mapped DVD will be used if you want to manually mount any ISO to the KVM session of the  server and install the OS. This option will be used when installing CoreOS during the OpenShift cluster expansion by adding additional worker node.

Virtual Media (vMedia) Policy

Virtual Media policy is used to mount the Red Hat CoreOS Discovery ISO to the server using CIMC Mapped DVD policy as previously explained. A file share service is required to be configured and must be accessed by OOB-Mgmt network. In this solution, the HTTP file share service is used to share the Discovery ISO over the network.

Procedure 1.     Bios Policy

Step 1.          Create the BIOS policy and select the pre-defined policy as shown below and click Next.

 

Step 2.          Expand Server Management and set Consistent Device Name (CDN) to enabled for Consistent Device Naming within the Operating System.

Step 3.          Click Create to complete the BIOS policy.

Procedure 2.     Firmware Policy (optional)

Step 1.          Create a Firmware policy (AA06-OCP-FW) and under the Policy Detail tab, set the Server Model as UCSX-210C-M7 and set Firmware Version to the latest version. The following screenshot shows the firmware policy used in this solution:

Procedure 3.     Create a Power Policy

Step 1.          Select All Platform (unless you want to create a dedicated power policy for FI-Attached servers). Select the following options and leave the rest of the settings at default. When you apply this policy to the server profile template, the system will take appropriate settings and apply to the server.

Management Configuration Policies

The following policies will be added to the management configuration:

●     IMC Access to define the pool of IP addresses for compute node KVM access

●     IPMI Over LAN to allow the servers to be managed by IPMI or redfish through the BMC or CIMC

●     Local User to provide local administrator to access KVM

●     Virtual KVM to allow the Tunneled KVM

Cisco IMC Access Policy

Create a CIMC Access Policy with settings as shown in the following screenshot.

IPMI over LAN and Local User Policies

The IPMI Over LAN Policy can be used to allow both IPMI and Redfish connectivity to Cisco UCS Servers. Red Hat OpenShift platform uses these two policies to power manage (power off, restart, and so on) the baremetal servers.

Create the IPMI over LAN policy (AA06-IPMIOvelLan) as shown below:

Virtual KVM Policy

The following screenshot shows the virtual KVM policy (AA06-OCP-VirtualKVM) used in the solution:

Create Server Profile Templates

When you have the required pools, polices, vNIC templates created, Server profile templates can be created. Two separate Server Profile Templates are used for control and workers node.

Table 14 lists the polices and pools used to create the Server Profile template (AA06-OCP-Master-M.2) for Control nodes.

Table 14.     Policies and Pools for Control Nodes

Table 15 lists the polices and pools used to create the Server Profile template (AA06-OCP-Worker-M.2) for worker nodes.

Table 15.     Policies and Pools for Worder Nodes

The following screenshot shows the two server profile templates created for the control and worker nodes:

Create Server Profiles

Once Server Profile Templates are created, the server profiles can be derived from the template. The following screenshot shows a total of six profiles are derived (three for control nodes and three for worker nodes).

When the Server profiles are created, associate these server profiles to the control and workers nodes as shown below:

Now the Cisco UCS X210c M7 blades are ready and OpenShift can be installed on these machines.

Pure Storage FlashArray Configuration

This chapter contains the following:

●     Configure iSCSI Interfaces

●     Claim Pure Storage FlashArray//XL170 into Intersight

In this solution, Pure Storage FlashArray//XL170 is used as the storage provider for all the application pods and virtual machines provisioned on the OpenShift cluster using Portworx Enterprise. The Pure Storage FlashArray//XL170 array will be used as Cloud Storage Provider for Portworx which allows us to store data on-premises with FlashArray while benefiting from Portworx Enterprise cloud drive features.

This chapter describes the high-level steps to configure Pure Storage FlashArray//X170 network interfaces required for storage connectivity over iSCSI. For this solution, Pure Storage FlashArray was loaded with Purity//FA Version 6.6.10.

The compute nodes are redundantly connected to the storage controllers through 4 x 100Gb connections (2 x 100Gb per storage controller module) from the redundant Cisco Nexus switches.

The Pure Storage FlashArray network settings were configured with three subnets across three VLANs. Storage Interfaces CT0.Eth0 and CT1.Eth0 were configured to access management for the storage on VLAN 1063. Storage Interfaces (CT0.Eth18, CT0.Eth19, CT1.Eth18, and CT1.Eth19) were configured to run iSCSI Storage network traffic on the VLAN 3010 and VLAN 3020.

The following tables provide the IP addressing configured on the interfaces used for storage access.

Table 16.     iSCSI A Pure Storage FlashArray//XL170 Interface Configuration Settings

Table 17.     iSCSI B Pure Storage FlashArray//XL170 Interface Configuration Settings

Procedure 1.     Configure iSCSI Interfaces

Step 1.          Log into Pure FlashArray//XL170 using its management IP addresses.

Step 2.          Click Settings > Network > Connectors > Ethernet.

Step 3.          Click Edit for Interface CT0.eth18.

Step 4.          Click Enable and add the IP information from Table 16 and Table 17 and set the MTU to 9000.

Step 5.          Click Save.

Step 6.          Repeat steps 1 through 5 to configure the remaining interfaces CT0.eth19, CT1.eth18 and CT1.eth19.

Procedure 2.     Claim Pure Storage FlashArray//XL170 into Intersight

Step 1.          Log into Cisco Intersight using your login credentials. From the drop-down menu select System.

Step 2.          Under Admin, select Target and click Claim a New Target. Under Categories, select Storage, click Pure Storage FlashArray and then click Start.

Step 3.          Select the Cisco Assist name which is already deployed and configured. Provide the Pure Storage FlashArray management IP address, username, and password details and click Claim.

Step 4.          When the storage is successfully claimed, from the drop-down list, select Infrastructure Services. Under Operate, click Storage. You will see the newly claimed Pure Storage FlashArray; browse through it to view the inventory details.

Pure Storage FlashBlade Configuration

This chapter contains the following:

●     Object Store Configuration

In this solution, Pure Storage FlashBlade//S200 is used as the persistent object storage provider for the Milvus vector database hosted on the OpenShift cluster. The FlashBlade is directly accessed by the Milvus vector database pod by using Workers node’s interface that carries object storage traffic (eno8). This section describes high-level steps to configure Pure Storage FlashBlade//S200 network interfaces required for storage connectivity over ethernet. Pure Storage FlashBlade was loaded with Purity//FB V4.1.12.

The FlashBlade//S200 provides up to 8x 100GbE interfaces for data traffic. In this solution, 2x 100GbE from each FIOM (with aggregated network bandwidth of 400 GbE) are connected to a pair of Nexus switches. For more details about FlashBlade connectivity, see section Pure Storage FlashBlade//S200 Ethernet Connectivity.

Table 18 provides the IP addressing configured on the interfaces used for objects storage access.

Table 18.     Link Aggregation Groups (LAG)

A Link Aggregation Group is created using CH1.FM1.ETH3, CH1.FM1.ETH4, CH1.FM2.ETH3, and CH1.FM2.ETH4 interfaces as shown below. Notice that the aggregated bandwidth of the LAG is 400GbE as it is created with 4x 100GbE interfaces as shown below:

Once the LAG is created, a Subnet (AA06-Object) and an interface (AA06-Obj-Interface) is created as shown below. For the subnet, ensure to set MTU as 9000, VLAN as 3040 and select  “aa03” for the LAG. Ensure to set Services as “Data” for the interface.

Object Store Configuration

FlashBlade Object Store configuration involves creating an Object Store Account, user, Access Keys and finally a bucket.

Procedure 1.     Create a Object Store Account, User, Access Keys and Bucket

Step 1.          Log into Pure FlashBlade//S200 using its management IP addresses.

Step 2.          Click Storage > Object Store > Accounts > Click + to create an account.

Step 3.          Provide a name for Account, Quota Limit and Bucket Default Quota Limit as per your requirements. Click on Create.

Step 4.          Click the newly created Account name and click + to create a new user for the account.

Step 5.          Provide a username and click Create.

Step 6.          In the Add Access Policies window, select the pre-defined access policies as shown below. Or create your own access policy with a set of rules and finally select it.

Step 7.          Click Add when Access Policy is selected.

Step 8.          Click Create a new key and click Create to create a pair of access and secret keys. Preserve the Access and Secret Keys for later use. Click Close.

Step 9.          Click Account name and go to the Buckets sections. Click + to create a bucket and add it to the account.

Step 10.       Provide a bucket name and Quota Limit. Click Create.

This completes the FlashBlade configuration for object store access by the Milvus vector database pods.

OpenShift Container Platform Installation and Configuration

This chapter contains the following:

●     OpenShift Container Platform – Installation Requirements

●     Prerequisites

●     Network Requirements

OpenShift 4.16 is deployed on the Cisco UCS infrastructure as M.2 booted bare metal servers. The Cisco UCS X210C M7 servers need to be equipped with an M.2 controller (SATA or NVMe) card and two identical M.2 drives. Three master nodes and three worker nodes are deployed in the validation environment and additional worker nodes can easily be added to increase the scalability of the solution. This document will guide you through the process of using the Assisted Installer to deploy OpenShift 4.17.

OpenShift Container Platform – Installation Requirements

The Red Hat OpenShift Assisted Installer provides support for installing OpenShift Container Platform on bare metal nodes. This guide provides a methodology to achieving a successful installation using the Assisted Installer.

Prerequisites

The FlashStack for OpenShift utilizes the Assisted Installer for OpenShift installation. Therefore, when provisioning and managing the FlashStack infrastructure, you must provide all the supporting cluster infrastructure and resources, including an installer VM or host, networking, storage, and individual cluster machines.

The following supporting cluster resources are required for the Assisted Installer installation:

●     The control plane and compute machines that make up the cluster

●     Cluster networking

●     Storage for the cluster infrastructure and applications

●     The Installer VM or Host

Network Requirements

The following infrastructure services need to be deployed to support the OpenShift cluster, during the validation of this solution we have provided VMs on your hypervisor of choice to run the required services. You can use the existing DNS and DHCP services available in the data center.

There are various infrastructure services prerequisites for deploying OpenShift 4.17. These prerequisites are as follows:

●     DNS and DHCP services – these services were configured on Microsoft Windows Server VMs in this validation

●     NTP Distribution was done with Nexus switches

●     Specific DNS entries for deploying OpenShift – added to the DNS server

●     A Linux VM for initial automated installation and cluster management – a Rocky Linux / RHEL VM with appropriate packages

NTP

Each OpenShift Container Platform node in the cluster must have access to at least two NTP servers.

NICs

NICs configured on the Cisco UCS servers based on the design previously discussed.

DNS

Clients access the OpenShift Container Platform cluster nodes over the bare metal network. Configure a subdomain or subzone where the canonical name extension is the cluster name.

The following domain and OpenShift cluster names are used in this deployment guide:

●     Base Domain: flashstack.local

●     OpenShift  Cluster Name: fs-ocp1

The DNS domain name for the OpenShift  cluster should be the cluster name followed by the base domain, for example fs-ocp1. flashstack.local.

Table 19 lists the information for fully qualified domain names used during validation. The API and Nameserver addresses begin with canonical name extensions. The hostnames of the control plane and worker nodes are exemplary, so you can use any host naming convention you prefer.

Table 19.     DNS FQDN Names Used

DHCP

For the bare metal network, a network administrator must reserve several IP addresses, including:

●     One IP address for the API endpoint

●     One IP address for the wildcard Ingress endpoint

●     One IP address for each master node (DHCP server assigns to the node)

●     One IP address for each worker node (DHCP server assigns to the node)

Procedure 1.     Gather MAC Addresses of Node Bare Metal Interfaces

Step 1.          Log into Cisco Intersight.

Step 2.          Go to Infrastructure Service > Profiles > UCS Server Profile (for example, AA06-OCP-Worker-M.2_3).

Step 3.          In the center pane, go to Inventory > Network Adapters > Network Adapter (for example, UCSX-ML-V5D200G).

Step 4.          In the center pane, click Interfaces.

Step 5.          Record the MAC address for NIC Interface eno5.

Step 6.          Select the General tab and click Identifiers.

Step 7.          Record the Management IP assigned from the AA06-OCP-OOB-MGMT-IP Pool.

Table 20 lists the IP addresses used for the OpenShift cluster including bare metal network IPs and UCS KVM Management IPs for IMPI or Redfish access.

Table 20.     Host BMC Information

Step 8.          From Table 20, enter the hostnames, IP addresses, and MAC addresses as reservations in your DHCP and DNS server(s) or configure the DHCP server to dynamically update DNS.

Step 9.          You need to pipe VLAN interfaces for all 3 storage VLANs (3010, 3020 and 3040) and 1 management VLANs (1061) into your DHCP server(s) and assign IPs in the storage networks on those interfaces.

Step 10.       Create a DHCP scope for each management and storage VLANs with the appropriate subnets.

Step 11.       Ensure that the IPs assigned by the scope do not overlap with the already consumed IPs (like FlashArray//XL170 storage iSCSI interface IPs, FlashBlade//S200 interfaces and OpenShift reserved IPs).

Step 12.       Enter the nodes in the DNS server or configure the DHCP server to forward entries to the DNS server. For the cluster nodes, create reservations to map the hostnames to the desired IP addresses as shown below:

Step 13.       Setup either a VM (installer/bastion node) or spare server with the network interface connected to the Bare Metal VLAN.

Step 14.       Install either Red Hat Enterprise Linux (RHEL) 9.4 or Rocky Linux 9.4 Server with GUI and create an administrator user. Once the VM or host is up and running, update it and install and configure XRDP. Connect to this host with a Windows Remote Desktop client as the admin user.

Step 15.       ssh into the installer node VM, open a terminal session and create an SSH key pair to use to communicate with the OpenShift hosts:

Step 16.       Copy the public SSH key to the user directory:

Step 17.       Add the private key to the ssh-agent:

Procedure 2.     Install Red Hat OpenShift Container Platform using the Assisted Installer

Step 1.          Launch Firefox and connect to https://console.redhat.com/openshift/cluster-list. Log into your Red Hat account.

Step 2.          Click Create cluster to create an OpenShift cluster.

Step 3.          Select Datacenter and then select Bare Metal (x86_64).

Step 4.          Select Interactive to launch the Assisted Installer.

Step 5.          Provide the cluster name and base domain.

Step 6.          Select the latest OpenShift version, scroll down and click Next.

Step 7.          Select the latest OpenShift version, scroll down and click Next.

Step 8.          Select Install OpenShift Virtualization operator and click Next.

Step 9.          Click Add hosts.

Step 10.       Under Provisioning type, from the drop-down list select the Full Image file. Under SSH public key, click Browse and browse to, select, and open the id_ed25519.pub file. The contents of the public key should now appear in the box. Click Generate Discovery ISO and click Download Discovery ISO to download the Discovery ISO.

Step 11.       Copy the Discovery ISO to a http or https file share server, use a web browser to get a copy of the URL for the Discovery ISO.

Step 12.       Log into Cisco Intersight and update the virtual Media policy with the Discovery ISO URL as shown below. This Discovery ISO image will be mapped to the server using CIMC Mapped DVD option defined in the Boot policy.

Step 13.       Go to Operate > Power > Reset System to reset first five UCSX-201c M7 servers.

Step 14.       When all five servers have booted RHEL CoreOS (Live) from the Discovery ISO, they will appear in the Assisted Installer. From the drop-down lists under Role assign the appropriate server roles. Scroll down and click Next.

Step 15.       Expand each node and confirm the role of the M.2 disk is set to Installation disk. Click Next.

Step 16.       Under Network Management, make sure Cluster-Managed Networking is selected. Under Machine network, from the drop-down list, select the subnet for the BareMetal VLAN. Enter the API IP for the api.cluster.basedomain entry in the DNS servers. For the Ingress IP, enter the IP for the *.apps.cluster.basedomain entry in the DNS servers.

Step 17.       Scroll down. All nodes should have a status of Ready.

Step 18.       When all the nodes are in ready status, click Next.

Step 19.       Review the information and click Install cluster to begin the cluster installation.

Step 20.       On the Installation progress page, expand the Host inventory. The installation will take 30-45 minutes. When the installation is complete, all nodes will show a Status of Installed.

Step 21.       Select Download kubeconfig to download the kubeconfig file. In a terminal window, setup a cluster directory and save credentials:

Step 22.       In the Assisted Installer, click the icon to copy the kubeadmin password:

Step 23.       Click Open console to launch the OpenShift Console. Log in using the kubeadmin and the kubeadmin password.

Step 24.       Click the ? mask. Links for various tools are provided in that page. Download oc for Linux for x86_64 and virtctl for Linux for x86_64 Common Line Tools.

Step 25.       To enable oc tab completion for bash, run the following:

Step 26.       In Cisco Intersight, edit the Virtual Media policy and remove the link to the Discovery ISO.

Step 27.       Click Save & Deploy then click Save & Proceed.

Step 28.       Do not select “Reboot Immediately to Activate.”

Step 29.       Click Deploy. The virtual media mount will be removed from the servers without rebooting them.

Step 30.       In Firefox, in the Assisted Installer page, click Open console to launch the OpenShift Console. Log in using the kubeadmin and the kubeadmin password.

Step 31.       Go to Compute > Nodes to see the status of the OpenShift nodes.

Step 32.       In the Red Hat OpenShift console, go to Compute > Bare Metal Hosts. For each Bare Metal Host, click the ellipses to the right of the host and select Edit Bare Metal Host. Select Enable power management.

Step 33.       From Table 20, fill in the BMC Address. Also, make sure the Boot MAC Address matches the MAC address in Table 20. For the BMC Username and BMC Password, use what was entered into the Cisco Intersight IPMI over LAN policy. Click Save to save the changes. Repeat this step for all Bare Metal Hosts.

Step 34.       Go to Compute > Bare Metal Hosts. When all hosts have been configured, the Status displays “Externally provisioned,” and the Management Address are populated. You can now manage power on the OpenShift hosts from the OpenShift console.

 

Step 35.       To automatically determine and allocate the system-reserved resources on nodes, create a KubeletConfig CUSTOM RESOURCE (CR) to set the autoSizingReserved: true parameter as shown below and apply the machine configuration files:

Expand OpenShift Cluster by Adding a Worker Node

This chapter contains the following:

●     OpenShift Cluster expansion

●     Link the Machine and Bare Metal Host, Node and Bare Metal Host

This chapter provides detailed steps to scale up the worker nodes of OpenShift cluster by adding a new worker node to the existing cluster. For this exercise, the sixth blade in the chassis will be used and be part of the cluster by the end of this exercise.

Procedure 1.     OpenShift Cluster expansion

Step 1.          Launch Firefox and go to https://console.redhat.com/openshift/cluster-list. Log into your Red Hat account.

Step 2.          Click your cluster name and go to Add Hosts.

Step 3.          Under Host Discovery, click Add hosts.

Step 4.          In the Add hosts wizard, for the CPU architecture select x86_64 and for the Host’s network configuration select DHCP Only. Click Next.

Step 5.          For the Provision type select Full image file from the drop-down list, for SSH public key browse or copy/paste the contents of id-ed25519.pub file. Click Generate Discovery ISO and when the file is generated and click Download Discovery ISO file.

Step 6.          Copy the Discovery ISO to a http or https file share server, use a web browser to get a copy of the URL for the new Discovery ISO.

Step 7.          Log into Cisco Intersight and update the virtual Media policy as explained in the previous section. This Discovery ISO image is a mapped server using the CIMC Mapped DVD option. Go to Power > Reset System to reset the sixth UCSX-201c M7 server.

Step 8.          When the server has booted “RHEL CoreOS (live)” from the newly generated Discovery ISO, it will appear in the assisted installer under Add hosts.

 

Step 9.          When the node status shows Ready, click Install ready hosts. After few minutes, the required components will be installed on the node and displays the status as Installed.

Step 10.       When the server successfully produces the CoreOS installed, log into Cisco Intersight, edit the vMedia policy and remove the virtual media mount. Go to Profiles > Server Profiles page, deploy the profile to the newly added worker profile without rebooting the host. The Inconsistent state on the remaining profiles should be cleared.

Step 11.       Log into the cluster with kubeadmin user and go to Compute > Nodes >  and select the newly added worker node and approve the Cluster join request of the worker node and request for server certificate signing.

Step 12.       Wait for a few seconds and the node will be ready and the pods are scheduled on the newly added worker node.

Step 13.       Create secret and BareMetalHost objects in openshift-machine-api namespace by executing the following manifest (bmh-worker3.yaml):

 

A new entry will be created for the newly added worker node under Compute > Bare Metal Hosts.

 

Step 14.       To increase the worker’s machineset count, go to Compute > MachineSets. Click the ellipses of worker-0 machineset and select Edit Machine Count and increase the count from 2 to 3. Click Save.

A new worker node will be provisioned to match to the worker machine count of 3. It will be under the provisioning state until the node is logically mapped to the Bare Metal Host.

Procedure 2.     Link the Machine and Bare Metal Host, Node and Bare Metal Host

Step 1.          To logically link Bare Metal Host to Machine, obtain the name of the newly created machine from its manifest file or by executing oc get machine -n openshift-machine-api:

Step 2.          Update the machine name in the Bare Metal Host’s manifest file under spec. consumerRef as shown below. Save the Yaml and reload:

After updating the machine name under Bare Metal Host yaml manifest, the newly created machine will turn to Provisioned as node state from Provisioning state.

The ProviderID is a combination of the name and UUID of the Bare Metal Host and is shown below. These details can be gathered by running providerID: ‘baremetalhost:///openshift-machine-api/<Bare Metal Host Name>/<Bare Metal Host UID>.’

By using the provided information, the providerID for of the newly added Bare Metal Host is built as providerID: ‘baremetalhost:///openshift-machine-api/worker3.fs-ocp1.flashstack.local/6410a65b-6fb1-4f34-84c2-6649e1aabba9.’

Step 3.          Copy the providerID of the Bare Metal Host into the third node yaml manifest file under spec. as shown below:

When the providerID of worker3 is updated, the node details are automatically populated for the newly added Bare Metal Host as shown below:

Install NVIDIA GPU Operator and Drivers

This chapter contains the following:

●     Install NVIDIA GPU Operator

●     Enable the GPU Monitoring Dashboard

This chapter provides the procedures to install the required NVIDIA GPU operators, CUDA drivers and so on, to work with NVIDIA GPUs.

Procedure 1.     Install NVIDIA GPU Operator

If you have GPUs installed in your Cisco UCS servers, you need to install the Node Feature Discovery (NFD) Operator to detect NVIDIA GPUs and the NVIDIA GPU Operator to make these GPUs available to containers and virtual machines.

Step 1.          In the OpenShift Container Platform web console, click Operators > OperatorHub.

Step 2.          Type Node Feature in the filter box and then click the Node Feature Discovery Operator with Red Hat in the upper right corner. Click Install.

Step 3.          Do not change any settings and click Install.

Step 4.          When the Install operator is ready for use, click View Operator.

Step 5.          In the bar to the right of Details, click NodeFeatureDiscovery.

Step 6.          Click Create NodeFeatureDiscovery.

Step 7.          Click Create.

Step 8.          When the nfd-instance has a status of Available, Upgradeable, select Compute > Nodes.

Step 9.          Select a node that has one or more GPUs and then click Details.

The label feature.node.kubernetes.io/pci-10de.present=true should be present on the host.

This label appears on all nodes with GPUs:

Step 10.       Go to Operators > OperatorHub.

Step 11.       Type NVIDIA in the filter box and then click the NVIDIA GPU Operator. Click Install.

Step 12.       Do not change any settings and click Install.

Step 13.       When the Install operator is ready for use, click View Operator.

Step 14.       In the bar to the right of Details, click ClusterPolicy.

Step 15.       Click Create ClusterPolicy.

Step 16.       Do not change any settings and scroll down and click Create. This will install the latest GPU driver.

Step 17.       Wait for the gpu-cluster-policy Status to become Ready.

Step 18.       Connect to a terminal window on the OpenShift Installer machine. Type the following commands. The output shown is for two servers that are equipped with GPUs:

Step 19.       Connect to one of the nvidia-driver-daemonset containers and view the GPU status:

Procedure 2.     Enable the GPU Monitoring Dashboard

Step 1.          Go to https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/enable-gpu-monitoring-dashboard.html, enable the GPU Monitoring Dashboard to monitor GPUs in the OpenShift Web-Console.

Verify Connectivity from Worker nodes to Pure Storage FlashBlade

Once the OpenShift cluster is up, each worker node will get IP addresses from the corresponding DHCP server scopes and assigned to the storage interfaces (eno6,eno7 and eno8) automatically for worker3 as shown below. The storage Interfaces eno6, eno7 and eno8 configured with corresponding IP addresses and MTU as 9000 as defined in the Intersight vNIC configuration.

Ensure that each worker node is able to reach the target object storage IP address configured on the FlashBlade//S200 with lager transfer unit and without fragmenting the packet.

Install and Configure Portworx Enterprise on OpenShift with Pure Storage FlashArray

This chapter contains the following:

●     Storage Architecture

●     Prerequisites

●     Configure Physical Environment

●     Portworx Enterprise Console Plugin for OpenShift

Portworx by Pure Storage is fully integrated with Red Hat OpenShift. Hence you can install and manage Portworx Enterprise from OpenShift web console itself. Portworx Enterprise can be installed with Pure Storage FlashArray as a cloud storage provider. This allows you to store your data on-premises with Pure Storage FlashArray while benefiting from Portworx Enterprise cloud drive features, such as:

●     Automatically provisioning block volumes

●     Expanding a cluster by adding new drives or expanding existing ones

●     Support for PX-Backup and Autopilot

Portworx Enterprise will create and manage the underlying storage pool volumes on the registered arrays.

Storage Architecture

This section provides the steps for installing Portworx Enterprise on OpenShift Container Platform running on Cisco UCSX-210C M7 bare metal servers. In this solution, Pure Storage FlashArray//XL170 is used as a backend storage connected over Ethernet to provide required Cloud Drives to be used by Portworx Enterprise. Figure 19 shows the high-level logical storage architecture of Portworx Enterprise deployment on Pure Storage FlashArray.

This is the high-level summary of the Portworx Enterprise implementation of distributed storage on a typical Kubernetes based Cluster:

●     Portworx Enterprise runs on each worker node as Daemonset pod and based on the configuration information provided in the StorageClass spec, Portworx Enterprise provisions one or more volumes on Pure Storage FlashArray for each worker node.

●     All these Pure Storage FlashArray volumes are pooled together to form one or more Distributed Storage Pools.

●     When a user creates a PVC, Portworx Enterprise provisions the volume from the storage pool.

●     The PVCs consume space on the storage pool, and if space begins to run low, Portworx Enterprise can add or expand drive space from Pure Storage FlashArray.

●     If a worker node goes down for less than 2 minutes, Portworx Enterprise will reattach Pure Storage FlashArray volumes when it recovers. If a node goes down for more than two minutes, a storageless node in the same zone or fault domain will take up the volumes and assume the identity of the downed storage node.

Prerequisites

These prerequisites must be met before installing the Portworx Enterprise on OpenShift with Pure Storage FlashArray:

●     SecureBoot mode option must be disabled.

●     The Pure Storage FlashArray should be time-synced with the same time service as the Kubernetes cluster.

●     Pure Storage FlashArray must be running a minimum Purity//FA version of at least 4.8. Refer to the Supported models and versions topic for more information.

●     Both multipath and iSCSI, if being used, should have their services enabled in systemd so that they start after reboots. These services are already enabled in systemd within the Red Hat CoreOS Linux.

Configure Physical Environment

Before you install Portworx Enterprise, ensure that your physical network is configured appropriately and that you meet the prerequisites. You must provide Portworx Enterprise with your Pure Storage FlashArray configuration details during installation:

●     Each Pure Storage FlashArray management IP address can be accessed by each node.

●     Your cluster contains an up-and-running Pure Storage FlashArray with an existing data plane connectivity layout (iSCSI, Fibre Channel).

●     If you're using iSCSI, the storage node iSCSI initiators are on the same VLAN as the Pure Storage FlashArray iSCSI target ports.

●     You have an API token for a user on your Pure Storage FlashArray with at least storage_admin permissions.

Procedure 1.     Prepare for the Portworx Enterprise Deployment

Step 1.          Secure the boot option already disabled at the Intersight Boot Policy level. To reconfirm at the OS level, SSH into any of the worker node from the installer VM and ensure that SecureBoot mode is disabled at the OS level.

Step 2.          Apply the following MachineConfig to the cluster configures each worker node with the following:

●     Enable and start the multipathd.service with the specified multipath.conf configuration file.

●     Enable and start the iscsid.service service.

●     Applies the Queue Settings with Udev rules.

The settings of multipath and Udev rules are defined as shown below:

The following is the MachineConfig file that takes the base64 encode results of the previous two files and copies them to the corresponding directory on each worker node. It also enables and starts iscsid and multipathd services:

Step 3.          This machine config is applied to each worker node one by one. To view the status of this process, from the cluster console go to Administration > Cluster Settings.

Step 4.          After the MachineConfig is applied on all the worker nodes, ssh into one of the worker nodes and verify:

Step 5.          From each worker node, ensure the Pure Storage FlashArray storage IPs are reachable with large packet size (8972) and without fragmenting the packets:

Step 6.          Create Kubernetes secret object constituting the Pure Storage FlashArray API endpoints and API Tokens that Portworx Enterprise needs to communicate with and manage the Pure Storage FlashArray storage device.

Step 7.          Log into Pure Storage FlashArray and go to Settings > Users and Polices. Create a dedicated user (for instance  ocp-user) with storage Admin role for Portworx Enterprise authentication.

Step 8.          Click the ellipses of the user previously created and select Create API Token. In the Create API Token wizard, set the number of weeks (for instance 24) for the API key to expire and click Create. A new API Token for the ocp-user will be created and will display. Copy the API key and preserve it since it will be used later to create Kubernetes Secret.

Step 9.          Create Kubernetes secret with the API key (located above) using the following manifest:

Procedure 2.     Deploy Portworx Enterprise

Step 1.          Log into the OpenShift cluster console using the kubeadmin account and go to Operators > OperatorHub.

Step 2.          In the right pane, enter Portworx Enterprise to filter the available operators in the Operator Hub. Select Portworx Enterprise and click Install.

Step 3.          In the Operator Installation window, from the Installed Namespace drop-down list, select Create Project and create a new project (for instance px) and select the newly created project to install the Portworx Operator.

Step 4.          Install the Portworx plugin for OpenShift by clicking Enable under the Console Plugin. Click Install.

Step 5.          When the Portworx operator is successfully installed, the StorageCluster needs to be created. To create the StorageCluster Specifications (manifest file) log into https://central.portworx.com/ and use your credentials and click Get Started.

Step 6.          Select Portworx Enterprise and click Continue.

Step 7.          From the Generate Spec page, select the latest Portworx version (version 3.1 was the latest when this solution was validated). Select Pure Storage FlashArray as the platform. Select OpenShift 4+ for the Distribution drop-down list, provide px for the Namespace field. Click Customize.

Step 8.          Get the Kubernetes version by running kubectl version | grep -i 'Server Version'. Click Next.

Step 9.          From the Storage tab, select iSCSI for Storage Area Network. Provide the size of the Cloud drive and click plus (+) to add additional disks. Click Next.

Step 10.       From the Network tab, set Auto for both Data and Management Network Interfaces. Click Next.

Step 11.       From the Customize tab, click Auto for both Data and Management Network Interfaces. Click Next.

Step 12.       Click Advanced Settings, enter the name of the portworx cluster (for instance, ocp-pxclus). Click Finish. Click Download.yaml to download the StorageCluster specification file.

Step 13.       From the OpenShift console, go to Operators > Installed Operators > Portworx Enterprise. Click the StorageCluster tab and click Create Storage Cluster to create StorageCluster. This opens the YAML view of Storage Cluster.

Step 14.       Copy the contents of the spec file previously downloaded and paste it in the yaml body. Verify that both the iSCSI subnet networks are listed under env: as shown below. Click Create to create the StorageCluster.

Step 15.       Wait until all Portworx related pods are online.

Step 16.       Verify the cluster status by running the command on any worker node: sudo  /opt/pwx/bin/pxctl status.

Step 17.       Run the sudo multipath -ll command on one of the worker nodes to verify all four paths from the worker node to storage target are being used. As shown below, there are four active running paths for each volume:

Now the Portworx StorageCluster is ready to use.

Procedure 3.     Dynamic Volume Provisioning and Data Protection

Portworx by Pure Storage allows platform administrators to create custom Kubernetes StorageClasses to offer different classes of service to their developers. Administrators can customize things like the replication factors, snapshot schedules, file system types, io-profiles, etc. in their storage class definitions.

This procedure details how to create customized Storage Classes (SCs) for dynamic provisioning of volumes (PVs) using PersistentVolumeClaims (PVCs).

For instance, the following manifest file shows a sample file Storage Class for provisioning  shared volume with ReadWriteMany (RWX) attribute to share the volume among multiple pods at the same time for read-write access. For provisioning volumes for typical application pods, predefined SCs can be leveraged.

Step 1.          Use the following manifests to create a sharedv4 service volumes to be consumed by multiple WordPress application accessing the sharev4 service volume (consumed by multiple WordPress pods) and ReadWriteOnce volume (Consumed by one MySQL pod):

The following screenshot shows that 3 WordPress pods are created accessing the same sharedv4 volume with ReadWriteMany access mode and one mysql pod created with single  volume with ReadWriteOnce access mode:

Sharedv4 Volume with ReadWriteMany (RWX) Access: Three WordPress pods are accessing the same sharedv4 volume with the ReadWriteMany access mode, meaning multiple pods can concurrently read from and write to this single volume. This is especially useful for applications like WordPress that may need to scale out to handle load but still rely on a shared data volume.

ReadWriteOnce (RWO) Access for MySQL: There is also a single MySQL pod accessing a volume with the ReadWriteOnce access mode, which allows only one pod to mount the volume at a time. This setup is common for databases, where concurrent access could lead to data inconsistencies.

This setup highlights Portworx's ability to support various access modes for different use cases, allowing flexible storage configurations that suit both shared applications (like WordPress) and single-instance databases (like MySQL). By providing the ReadWriteMany access mode with sharedv4, Portworx enables efficient scaling and resource usage, allowing applications to use shared storage across multiple pods and hosts

Snapshots and Clones: Portworx Enterprise offers data protection of volumes using volume snapshots and restore them for point in time recovery of the data. Any Storage Class that implements portworx csi driver pxd.portworx.com supports volume Snapshots.

Step 2.          Run the following scripts to create the VolumeSnapshotClass, a PVC and then a Snapshot of a PVC and then restore the snapshot as a new PVC.

Step 3.          Use the following sample manifest to create a sample PVC:

The following screenshot shows px-snaptest-pvc-snap1 is the snapshot of the PVC px-snaptest-pvc:

Step 4.          You can now restore this as snapshot as a new PVC and then can be mounted to any other pod.

Step 5.          Click the ellipses of the snapshot and select Restore as new PVC. In the Restore as new PVC window, click Restore.

Step 6.          You can view the original and restored PVC under PVC list.

Step 7.          To clone of a PVC (px-snaptest-pvc), click three of the PVC and select Clone PVC. Click Clone.

Portworx Enterprise Console Plugin for OpenShift

Portworx by Pure Storage has built an OpenShift Dynamic console plugin that enables single-pane-of-glass management of storage resources running on Red Hat OpenShift clusters. This allows platform administrators to use the OpenShift web console to manage not just their applications and their OpenShift cluster, but also their Portworx Enterprise installation and their stateful applications running on OpenShift.

This plugin can be installed with a single click during the Installation of Portworx Operator as explained in the previous sections. Once the plugin is enabled, the Portworx Operator will automatically install the plugin pods in the same OpenShift project as the Portworx storage cluster. When the pods are up and running, administrators will see a message in the OpenShift web console to refresh their browser window for the Portworx tabs to show up in the UI.

With this plugin, Portworx has built three different UI pages, including a Portworx Cluster Dashboard that shows up in the left navigation menu, a Portworx tab under Storage > Storage Class section, and another Portworx tab under Storage > Persistent Volume Claims.

Portworx Cluster Dashboard

Platform administrators can use the Portworx Cluster Dashboard to monitor the status of their Portworx Storage Cluster and their persistent volumes and storage nodes. Here are a few operations that are now streamlined by the OpenShift Dynamic plugin from Portworx.

To obtain detailed inventory information of the Portworx Cluster, click the Drives and Pools tabs.

Portworx PVC Dashboard

This dashboard shows some of the important attributes of a PVC like Replication Factor, node details of replicas, attached node and so on. You may have to use multiple pxctl inspect volume CLI commands to get these details. Instead, all this information can be found in Console Plugin.

Portworx StorageClass Dashboard

From the Portworx storage cluster tab, administrators can get details about the custom parameters set for each storage class, the number of persistent volumes dynamically provisioned using the storage class, and a table that lists all the persistent volumes deployed using that storage class. The OpenShift dynamic plugin eliminates the need for administrators to use multiple “kubectl get” and “kubectl describe” commands to find all these details—instead, they can just use a simple UI to monitor their storage classes.

RAG Pipeline Installation and Configuration

This chapter contains the following:

●     NVIDIA NIM Operator and NIMs

●     Vector Database

●     RAG Blueprint

●     Access Services and Application

The reference RAG Blueprint leverages the NIMs deployed for LLMs, Text Embedding, and Text Reranking. It uses the Milvus vector database to store the vector embeddings.

NVIDIA NIM Operator and NIMs

NIMs can be installed independently using its respective helm charts from https://catalog.ngc.nvidia.com/helm-charts. It can also be installed and managed using NVIDIA NIM Operator. In this design, NIMs will be installed and managed using NVIDIA NIM Operator.

Procedure 1.     Install NIM Operator

Step 1.          In the Red Hat OpenShift console, go to Operators > OperatorHub.

Step 2.          Search for NVIDIA NIM Operator.

Step 3.          Click Install.

Step 4.          Retain Installed Namespace as openshift-operators. Click Install.

Create Caching Models

NIM Operator uses a custom resource called NIM cache (nimcaches.apps.nvidia.com). This custom resource enables downloading models from NVIDIA NGC and persisting them on Portworx persistent storage.

One advantage to caching a model is that when multiple instances of the same NIM microservice start, the microservices use the single cached model. However, caching is optional. Without caching, each NIM microservice instance downloads a copy of the model when it starts.

It is recommended to cache the models for NVIDIA NIM microservices. By caching a model, the microservice startup time is improved. For deployments that scale to more than one NIM microservice pod, a single cached model in persistent volume with a storage class from Portworx provisioner can serve multiple pods.

This section provides the procedures to create caching models for LLMs, Text Embedding, and Text Reranking.

Procedure 1.     Create namespace

Step 1.          To create the namespace, run the following command:

Procedure 2.     Secrets for NGC API key

Step 1.          NIM cache uses the NGC API key as an image pull secret to download container images and models from NVIDIA NGC. Add a Docker registry secret for downloading the NIM container image from NVIDIA NGC:

Step 2.          Add a generic secret that the model puller init container uses to download the model from NVIDIA NGC:

NIM cache for LLM

Refer to the following sample manifest as an example:

When you create a NIM cache resource, the NIM Operator starts a pod that lists the available model profiles. The Operator creates a config map of the model profiles.

If you specify one or more model profile IDs to cache, the Operator starts a job that caches the model profiles that you specified.

If you did not specify model profile IDs but do specify engine: tensorrt_llm or engine: tensorrt, the Operator attempts to match the model profiles with the GPUs on the nodes in the cluster. The Operator uses the value of the nvidia.com/gpu.product node label that is set by Node Feature Discovery.

You can let the Operator automatically detect the model profiles to cache or you can constrain the model profiles by specifying values for spec.source.ngc.model, such as the engine, GPU model, and so on, that must match the model profile.

More information about the option is available here: https://docs.nvidia.com/nim-operator/latest/cache.html

NIM Cache for Text Embedding

Refer to the following example sample manifest:

NIM Cache for Text Reranking

Refer to the following example sample manifest:

Procedure 1.     Create Resources

Step 1.          Apply the YAML manifests to all three models:

Step 2.          View the NIM cache resource to view the status:

Create Services

NIM Services (nimservices.apps.nvidia.com) custom resource represents a NIM microservice. Adding and updating a NIM service resource creates a Kubernetes deployment for the microservice in a namespace. The custom resource supports using a model from an existing NIM cache resource or a persistent volume claim.

NIM Service for LLM

Refer to the following example sample manifest:

NIM Service for Text Embedding

Refer to the following example sample manifest:

NIM Service for Text Reranking

Refer to the following example sample manifest:

View the NIM services custom resources:

Create NIM Pipelines

As an alternative to managing NIM services individually using multiple NIMService custom resources, you can manage multiple NIM services using one NIMPipeline custom resource.

The following sample manifest deploys all NIMs mentioned above without creating individual NIM services:

More information about NIM Pipelines is available here: https://docs.nvidia.com/nim-operator/latest/pipelines.html

Procedure 1.     Verification

After all the NIMs are deployed, you can verify the status of the services by calling “/v1/health/ready” API endpoint. First, get the services exposed to access the LLM, Embedding and Reranking models.

Step 1.          Note that the service names will be the same as name given while creating the NIM Service. In the previous example, NIM service for Llama-3.1 is created using name llm, Llama-3.2 Text Embedding NIM with embedqa and Llama 3.2 1B Reranking with the name rerankqa:

Step 2.          From the previous output, Llama-3.1 is available using NodePort 30604, embedding model at 30765 and rerank model at 32630. All services with ClusterIP Port 8000.

Step 3.          Get the IP or DNS name of the node ports by running the following command:

Step 4.          Verify the status of each service running using “/v1/health/ready” endpoint:

Vector Database

This section provides the procedures to deploy the Milvus vector database:

●     Before deploying Milvus vector database deployment, ensure the following pre-requisites are configured on the FlashBlade//S.

●     For the Object storage traffic between the FlashBlade//S and workers nodes, create required Link Aggregation Groups (LAGs), Subnets and Interface on the FlashBlade//S and assign IP address to the Interface.

●     On each worker node, ensure an interface is configured with IP addresses in the same subnet as that of FlashBlade//S interface and ensure IPs are reachable from each other.

●     In the FlashBlade//S array, ensure that an account with at least one User ID is created and configured with required access policy and Access key. Also ensure, at least one Bucket is created and mapped to the account created in the above step.

In this architecture, FlashBlade//S is used as S3 compatible backend object storage as required by Milvus vector database for storing Milvus collections (Vector embeddings) and Indexes. While the FlashArray//XL170 is used for storing other Milvus components like Pulsar’s bookie journal, Zookeeper and so on.

Procedure 1.     Configure Portworx storage class

Step 1.          Run the following script to configure one of portworx storage classes configured with FlashArray //XL170 and back-end storage as ‘Default storage class’:

Step 2.          Ensure the routingViaHost is set to true by running the below script. This will ensure the Milvus Pod to directly access the FlashBlade//S interfaces via worker node network interfaces for object storage traffic:

Step 3.          Run the following script on oc terminal to update the Security context (SCC) to allow anyuid for the service accounts created by the milvus helm chart:

Before proceeding with the Milvus deployment, gather the following details listed in Table 21 and keep them available.

Table 21.     Storage Details required for Vector Database deployment

Procedure 2.     Milvus Deployment with Pure Storage FlashBlade//S

Step 1.          Create a new namespace for milvus:

Step 2.          Add the milvus repository:

Step 3.          Update the helm repository:

Step 4.          Create a file named custom_value.yaml for customizing the Milvus deployment. Specify Attu details, milvus version that supports GPU based indexes, number of replicas, GPU resources and FlashBlade details for object storage as shown in sample manifest file:

Step 5.          Install the helm chart and point to the previously created file using -f argument as shown below:

Step 6.          Check the status of the pods:

All pods should be running and in a ready state within couple of minutes:

The milvus version used in the solution is 2.5.3:

When GPU-optimized Milvus is deployed, the consumption of GPUs by Milvus components such as query node or Index node can be verified using nvidia-smi command as shown below. In this screenshot, two GPU are being consumed by the distributed Milvus deployment as query node and index node pods are scheduled on the same worker node.

RAG Blueprint

This blueprint serves as a reference solution for a foundational Retrieval Augmented Generation (RAG) pipeline. Foundational RAG Pipeline blueprint is available in NVIDIA AI Blueprints GitHub.

Procedure 1.     Deploy the RAG pipeline using the RAG blueprint version 1.0.1.

Step 1.          Fork the NVIDIA-AI-Blueprints/rag repository on GitHub - https://github.com/NVIDIA-AI-Blueprints.

Step 2.          Clone the forked copy of this repository:

Step 3.          Navigate to rag/deploy/helm/charts.

Step 4.          Navigate to rag/deploy/templates directory. Open the file rag-server-sts.yaml:

Step 5.          In the previous example, embedqa is the service for Embedding model, rerankqa is the service for re-ranking model in nim namespace and milvus-deployment is the service for Milvus in milvus namespace. To update, run the following:

Step 6.          The RAG Blueprint application can be kept in a separate namespace. You are creating a namespace called rag and deploy the RAG Blueprint in rag namespace:

Step 7.          The RAG Blueprint makes use of default service account. This default service account in the rag namespace should have the ability to run pods with any user ID, overriding the default restrictions. You can also configure a custom service account if required:

Step 8.          Blueprint uses the NGC API key as an image pull secret to download container images and models from NVIDIA NGC. Add a Docker registry secret for downloading the NIM container image from NVIDIA NGC:

Step 9.          Add a generic secret that the model puller init container uses to download the model from NVIDIA NGC:

Step 10.       Open the file rag/deploy/helm/values.yaml.

Step 11.       Update the file with server URL and Model name for LLM, Embedding and Reranking models in the env section.

Step 12.       Get the model’s name and other information of the models deployed using v1/models API endpoint:

Sample environment values:

Step 13.       Update the imagePullSecret:

Step 14.       Set the concurrent users:

Step 15.       Update the LLM model in the ragPlayground:

Procedure 2.     Install RAG Blueprint

Step 1.          Navigate to rag/deploy/helm directory and do helm install:

Step 2.          Make sure pods and services are up for rag-server and rag-playground:

The RAG Server is running on NodePort 32068, and RAG Playground is running on 32665.

Access Services and Application

NIMs can be installed independently using its respective helm charts from https://catalog.ngc.nvidia.com/helm-charts. It can also be installed and managed using NVIDIA NIM Operator. In this design, NIMs will be installed and managed using NVIDIA NIM Operator.

Procedure 1.     Access RAG Playground

Step 1.          RGA Playground can be accessed using any of the node’s IP or DNS name and port:

Step 2.          You can access the application running on port 8090 of rag-playground service in rag namespace on localhost port 30001 using the following command:

Step 3.          Open browser and access the rag-playground UI.

Step 4.          Submit a question and make sure you get response from LLM.

Step 5.          Click Knowledge Base and upload a document to Knowledge Base and make sure it’s updated.

Step 6.          Click Converse. Ask a question. Click Use Knowledge Base so that LLM refers to the uploaded document to provide an accurate answer.

Procedure 2.     Milvus Vector Database

Step 1.          Get the compute node IP/DNS for one of the nodes and get the node port for Attu:

Step 2.          Access Attu, the UI interface for Milvus vector database using Node IP and NodePort and click on Connect.

Step 3.          Click the default database and select the collection and navigate to Data and observe that embeddings are stored in Milvus.

Procedure 3.     RAG Server

Step 1.          Get the compute node IP/DNS for one of the nodes and get the node port for rag-server:

Step 2.          View all the APIs by accessing <Node_IP/DNS>:NodePort/docs endpoint.

NIM Services

Like the RAG Server, API documentation can be accessed for any NIMs with /docs endpoint.

APIs available for NIM for LLMs is shown below:

APIs available for NIM for NeMo Retriever Text Embedding NIM is shown below:

Any API can be called referring to the API documentation. The example below shows the API to get embeddings for the test.

Request URL:

Curl Request:

Benchmarking, Evaluation, and Sizing

This chapter contains the following:

●     NIM for LLM Benchmarking

●     Sizing

●     RAG Evaluation

●     Milvus Benchmarking with VectorDBBench

This chapter examines the performance benchmarking of NIM for LLM, RAG Evaluation using RAGAS framework, Sizing guidance for GPU and model selection and vector database benchmarking.

NIM for LLM Benchmarking

Benchmarking the deployment of Large Language Models (LLMs) involves understanding key metrics such as inference latency and throughput, which are crucial for evaluating performance. Developers and enterprise system owners can utilize various benchmarking tools, each with distinct features and capabilities, to measure these metrics effectively.

GenAI-Perf is a tool for measuring and analyzing the performance indicators, ensuring their LLM deployments are optimized for efficiency and scalability.

How LLM Inference Works

Before examining benchmark metrics, it is crucial to understand the mechanics of LLM inference and the associated terminologies. An LLM application generates results through a multi-stage inference process. Initially, the user submits a query (prompt), which enters a queue, awaiting its turn for processing—this is the queuing phase. Next, the LLM model processes the prompt during the prefill phase. Finally, the model generates a response token by token in the generation phase.

A token, a fundamental concept in LLMs, serves as a core performance metric for LLM inference. It represents the smallest unit of natural language that the LLM processes. The aggregation of all tokens forms the vocabulary, with each LLM employing a tokenizer learned from data to efficiently represent input text. Typically, for many popular LLMs, one token approximates 0.75 English words.

Sequence length pertains to the length of the data sequence. The Input Sequence Length (ISL) denotes the number of tokens the LLM receives, encompassing the user query, system prompts (e.g., model instructions), previous chat history, chain-of-thought reasoning, and documents from the retrieval-augmented generation (RAG) pipeline. The Output Sequence Length (OSL) indicates the number of tokens the LLM generates. Context length refers to the number of tokens the LLM utilizes at each generation step, including both input and output tokens generated thus far. Each LLM has a maximum context length that can be allocated to both input and output tokens.

Streaming is a feature that allows partial LLM outputs to be returned to users incrementally, in chunks of tokens generated so far. This is particularly advantageous for chatbot applications, where receiving an initial response quickly is desirable. While the user processes the partial content, the subsequent chunk of the result is generated in the background. Conversely, in non-streaming mode, the complete answer is returned in a single response.

Metrics

There can be variations in the benchmarking results between different tools. Figure 20 illustrates some of the widely used LLM inference metrics.

Time to First Token

Time to First Token (TTFT) measures the duration from submitting a query to receiving the first token of the model’s output. This metric encompasses request queuing time, prefill time, and network latency. Generally, a longer prompt results in a higher TTFT due to the attention mechanism, which requires the entire input sequence to compute and create the key-value cache (KV-cache) before the iterative generation loop can commence. In production environments, multiple requests may be processed simultaneously, causing the prefill phase of one request to overlap with the generation phase of another, further impacting TTFT.

Generation Time

Generation time is the duration from the first token received to the final token received. GenAI-Perf removes the last [done] signal or empty response, so they don’t get included in the e2e latency.

End-to-End Request Latency (e2e_latency)

End-to-End Request Latency (e2e_latency) quantifies the total time from submitting a query to receiving the complete response. This metric encompasses the performance of queuing/batching mechanisms and network latencies as demonstrated in Figure 22.

Mathematically, it is expressed as: e2e_latency=TTFT + Generation Time

For an individual request, the end-to-end request latency is the time difference between when the request is sent and when the final token is received. This metric provides a comprehensive measure of the system’s responsiveness, accounting for all stages of the request processing pipeline.

Inter Token Latency (ITL)

This is defined as the average time between consecutive tokens and is also known as time per output token (TPOT).

GenAI-Perf defines ITL as follows:

The equation for this metric excludes the first token (hence subtracting 1 in the denominator) to ensure that Inter-Token Latency (ITL) reflects only the decoding phase of request processing.

Longer output sequences increase the size of the KV cache, thereby raising memory costs. Additionally, the cost of attention computation grows linearly with the length of the combined input and output sequence for each new token, although this computation is typically not compute-bound. Consistent inter-token latencies indicate efficient memory management, optimal memory bandwidth utilization, and effective attention computation.

Tokens Per Second (TPS)

Total TPS per system represents the total output tokens per seconds throughput, accounting for all the requests happening simultaneously. As the number of requests increases, the total TPS per system increases, until it reaches a saturation point for all the available GPU compute resources, beyond which it might decrease.

Given the following timeline of the entire benchmark with n total requests.

Li : End-to-end latency of i-th request

T_start : start of benchmark

Tx : timestamp of the first request

Ty : timestamp of the last response of the last request

T_end : end of benchmark

GenAI-perf defines the TPS as total output tokens divided by the end-to-end latency between the first request and the last response of the last request.

Requests Per Second (RPS)

This is the average number of requests that can be successfully completed by the system in a 1-second period. It is calculated as:

Use Cases

An application’s specific use cases influence sequence lengths - Input Sequence Length(ISL) and Output Sequence Length(OSL), which in turn affect how efficiently a system processes input to form the KV-cache and generate output tokens. Longer ISL increases memory requirements for the prefill stage, thereby extending TTFT, while longer OSL increases memory requirements (both bandwidth and capacity) for the generation stage, thus raising ITL. Understanding the distribution of inputs and outputs in an LLM deployment is crucial for optimizing hardware utilization. Common use cases and their typical ISL/OSL pairs include:

●     Translation

Involves translating between languages and code, characterized by similar ISL and OSL, typically ranging from 200 to 2000 tokens each.

●     Generation

Encompasses generating code, stories, emails, and generic content via search, characterized by an OSL of O(1000) tokens, significantly longer than an ISL of O(100) tokens.

●     Classification

Involves categorizing input data into predefined classes or categories. Usually characterized by an ISL of O(200) tokens and OSL of O(5) tokens.

●     Summarization

Includes retrieval, chain-of-thought prompting, and multi-turn conversations, characterized by an ISL of O(1000) tokens, much longer than an OSL of O(200) tokens.

Real data can also be used as inputs. GenAI-Perf supports datasets such as HuggingFace OpenOrca and CNN Dailymail.

Load Control

●     Concurrency (N)

Refers to the number of concurrent users, each with one active request, or equivalently, the number of requests being served simultaneously by an LLM service. As soon as a user’s request receives a complete response, another request is sent, ensuring the system always has exactly N active requests.

LLMperf sends requests in batches of N but includes a draining period where it waits for all requests to complete before sending the next batch. Consequently, the number of concurrent requests gradually reduces to zero towards the end of the batch. This differs from GenAI-perf, which maintains N active requests throughout the benchmarking period.

Concurrency is primarily used to describe and control the load on the inference system.

●     Max Batch Size

A batch is a group of simultaneous requests processed by the inference engine, which may be a subset of the concurrent requests. The maximum batch size parameter defines the maximum number of requests the inference engine can process simultaneously.

If concurrency exceeds the maximum batch size multiplied by the number of active replicas, some requests will queue for later processing, potentially increasing the Time-to-First-Token (TTFT) due to the queuing effect.

●     Request Rate

This parameter controls load by determining the rate at which new requests are sent. A constant (static) request rate ( r ) means one request is sent every ( \frac{1}{r} ) seconds, while a Poisson (exponential) request rate determines the average inter-arrival time.

GenAI-perf supports both concurrency and request rate, but concurrency is recommended. With request rate, the number of outstanding requests may grow unbounded if the requests per second exceed system throughput.

When specifying concurrencies to test, it is useful to sweep over a range of values, from a minimum of 1 to a maximum not much greater than the max batch size. When concurrency exceeds the max batch size of the engine, some requests will queue, causing system throughput to saturate around the max batch size while latency steadily increases.

Procedure 1.     Set up GenAI-Perf

Step 1.          Make sure that the NIM for LLM with llama-3.1-8b-instruct service is running with NIM Service:

Step 2.          Observe that the POD is running. The cluster IP of the service is 172.30.73.126 and it’s running on Port 8000. It is also exposed as NodePort on Port 30604:

Step 3.          Make a test chat completion to make sure that the LLM endpoint is working:

A sample deployment YAML manifest is provided below:

Step 4.          The image pull secret is required with a NGC API key to download the image. It can be created using:

Step 5.          A repository is required to store the NeMo check points and other temp folders. A sample PVC configuration to provide persistent storage to the pod is provided below:

Step 6.          Apply the YAML manifests against the OpenShift cluster and make sure all the resources are created

Step 7.          From the output, you can see that the pod is running. Run the following commands to get access to the pod:

Step 8.          Once inside the pod, you can start the GenAI-perf evaluation harness as follows, which runs a warming load test on the NIM backend:

 

This example specifies the input and output sequence length and a concurrency to test.

Step 9.          This test will use the llama-3 tokenizer from HuggingFace, which is a guarded repository. You will need to apply for access, then login with your HF credential:

Step 10.       With a successful execution, you will see similar results in the terminal as shown below:

When the tests complete, GenAI-perf generates the structured outputs in a default directory named “artifacts” under your mounted working directory (/workdir in these examples), organized by model name, concurrency, and input/output length. Your results should look similar to the following:

Procedure 2.     Sweep through Use Cases

Typically, with benchmarking, a test would be set up to sweep over a number of use cases, such as input/output length combinations, and load scenarios, such as different concurrency values.

Step 1.          Use the following bash script to define the parameters so that GenAI-perf executes through all the combinations:

Step 2.          Save this script in a working directory, such as under /workdir/benchmark.sh. You can then execute it with the following command:

The following tables lists the details of performance for three different use cases.

Table 22.     NIM for LLM Benchmarking for Classification Use Case (I/O = 200/5)

Table 23.     NIM for LLM Benchmarking for Summarization Use Case (I/O = 1000/200)

Table 24.     NIM for LLM Benchmarking for Translation Use Case (I/O = 200/200)

Table 25.     NIM for LLM Benchmarking for Generation Use Case (I/O = 100/1000)

Analyze the Output

The figure below shows the total output tokens per seconds throughput, accounting for all the requests happening simultaneously from all 4 uses cases.

The figure below shows how long it takes from submitting a query to receiving the full response for 4 different use cases.

The figure below shows the total output tokens per seconds throughput, accounting for all the requests happening simultaneously with 1 L40S GPU vs 2 X L40S GPU.

The figure below shows the average number of requests that can be successfully completed by the system in a 1-second period with 1 L40S GPU vs 2 X L40S GPU.

The figure below shows how long a user needs to wait before seeing the model’s output with 1 L40S GPU vs 2 X L40S GPU.

The figure below shows the average time between consecutive tokens with 1 L40S GPU vs 2 X L40S GPU.

Interpreting the Results

The following plots illustrate the Latency-Throughput curves for four different use cases with different Input Sequence Length and Output Sequence Lengths. 

Summarization (I/O : 1000/200), Classification (I/O : 200/5), Translation (I/O : 200/200) and Generation (I/O : 100/1000) .

●     Axes Definitions

◦     X-axis: Time to First Token (TTFT) - The duration a user waits before the model starts generating output.

◦     Y-axis: Total System Throughput - The total number of output tokens generated per second.

◦     Dots: Represent different concurrency levels.

Usage Guidelines

●     Latency Budget Approach

Objective: Determine the highest achievable throughput within a specified latency limit.

Method: Identify the maximum acceptable TTFT on the X-axis, then find the corresponding Y value and concurrency level. This indicates the highest throughput achievable within the given latency constraint.

●     Concurrency-Based Approach

Objective: Understand the latency and throughput for a specific concurrency level.

Method: Locate the desired concurrency level on the plot. The intersecting X and Y values represent the latency and throughput for that concurrency.

●     Key Observations

The Latency-Throughput Curve for Classification plot highlights concurrency levels where latency increases significantly with minimal or no throughput gain. Concurrency = 50 is an example of such a point.

●     Alternative Metrics

Similar plots can be generated using different metrics on the X-axis, such as ITL (Inference Time Latency), e2e_latency (End-to-End Latency), or TPS_per_user (Transactions Per Second per User). These plots help visualize the trade-offs between total system throughput and individual user latency.

Sizing

Memory Calculations for LLM Inferencing

Total Memory required is the sum of model memory size and the KV cache.

Calculations for the required total memory are provided below:

Model Memory Size = Model Parameters * Precision

KV Cache Size = 2 x Batch Size x Context Size  x Number of Layers x Model Dimensions x Precision

Total Memory Requirements (GB) = Model Memory Size (GB) + KV Cache Size (GB)

For some models, model dimension data might not be available. In that case, model dimension can be calculated as:

Model Dimensions = Attention Head Size X Number of Attention Heads

Model Parameters, Precision, Number of layers, Model Dimension are specific to models, and it can be found in the Model card for the model.

Context Size and batch size are input from users.

The following is an example memory calculation for Llama 2:

For the Llama 2 model:

Total model parameters:         6.74B Parameters.

Precision:                                 FP16. (2 Bytes)

Number of layers:               32

Model Dimension:                4096

Therefore, the model memory is calculated as shown below:

Model Memory Size = Model Parameters * Precision

 

Model Memory Size for Llama 2   = 6,740,000,000 * 2 Bytes/Parameter

                                 = 13,480,000,000 Bytes

                                  = 13.48 Giga Bytes                                 

Also, considering an example of maximum Input Tokens Length of 1024, Maximum Output Tokens Length of 1024,  and the Batch size of 8, below are the calculations for KV Cache Size:

KV Cache Size         = 2 x Batch Size x Context Size  x Number of Layers x Model Dimensions x Precision

KV Cache Size         = 2 x 8 x (1024+1024) x 32 x 4096 x 2 Bytes/Parameter

= 8,589,934,592 Bytes

                      =  8.59 Giga Bytes

Therefore, Llama2 with maximum Input Tokens Length of 1024, Maximum Output Tokens Length of 1024, and the Batch size of 8, the total memory required is shown below:

Total Memory Requirements (GB) = Model Memory Size (GB) + KV Cache Size (GB)

                                 = 13.48 + 8.59 Giga Bytes

                                = 22.07 Giga Bytes

Performance Calculations

Based on the performance requirement, number of users, number of input and output tokens, latency and throughput required, you can choose the appropriate Large Language Model, Inferencing backend, GPUs, and compute infrastructure.

Performance of the model depends on the prefill and decode phases. These two phases have different impacts on the performance of the LLM. While the prefill phase effectively saturates GPU compute at small batch sizes, the decode phase results in low compute utilization as it generates one token at a time per request.

The prefill phase is compute-bound, while the decode phase is memory-bound. So, the following factors need to be considered and measured:

●     Prefill Latency

●     Prefill Throughput

●     Decode Total Latency

●     Decode Token Latency

●     Decode Throughput

The performance benchmark can be run with different sizes (1,2,4,8,10,25,250, 100 and so on). Also, separate tests can be run focused on performance comparison between 2 different models.

RAG Evaluation

How RAG Works

Retrieval-Augmented Generation (RAG) is a technique used to enrich LLM outputs by using additional relevant information from an external knowledge base. This allows an LLM to generate responses based on context beyond the scope of its training data.

Why Evaluate RAG?

RAG enhances content generation by leveraging existing information effectively. It can amalgamate specific, relevant details from multiple sources to generate more accurate and relevant query results. This makes RAG potentially invaluable in various domains, including content creation, question & answer applications, and information synthesis. RAG does this by combining the strengths of retrieval, usually using dense vector search, and text generation, but to see what is and isn't working in your RAG system, to refine and optimize, you must evaluate it.

Evaluation Process

Evaluating RAG goes through the following steps:

1.     Choose Right Evaluation Framework.

2.     Generate Synthetic Dataset.

3.     Generate RAG Outputs for the Questions from Synthetic Dataset.

4.     Evaluating Using the Generated Dataset.

5.     Understanding The Metrics.

Choose the Right Evaluation Framework

RAG evaluation quantifies the accuracy of retrieval phrase by calculating metrics on the top results your system returns, enabling you to programmatically monitor pipeline’s precision, recall ability, and faithfulness to facts.

It is not only important to have good metrics, but that the ability to measure things separately.

To see where things are going well, can be improved, and where errors may originate, it's important to evaluate each component in isolation. The Figure 28 classifies the RAG’s components – along with what needs evaluation in each:

The Evaluation Framework is meant to ensure granular and thorough measurement, addressing the challenges faced in all three components.

To meet the evaluation challenges systematically, it’s a best practice to breakdown evaluation into different levels.

Embedding Model Evaluation

The Massive Text Embedding Benchmark (MTEB) leverages different public/private datasets to evaluate and report on the different capabilities of individual models. You can use the MTEB to evaluate any model in its list

The Model used in this deployment type is “snowflake-arctic-embed-I” and below is the model’s performance listed on leaderboard:

Data Ingestion Evaluation

After evaluating model’s performance using benchmarks, and (optionally) fine-tune it, then configure data ingestion into semantic retrieval store (vector store).

To evaluate data ingestion, observe and measure how changes in related variables affect ingestion outcomes:

●     Chunk Size: The size of each data segment, which depends on the token limit of the embedding model. Chunk size substantially determines data granularity and the contextual understanding of the data, which impacts the precision, recall, and relevancy of results.

●     Chunk Overlap: The extent to which data points of events are shared by adjacent data chunks. Overlap helps with retention of context information contained in chunks but should be paired with strategies like deduplication and content normalization to eradicate adverse effects (redundancy or inconsistency).

●     Chunking/ Text Splitting Strategy: The process of data splitting and further treatment, based on both data type (for example, html, markdown, code, or pdf, and so on) and nuances of your use case.

Semantic Retrieval Evaluation

In this deployment, the Milvus Vector Database has been deployed, and milvus offers similarity metrics.

Similarity metrics are used to measure similarities among vectors. Choosing a good distance metric helps improve the classification and clustering performance significantly.

Euclidean distance (L2)—essentially, Euclidean distance measures the length of a segment that connects 2 points.

The formula for Euclidean distance is as follows:

where a = (a1, a2,…, an) and b = (b1, b2,…, bn) are two points in n-dimensional Euclidean space

It’s the most used distance metric and is very useful when the data are continuous.

Inner Product (IP)—The IP distance between two embeddings is defined as follows:

Where A and B are embeddings, ||A|| and ||B|| are the norms of A and B.

IP is more useful if you are more interested in measuring the orientation but not the magnitude of the vectors.

If you use IP to calculate embeddings similarities, you must normalize your embeddings. After normalization, the inner product equals cosine similarity.

Suppose X’ is normalized from embedding X:

The correlation between the two embeddings is as follows:

End-to-End Evaluation

An end-to-end evaluation of a RAG application assesses the final outputs generated by LLMs in response to given inputs. It requires addressing issues discussed above related to data heterogeneity, domain specificity, and user query and preference diversity. It's impossible to devise a fixed metric or methodology that fits all domains and use cases.

E2E evaluation frameworks range from proprietary solutions to open-source tools. Selecting the right solution requires balancing considerations around ease of maintenance and operational burden, plus how well the metrics observed by the tool map to your Retrieval Augmented Generation pipeline’s use case.

Table 26.     Recommended RAG Frameworks based on Use Case

Since a satisfactory LLM output depends entirely on the quality of the retriever and generator, RAG evaluation focuses on evaluating the retriever and generator in RAG pipeline separately. This allows for easier debugging and to pinpoint issues on a component level.

This document provides the steps necessary to run RAGAS Evaluation Framework. RAGAS is a framework that helps evaluate Retrieval Augmented Generation (RAG) Initial pipelines.

Generate Synthetic Dataset

To set up RAGAS evaluation Framework, apply ground truth concept. Some ground truth data - a golden set - provides a meaningful context for the metrics to evaluate our RAG pipeline's generated responses.

The first step in setting up RAGAS is creating an evaluation dataset, complete with questions, answers, and ground-truth data, which takes account of relevant context.

Procedure 1.     Create an evaluation dataset

Step 1.          Create LLM Prompt template:

Step 2.          Create the System template:

Step 3.          Generate Synthetic Data:

Step 4.          Call this function with path to the document and file name for q&a:

This creates a JSON file called “flashstack_qa_generation.json”

Procedure 2.     Generate RAG Outputs for the Questions from Synthetic Dataset

Step 1.          Load the JSON file and iterate through each question, retrieve context from vector store and ask LLM to generate the response:

Step 2.          Once the new list of objects is created, store them on the file system as shown below:

This creates a JSON file called “flashstack_eval.json”

Procedure 3.     Evaluate using the Generated Dataset

Step 1.          Create the Prompt Template:               

Step 2.          Create System Prompt:

The performance of individual components within the LLM and RAG pipeline has a significant impact on the overall experience. Ragas offers metrics tailored for evaluating each component of RAG pipeline in isolation.

Step 3.          Create a Function to choose the metric types: 

Step 4.          Create a Function to run RAGAS Evaluation from the evaluation json dataset:

The following are the results:

Understanding the Metrics

●     Answer Similarity

The concept of Answer Semantic Similarity pertains to the assessment of the semantic resemblance between the generated answer and the ground truth. This evaluation is based on the ground truth and the answer, with values falling within the range of 0 to 1. A higher score signifies a better alignment between the generated answer and the ground truth.

Measuring the semantic similarity between answers can offer valuable insights into the quality of the generated response. This evaluation utilizes a cross-encoder model to calculate the semantic similarity score.

●     Faithfulness

This measures the factual consistency of the generated answer against the given context. It is calculated from answer and retrieved context. The answer is scaled to (0,1) range. Higher the better.

The generated answer is regarded as faithful if all the claims made in the answer can be inferred from the given context. To calculate this, a set of claims from the generated answer is first identified. Each of these claims is cross-checked with the given context to determine if it can be inferred from the context. The faithfulness score is determined by:

●     Context Precision

Context Precision is a metric that evaluates whether all the ground-truth relevant items present in the contexts are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks. This metric is computed using the question, ground_truth and the contexts, with values ranging between 0 and 1, where higher scores indicate better precision.

Where K is the total number of chunks in contexts and vk∈ {0,1} is the relevance indicator at rank k.

●     Context Relevancy

This metric gauge the relevancy of the retrieved context, calculated based on both the question and contexts. The values fall within the range of (0, 1), with higher values indicating better relevancy.

Ideally, the retrieved context should exclusively contain essential information to address the provided query. To compute this, we initially estimate the value of |S| by identifying sentences within the retrieved context that are relevant for answering the given question. The final score is determined by the following formula:

●     Answer Relevancy

The evaluation metric, Answer Relevancy, focuses on assessing how pertinent the generated answer is to the given prompt. A lower score is assigned to answers that are incomplete or contain redundant information and higher scores indicate better relevancy. This metric is computed using the question, the context, and the answer.

The Answer Relevancy is defined as the mean cosine similarity of the original question to a number of artificial questions, which were generated (reverse engineered) based on the answer:

Where:

●     Egi is the embedding of the generated question i.

●     Eo is the embedding of the original question.

●     N is the number of generated questions, which is 3 default.

●     Context Recall

Context recall measures the extent to which the retrieved context aligns with the annotated answer, treated as the ground truth. It is computed using question, ground truth and the retrieved context, and the values range between 0 and 1, with higher values indicating better performance. To estimate context recall from the ground truth answer, each claim in the ground truth answer is analyzed to determine whether it can be attributed to the retrieved context or not. In an ideal scenario, all claims in the ground truth answer should be attributable to the retrieved context. A reference free version of this is available as context_utilization.

The formula for calculating context recall is as follows:

●     RAGAS Score 

This metric provides a measure of how well the summary captures the important information from the contexts. The intuition behind this metric is that a good summary shall contain all the important information present in the context (or text).

First extract a set of important key phrases from the context. These key phrases are then used to generate a set of questions. The answers to these questions are always yes (1) for the context. Ask these questions to the summary and calculate the summarization score as the ratio of correctly answered questions to the total number of questions.

Compute the question-answer score using the answers, which is a list of 1s and 0s. The question-answer score is then calculated as the ratio of correctly answered questions (answer = 1) to the total number of questions.

Milvus Benchmarking with VectorDBBench

Today, the growth of unstructured data and the rise of AI and LLMs have highlighted vector databases as a crucial component of the infrastructure. As the focus shifts to these tools, enterprises need to assess and select the right one for their business. Vector databases manage unstructured data like images, video, text and so on, using vector embeddings. Vector databases specialize in semantic similarity searches using a machine-learning technique called Approximate Nearest Neighbor (ANN). Vector databases are the vector store for RAG applications.

Milvus is a high-performance, highly scalable vector database that runs efficiently across a wide range of environments. Unstructured data varies in format and carries rich underlying semantics, making it challenging to analyze. To manage this complexity, embeddings are used to convert unstructured data into numerical vectors that capture its essential characteristics. These vectors are then stored in a vector database, enabling fast and scalable searches and analytics. Milvus offers robust data modeling capabilities, enabling you to organize your unstructured or multi-modal data into structured collections. Milvus allocates over 80% of its computing resources to its vector databases and search engine. Given the computational demands of high-performance computing, GPUs emerge as a pivotal element of vector database platform, especially within the vector search domain.

NVIDIA’s latest innovation, the GPU-based graph index CAGRA (CUDA ANNs GRAph-based), represents a significant milestone. With NVIDIA’s assistance, Milvus integrated support for CAGRA in its 2.4 version, marking a significant stride toward overcoming the obstacles of efficient GPU implementation in vector search.

The decision to leverage GPU indexes was primarily motivated by performance considerations. We undertook a comprehensive evaluation of Milvus’ performance utilizing the VectorDBBench tool, focusing on CAGRA index and observing key evaluation metrics like QPS (Queries Per Second), latency, and recall.

VectorDBBench is the best performing and cost-effective comparison open-source tool, and it offers more than just benchmark results for popular vector databases and cloud services.

Query Performance Metrics

Assessing the query performance of vector databases typically involves three key metrics: latency, queries per second (QPS), and recall rate.

Latency testing measures the time taken for a single query under serial testing conditions. P99 latency is a commonly used metric representing the duration within which 99% of queries are completed. It offers a more nuanced perspective than average latency and aligns closely with user experience.

QPS refers to a database's query capability under high concurrency. It is achieved by simultaneously sending multiple requests from the test client to maximize database CPU/ GPU utilization and observe throughput. Unlike latency, QPS is less susceptible to network fluctuations, providing a comprehensive evaluation of a vector database's real-world performance.

Recall is the proportion of the ground truth documents that are represented in the retrieved chunks. This is a measure of the completeness of the results.

High recall ensures that the LLM is comprehensive in its responses, capturing all the relevant information necessary for the task.

Procedure 1.     Milvus Distributed deployment with FlashBlade//S

Vector database benchmarking is executed with Milvus Distributed vector database with NVIDIA L40S GPUs. For detailed steps on deploying Distributed Milvus vector database, go to section Vector Database.

For detailed information, click the following links:

●     https://milvus.io/docs/install_cluster-helm-gpu.md

●     https://github.com/milvus-io/milvus

VectorDBBench

VectorDBBench is an open-source benchmarking tool designed for high-performance data storage and retrieval systems. This tool allows users to test and compare different vector database systems' performance to determine their specific use case's most suitable database system. Using VectorDBBench, users can make informed decisions based on the actual vector database performance of the systems they are evaluating.

VectorDBBench is written in Python and licensed under the MIT open-source license, The tool is actively maintained by a community of developers committed to improving its features and performance.

Procedure 1.     Install and run the VectorDBBench tool

Step 1.          Prerequisite:

Step 2.          Create a virtual python environment “milvus” and activate it:

Step 3.          Install vectordb-bench:

Step 4.          Run vectordb-bench:

OR

After completion, the tool is up and running and can access the streamlit web application using the network URL.

Step 5.          After completion of the benchmark testing with vectordb-bench tool, to come out of the python virtual environment, run the following script:

For more information, see: https://github.com/zilliztech/VectorDBBench

Procedure 2.     Run the Test

Step 1.          Select the database(s). There are various options available for vector databases like Milvus, Pinecone, PgVector, and so on.

Step 2.          Provide the required configuration information for the selected database:

Step 3.          Select the test case to run the benchmarking test.

Step 4.          Select the index type. You can select the index type depending upon their deployment characteristics and datasets. Some of the options provided by vectordbbench are DISKANN, HNSW, GPU_IVF_FLAT, GPU_IVF_PQ and so on. There is a flexibility provided to change the test parameters pertaining to a particular index type.

●     HSNW index type (CPU based):

The HNSW index is a graph-based indexing algorithm that can improve performance when searching for high-dimensional floating vectors. It offers excellent search accuracy and low latency but with the tradeoff of long index build time and it requires high memory overhead to maintain its hierarchical graph structure. For more details on this index type, refer: https://milvus.io/docs/hnsw.md

●     GPU_CAGRA Index Type (GPU-based)

GPU_CAGRA is a graph-based index optimized for GPUs. Using inference-grade GPUs to run the Milvus GPU version can be more cost-effective compared to using expensive training-grade GPUs.

 

Step 5.          You can provide your custom task label or go by default as provided by the tool.

Benchmarking Test Details

In this solution, we executed Search Performance test of Milvus vectorstore DB with Cohere 1M dataset size and 768 dimensions and varied the batch size for the ML model. Latencyp99, recall, load-duration, and QPS (Queries Per Second) are the few KPIs used to validate the test results. These tests were executed for both HSNW and GPU_CAGRA index types.

To update the batch size between test iterations, update the NUM_PER_BATCH variable in the __init__.py file which is stored under python3.12/site-packages/vectordb_bench/ directory.

 

Since these tests are executed with HSNW (CPU based) and GPU_CAGRA (GPU based) indexes, CPU utilization for HSNW and GPU utilization for GPU_CAGRA index were captured during the tests.

Test Results

This chapter contains the following:

●     Test Results for HSNW Index Search Performance

●     Test Results for GPU_CAGRA Index Search Performance

While running the test, we noticed that the CPU and GPU were being utilized during various stages of the test execution like data ingestion, query execution , and so on. The CPU and GPU utilizations are captured during Search Performance test execution with linux top and nvidia-smi tool.

Test Results for HSNW Index Search Performance

The following table lists the performance metrics for various batch sizes with Search Performance test executed with HSNW index type.

The following screenshot shows the CPU utilization by the query node during the Search Performance test execution. Milvus query node has consumed all the compute resources available on the worker node.

The HSNW index Search Performance numbers would vary and depend on the resources available on the underlying worker node where the query node is hosted.

Test Results for GPU_CAGRA Index Search Performance

The following table lists the performance metrics for various batch sizes with Search Performance test executed with GPU_CAGRA index type with one NVIDIA L40S GPU assigned to the milvus query node.

The following screenshot shows the GPU utilization by the query node during the Search Performance test execution.

From these results, we can infer that as the batch size increases, the Multi Modal RAG performs better, that is validated by the trend in the output KPIs with Milvus vector store using VectorDBBench tool.

QPS of GPU_CAGRA index is more than two times as that of HSNW index type. As explained earlier, the performance differences would vary depending on the type of GPU and CPUs used on the worker node.

The benchmark results underscore the substantial performance benefits of adopting GPU-accelerated indexes like CAGRA in Milvus. Not only does it excel in accelerating search tasks across batch sizes, but it also significantly enhances index construction speed, affirming the value of GPUs in optimizing vector database performance.

Conclusion

This Cisco Validated Design presents a robust and meticulously tested solution for deploying enterprise-grade Retrieval Augmented Generation (RAG) pipelines. By synergistically integrating the high-performance Cisco UCS X-Series based FlashStack infrastructure with the advanced capabilities of NVIDIA AI Enterprise software, including the RAG Blueprint and NVIDIA Inference Microservices (NIM), this architecture provides a streamlined path for leveraging generative AI with proprietary data.

The infrastructure is  designed using the Cisco UCS X-Series modular platform based FlashStack Datacenter, managed through Cisco Intersight. Deployment involves Red Hat OpenShift Container Platform clusters running on Cisco UCS X210c M7 bare metal compute nodes equipped with NVIDIA GPUs. The NVIDIA AI Enterprise software powers the inferencing workflow, while Portworx Enterprise Storage, backed by Pure Storage FlashArray and FlashBlade, ensures cloud-native storage for model repositories and other services.

Automation facilitated by Red Hat Ansible, provides Infrastructure as Code (IaC) for accelerating deployments.

The FlashStack solution is a validated approach for deploying Cisco and Pure Storage technologies in an enterprise data center. This release of the FlashStack VSI solution brings the following capabilities:

●     Fourth generation Intel Xeon Scalable processors with Cisco UCS X210 M7, C220 M7 and C240 M7 servers, enabling up to 60 cores per processor and 8TB of DDR-4800 DIMMs.

●     Sustainability monitoring and optimizations to meet Enterprise ESG targets that include power usage monitoring features across all layers of the stack and utilizing the Cisco UCS X-Series advanced power and cooling policies.

●     Pure Storage FlashArray Unified Block and File consisting of FC-SCSI, FC-NVMe, iSCSI, NVMe-TCP, NVMe-RoCEv2 as well as NFS and SMB storage.

●     Pure Storage FlashBlade//S, a high-performance consolidated storage platform for both file (with native SMB and NFS support) and object (S3) workloads, delivering a simplified experience for infrastructure and data management.

●     Red Hat Openshift innovations.

●     Cisco Intersight continues to deliver features that simplify enterprise IT operations, with services and workflows that provide complete visibility and operations across all elements of FlashStack datacenter. Also, Cisco Intersight integration with VMware vCenter and Pure Storage FlashArray extends these capabilities and enable workload optimization to all layers of the FlashStack infrastructure.

Ultimately, this validated solution stands as a foundational reference architecture, significantly mitigating the complexities and risks associated with deploying sophisticated RAG workloads. It empowers enterprises to confidently build, manage, and scale secure, high-throughput private RAG applications, accelerating time-to-value for critical AI initiatives.

About the Authors

Paniraja Koppa, Technical Marketing Engineer, Cisco Systems, Inc.

Paniraja Koppa is a member of the Cisco Unified Computing System (Cisco UCS) solutions team. He has over 15 years of experience designing, implementing, and operating solutions in the data center. In his current role, he works on design and development, best practices, optimization, automation and technical content creation of compute and hybrid cloud solutions. He also worked as technical consulting engineer in the data center virtualization space. Paniraja holds a master’s degree in computer science. He has presented several papers at international conferences and speaker at events like Cisco Live US and Europe, Open Infrastructure Summit, and other partner events. Paniraja’s current focus is on Generative AI solutions.

Gopu Narasimha Reddy, Technical Marketing Engineer, Cisco Systems, Inc.

Gopu Narasimha Reddy is a Technical Marketing engineer with the UCS Solutions team at Cisco. He is currently focused on validating and developing Cisco UCS infrastructure solutions for enterprise workloads with different operating environments including Windows, VMware, Linux, and Kubernetes. Gopu is also involved in publishing database benchmarks on Cisco UCS servers. His areas of interest include building and validating reference architectures, and development of sizing tools in addition to assisting customers in database deployments.

Mahesh Pabba, Customer Delivery Architect, Cisco Systems, Inc.

Mahesh Pabba is a seasoned technology professional with vast experience in the IT industry. With a strong foundation in enterprise applications, he has successfully navigated various domains, excelling in automation and development using cutting-edge technologies. As a skilled Solution Architect, he has designed and implemented numerous innovative solutions throughout his career. Additionally, he possesses expertise in artificial intelligence, specializing in training, inferencing, and deploying AI solutions on Cisco hardware, encompassing compute, network, and storage environments. His unique blend of technical acumen and solution-oriented approach positions him as a valuable contributor to any AI design initiative.

Hang Yu, NVIDIA

Hang Yu is a Solutions Architect with expertise in AI/ML development and production. He focuses on the full-stack AI software, building and deploying enterprise-grade AI solutions, leading projects from initial design to production deployment. He has been working closely with Cisco on enabling this validated design.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

●     Chris O’Brien, Senior Director, Technical Marketing, Cisco Systems, Inc.

●     Umar Ali, Business Product Manager, Cisco Systems, Inc.

●     Nitin Garg, Technical Marketing Engineering Technical Leader, Cisco Systems, Inc.

●     Meenakshi Kaushik, NVIDIA

●     Ruchika Kharwar, NVIDIA

●     Sumit Bhattacharya, NVIDIA

●     Nikhil Kulkarni, NVIDIA

●     Nikita Jain, NVIDIA

●     Anurag Guda, Technical Marketing Engineer, NVIDIA

●     Craig Waters, Solutions Director, Pure Storage, Inc.

●     Simranjit Singh, Solutions Architect, Pure Storage, Inc.

●     Robert Alvarez, AI Solutions Architect, Pure Storage, Inc.

●     Unnikrishnan R, Senior Solutions Architect, Pure Storage, Inc.

Appendix

This appendix contains the following

●     Appendix A – References used in this guide

Appendix A – References used in this guide

Compute

Cisco Intersight: https://www.intersight.com

Cisco Intersight Managed Mode: https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Intersight_Managed_Mode_Configuration_Guide.html

Cisco Unified Computing System X-Series Modular System: https://www.cisco.com/go/ucsx

Cisco UCS 6536 Fabric Interconnect Data Sheet: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs6536-fabric-interconnect-ds.html

AI-ready infrastructure - Cisco Validated Design Zone: https://www.cisco.com/c/en/us/solutions/design-zone/ai-ready-infrastructure.html

Cisco AI native infrastructure for Data Center: https://www.cisco.com/site/us/en/solutions/artificial-intelligence/infrastructure/index.html

Cisco Unified Compute System automation repositories for Cisco Validate Designs (CVD) and white papers: https://github.com/ucs-compute-solutions

Network

Cisco Nexus 9300-GX Series Switches: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/nexus-9300-gx-series-switches-ds.html

Pure Storage

Pure Storage FlashArray//XL: https://www.purestorage.com/products/unified-block-file-storage/flasharray-xl.html

Pure Storage FlashBlade//S: https://www.purestorage.com/products/unstructured-data-storage/flashblade-s.html

Portworx by Pure Storage: https://docs.portworx.com/

FlashStack

FlashStack Design Guides: https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/data-center-design-guides-all.html#FlashStack

FlashStack with Red Hat OCP and Virtualization platform CVD (Base CVD): https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flashstack_ocp_baremetal_imm.html

Red Hat OpenShift

Documentation: https://docs.openshift.com/

Red Hat OpenShift Container Platform: https://www.redhat.com/en/technologies/cloud-computing/openshift/container-platform

Red Hat Hybrid Cloud Console: https://cloud.redhat.com/

NVIDIA Enterprise

GitHub page for NVIDIA Blueprints: https://github.com/NVIDIA-AI-Blueprints

NVIDIA NIM Operator: https://docs.nvidia.com/nim-operator/latest/index.html

NVIDIA NIM Catalog: https://catalog.ngc.nvidia.com/

Interoperability Matrix

Cisco UCS Hardware Compatibility Matrix: https://ucshcltool.cloudapps.cisco.com/public/ 

Pure Storage FlashStack Compatibility Matrix. Note, this interoperability list will require a support login from Pure: https://support.purestorage.com/FlashStack/Product_Information/FlashStack_Compatibility_Matrix

Feedback

For comments and suggestions about this guide and related guides, join the discussion on Cisco Community here: https://cs.co/en-cvds.

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_P1)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)