FlexPod Datacenter with Red Hat OpenShift AI for MLOps Design and Deployment Guide

◦      

Published: May 2025

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: http://www.cisco.com/go/designzone.

Executive Summary

Artificial Intelligence (AI) and Machine Learning (ML) are driving unprecedented investment and innovation in enterprise data centers as businesses seek to leverage these technologies across a wide range of applications and quickly deliver them in production. This is particularly challenging considering that each application has its own unique set of requirements based on the specific ML use case it implements and the SLAs it must meet. The ML use case will impact several downstream decisions, such as the model type (generative or predictive or both), selection of model(s), and decisions around whether to build the models in-house or use pre-trained ones and may others. These applications will also utilize Enterprise data, which must be prepped and curated before they can be integrated, further complicating the process.

The delivery of these ML applications in production also requires extensive collaboration as well as integration of tools, processes and workflows between existing teams and newer ML and data teams. Once deployed, these applications, along with the models and data pipeline will need to be improved and maintained over the long term. Application teams will need to integrate the model delivery and data pipelines into their existing application delivery framework which may include continuous integration and continuous delivery (CI/CD), automation and other DevOps best practices. As a result, the application delivery pipeline — with integrated model and data pipelines — is significantly more challenging and complex. Scaling this environment both in terms of the number of models and the applications leveraging them, further adds to the challenges that an Enterprise faces. Gartner estimates that, on average, only 54 percent of AI projects make it from pilot to production, and so, despite the heavy investments in AI, delivering production applications with integrated models and data will continue to be a challenge for the foreseeable future.

To address these challenges and ensure successful outcomes, Enterprises need a strategic, holistic approach to accelerate and quickly operationalize AI/ML models for use by Enterprise applications. A crucial first step in addressing these challenges is implementing Machine Learning Operations (MLOps) to streamline model delivery. Unlike siloed, ad-hoc efforts that are inefficient, MLOps allows organizations to innovate, scale, and bring sustainable value to the business. MLOps uses DevOps principles to accelerate model delivery with consistency and efficiency. Like DevOps practices that integrate software development and IT operations with CI/CD to make the application delivery process more agile, MLOps bring DevOps principles used in application development to model delivery. Adopting MLOps is therefore essential for any organization seeking to scale AI/ML initiatives sustainably.

The solution in this design guide delivers a complete infrastructure stack with MLOps that Enterprises can deploy to efficiently manage, accelerate, and scale multiple AI/ML initiatives from incubation to production. The solution uses Red Hat OpenShift AI as the MLOps platform running on FlexPod baremetal infrastructure, (Cisco UCS X-Series, NetApp, and Cisco Nexus) with Red Hat Openshift to support Enterprise AI/ML initiatives at scale.

Audience

This document is intended for, but not limited to, sales engineers, technical consultants, solution architect and enterprise IT, and machine learning teams interested in learning how to design, deploy, and manage a production-ready AI/ML infrastructure for hosting machine learning models and AI-enabled applications.

Purpose of this document

This document serves as a reference architecture for MLOps using Red Hat OpenShift AI to accelerate AI/ML efforts and deliver models that application teams can use to build and deliver integrated ML applications. This document also provides design guidance for building a production-ready AI/ML infrastructure based on Red Hat OpenShift running on FlexPod datacenter infrastructure, leveraging optional NVIDIA AI Enterprise software and GPUs. 

Technology Overview

This chapter contains the following:

●    Model Delivery Lifecycle

●    Machine Learning Operations

●    Red Hat OpenShift AI

●    Red Hat OpenShift

●    Kubernetes Operators

●    NVIDIA AI Enterprise

●    NVIDIA L40S Tensor Core GPU

●    Compute Unified Device Architecture

●    FlexPod Datacenter

●    FlexPod AI

Model Delivery Lifecycle

The end goal of an Enterprise’s AI/ML initiatives are to deploy ML-integrated applications that bring value to the business. Figure 1 illustrates the model delivery pipeline of an ML model before it can integrate into the application and the continuous integration and continuous delivery lifecycle that is required to maintain that model. The workflow also highlights integration of the data pipeline with the model pipeline before it is integrated into the application workflow.

Typically, for a given model, the life cycle will have the following stages which are then maintained for the lifetime of that ML model to ensure accuracy and minimize drift.

Data Pipeline – This is represented by the first box in the workflow. In this stage, data is collected and consolidated from various data sources. This is the Enterprise data that is necessary to for the ML use case in question – either for differentiation or to address the limitations that Large Language Models (LLMs) have in an Enterprise use case. Enterprises must continuously manage a steady-stream of continuously changing data from different sources and curate them for use in a model delivery pipeline. Data engineers may have to perform activities such as ingestion, exploration, labeling and preparation to deliver the curated data to the second stage in the above workflow, the model delivery or ML pipeline. This stage is part of the data delivery pipeline with a life cycle of its own and serves as input to the model workflow.

Model Delivery – This is represented by the second box in the workflow (see Figure 1). It includes the development and deployment stages of delivering a model in production, also know as model serving. The published models are made available using a standard interface (RPC, HTTPs) that applications teams can integrate into their CI/CD or development workflow to build and deliver the ML-enabled application. The model delivery pipeline is the focus of MLOps and typically involves the following stages. The next section details the ML pipelines and MLOps.

●    Access ML-ready data – This is the output from the Gather and Prepare Data box in the above pipeline. It takes as input consolidated structured or unstructured data that has been cleaned, labeled, and formatted for use by ML engineers to evaluate, train and test the models. The data pipeline for AI delivers a curated dataset that serves as input to ML pipeline.

●    Model Training – In this stage, data from the previous stage are used to develop, train, and test a new model from scratch or evaluate and re-train/customize a foundational model using Enterprise data. This stage involves any experimentation and evaluation work involved identify and select a model that best suits the needs of the use case in question. Other model customizations such as fine-tuning, and prompt engineering may also be done in this stage.

●    Model Validation – In this stage, the model that has been selected and trained using Enterprise data is tested to ensure that it is ready for production deployment.

●    Model Serving – In this stage, the model is deployed into production and available as an inferencing endpoint that applications can then use in their applications to implement the specific ML use case. Models are hosted on inferencing engines (for example, Virtual LLM) and delivered as an API endpoint accessible via a standard interface (RPC or HTTPS) by the application. The inference server or engine must meet the performance criteria of the application – however, the overall application design will determine how the inferencing engine hosting the model is deployed at scale, with resiliency.

●    Automation – This stage represents the automation required to maintain the ML model with CI/CD to adapt to new data and other changes that maybe necessary based on feedback from its use in production.

●    AI-Enabled Application Deployment – This stage represents the output of ML delivery pipeline which a published model for use in product. Application teams takes the delivered model and integrates it into their  software development processes with CI/CD, and other DevOps and GitOps practices to deliver ML-enabled applications that implements a given ML use case. Models are continuously monitored in production with a feedback loop to continuously improve the model’s performance and accuracy.

Machine Learning Operations

Machine Learning Operations (MLOps) are a set of best practices to streamline and accelerate the delivery of machine learning (ML) models. The delivery of these ML models for production use or model serving is key to operationalizing AI so that Enterprises can build and deliver ML-enabled applications. Once delivered, the maintenance of the models are critical for ensuring the accuracy and reliability of model predictions and other outputs. MLOps leverages DevOps and GitOps principles to enable continuous retraining, integration, and delivery. Delivering ML-enabled applications involves new roles such as data scientists and ML engineers that weren’t part of traditional software/application development. These new roles will also require new tools and environments to do their work. As such, MLOps platforms will typically include a wide ecosystem of tools, technologies, libraries, and other components, including automation capabilities.

Automation is integral to MLOps to accelerate efforts that minimizes technical debt, enabling Enterprises to deliver and maintain models at scale. MLOps pipelines also need to continuously retrain models to keep up with everchanging data to ensure model performance. MLOps brings consistency and efficiency to the model delivery process.

In this solution, MLOps is provided by Red Hat OpenShift AI serves as the MLOps platform to streamline, scale and accelerate model delivery, with Red Hat OpenShift providing cloud-native (Kubernetes) cluster management and orchestration.

Red Hat OpenShift AI

Red Hat OpenShift AI (previously known as Red Hat OpenShift Data Science or RHODS) is a flexible and scalable platform for AI/ML and MLOps that enables enterprises to create and deliver AI-enabled applications at scale. Built using open-source technologies and Red Hat OpenShift as the foundation, OpenShift AI provides a trusted, operationally consistent environment for Enterprise teams to experiment, serve models, and deliver ML-enabled applications. Red Hat OpenShift AI running on OpenShift provides a single enterprise-grade application platform for ML models and applications that use them. Data scientists, engineers, and app developers can collaborate in a single destination that promotes consistency, security, and scalability. OpenShift administrators that manage existing application environment can continue to do the same for OpenShift AI and ML workloads. This also allows application, ML, and data science teams to focus on their areas of work and spend less time managing the infrastructure. 

Red Hat OpenShift AI includes key capabilities to accelerate the delivery of AI/ML models and applications in a seamless, consistent manner, at scale. The platform provides the development environment, tools, and frameworks that data scientists and machine learning teams need to build, deploy, and maintain AI/ML models in production. OpenShift AI streamlines the ML model delivery process from development to production deployment (model serving) with efficient life cycle management and pipeline automation. From the OpenShift AI console, AI teams can select from a pre-integrated, Red Hat supported set of tools and technologies or custom components that are enterprise managed, providing the flexibility that teams need to innovate and operate with efficiency. OpenShift AI also makes it easier for multiple teams to collaborate on one or more efforts in parallel.

OpenShift AI is compatible with leading AI tools and frameworks such as TensorFlow, PyTorch, and can work seamlessly with NVIDIA GPUs, to accelerate AI workloads. It provides pre-configured Jupyter notebook images with popular data science libraries. Red Hat tracks, integrates, tests, and supports common AI/ML tooling and model serving on RedHat OpenShift. The latest release of Red Hat OpenShift AI delivers enhanced support for predictive and generative AI model serving and improves efficiency of data processing and model training.

Other key features of OpenShift AI include:

●    Collaborative Workspaces: OpenShift offers a collaborative workspace where teams can work together and collaborate on one or more models in parallel.

●    Development Environments: ML teams can use Jupyter notebooks as a service using pre-built images, common Python libraries and open-source technologies such as TensorFlow and PyTorch to work on their models.  In addition, administrators can add customized environments for specific dependencies or for additional IDEs such as RStudio and VSCode.

●    Model Serving at scale: Multiple Models can be served for integration into intelligent AI-enabled applications using inferencing servers (for example, Intel OpenVINO, NVIDIA Triton) using GPU or CPU resources provided by the underlying OpenShift cluster without writing a custom API server. These models can be rebuilt, redeployed, and monitored by making changes to the source notebook.

●    Support for enhanced model serving with the ability to use multiple model servers to support both predictive and GenAI, including support for KServe, a Kubernetes custom resource definition that orchestrates serving for all types of models, vLLM and text generation inference server (TGIS), serving engines for LLMs and Caikit-nlp-tgis runtime, which handles natural language processing (NLP) models and tasks. Enhanced model serving allows users to run predictive and GenAI on a single platform for multiple use cases, reducing costs and simplifying operations. This enables out-of-the-box model serving for LLMs and simplifies the surrounding user workflow.

●    Innovate with open-source capabilities: Like Red Hat OpenShift, OpenShift AI integrates with open-source tools and leverages a partner ecosystem to enhance the capabilities of the platform, minimizing vendor lock-ins.

●    Data Science Pipelines for GUI-based automation using OpenShift pipelines: OpenShift AI leverages OpenShift pipelines to automate ML workflow using an easy to drag-and-drop web UI as well as code driven development of pipelines using a Python SDK.

●    Model monitoring visualizations for performance and operational metrics, improving observability into how AI models are performing.

●    New accelerator profiles enable administrators to configure different types of hardware accelerators available for model development and model-serving workflows. This provides simple, self-service user access to the appropriate accelerator type for a specific workload.

By using Red Hat OpenShift AI, enterprises can manage and maintain AI/ML models and the applications using the same models on a single, unified platform and simplify overall management of the environment.

Red Hat OpenShift

Red Hat OpenShift is a leading enterprise application platform that brings together a comprehensive set of tools and services that streamline the entire application lifecycle, from development to delivery and maintenance of application workloads. It allows organizations to modernize their applications and includes multiple advanced open-source capabilities that are tested and integrated with the underlying certified Kubernetes environment, such as Red Hat OpenShift Serverless, Red Hat OpenShift Pipelines and Red Hat OpenShift GitOps. Red Hat OpenShift offers a complete set of services that helps developers code applications with speed, flexibility, and efficiency. OpenShift is designed to support anywhere from a few machines and applications to thousands of machines and applications and allows enterprises to extend their application environment from on-prem to public cloud and multi-cloud environments.

Figure 2 shows the high-level architecture of Red Hat Openshift.

Red Hat OpenShift uses Red Hat Enterprise Linux CoreOS (RHCOS), a container-oriented operating system that is specifically designed for running containerized applications and provides several tools for fast installation, Operator-based management, and simplified upgrades. RHCOS includes:

●    Ignition, which is used as a first-boot system configuration for initially bringing up and configuring machines.

●    CRI-O, a Kubernetes native container runtime implementation that integrates closely with the operating system to deliver an efficient and optimized Kubernetes experience. CRI-O provides facilities for running, stopping, and restarting containers.

●    Kubelet, the primary node agent for Kubernetes that is responsible for launching and monitoring containers.

Kubernetes Operators

AI/ML workloads, like many modern applications, uses containers and Kubernetes (K8s) orchestration as the de facto development environment for model development and AI-enabled applications. Kubernetes offer several benefits, but one key attribute is its extensibility. Kubernetes provides an Operator framework that vendors and open-source communities can use to develop and deploy self-contained operators that extend the capabilities of the K8s cluster. These operators generally require minimum provisioning and are usually self-managed with automatic updates (unless disabled) and handle life-cycle management. Kubernetes operators are probably the closest thing to an easy-button in infrastructure provisioning (short of IaC). In the Red Hat OpenShift environment that this solution uses, it is even easier to deploy and use operators. Red Hat OpenShift provides an embedded OperatorHub, directly accessible from the cluster console. The Red Hat OperatorHub has hundreds of Red Hat and community certified operators that can be deployed with a few clicks.

To support AI/ML workloads and OpenShift AI, the following Red Hat OpenShift operators are deployed in this solution to enable GPU, storage, and other resources:

●    Red Hat Node Feature Discovery Operator to identify and label hardware resources (for example, NVIDIA GPUs)

●    NVIDIA GPU Operator deploys and manages the GPU resource on a Red Hat OpenShift cluster (for example, Guest OS vGPU drivers)

●    NetApp Trident Operator for managing container-native persistent storage required for model delivery, backed by NetApp ONTAP storage – file, block, and object store.

●    Red Hat OpenShift AI Operator deploys OpenShift AI on any OpenShift cluster

For more information on Red Hat OpenShift Operators, see: https://www.redhat.com/en/technologies/cloud-computing/openshift/what-are-openshift-operators.

NVIDIA AI Enterprise

NVIDIA AI Enterprise (NVAIE) is a comprehensive suite of enterprise-grade, cloud-native software, hardware, and support services offered by NVIDIA for artificial intelligence (AI) and machine learning (ML) applications. NVIDIA describes NVAIE as the “Operating System” for enterprise AI. NVIDIA AI Enterprise includes key enabling technologies for rapid deployment, management, and scaling of AI workloads. It includes NVIDIA GPUs, Kubernetes Operators for GPUs, virtual GPU (vGPU) technology, and an extensive software library of tools and frameworks optimized for AI that make it easier for enterprises to adopt and scale AI solutions on NVIDIA infrastructure.

NVAIE can be broadly categorized into Infrastructure Management, AI Development, and Application Frameworks optimized for AI. For more details on NVAIE, see: https://www.nvidia.com/en-us/data-center/products/ai-enterprise/.

This solution optionally leverages the NVIDIA AI Enterprise Software suite (along with other complementary partner components) to extend and operationalize a robust, production-ready FlexPod AI infrastructure to support a range of use cases.

NVIDIA’s GPU operator for Red Hat OpenShift provides seamless deployment and management of GPU resources and CUDA libraries for optimal use of GPU to support various AI/ML use cases. NVIDIA AI Enterprise can be used to extend those capabilities even further. 

NVAIE is a licensed software from NVIDIA that must be certified to run on the infrastructure servers. For more information on the licensing and certification, click the links below:

●    Licensing: https://resources.nvidia.com/en-us-ai-enterprise/en-us-nvidia-ai-enterprise/nvidia-ai-enterprise-licensing-guide?pflpid=5224&lb-mode=preview

●    Certification: https://www.nvidia.com/en-us/data-center/products/certified-systems/

For additional information on NVIDIA AI Enterprise, go to: https://www.nvidia.com/.        

NVIDIA L40S Tensor Core GPU

The NVIDIA L40S GPU Accelerator is a full height, full-length (FHFL), PCI Express Gen4 graphics solution based on the NVIDIA Ada Lovelace architecture. The NVIDIA L40S GPU delivers acceleration for the next generation of AI-enabled applications—from gen AI, LLM inference, small-model training and fine-tuning to 3D graphics, rendering, and video applications.

Table 1.        NVIDIA L40S – Technical Specification

For more information, see: https://resources.nvidia.com/en-us-l40s/l40s-datasheet-28413

Compute Unified Device Architecture

Compute Unified Device Architecture (CUDA) is a parallel computing platform and application programming interface (API) model from NVIDIA that enables general purpose computing on GPUs that were originally designed for graphics. CUDA excels in complex mathematical computations and data processing tasks that can run on thousands of GPU cores in parallel, making it well suited for compute-intensive AI/ML use cases. It also provides memory management to enable efficient data transfers between the CPU and GPU. NVIDIA’s CUDA Toolkit provides developers with the software tools and libraries for developing GPU-accelerated applications that harness the parallel processing capabilities of the GPUs. It includes a compiler, debugger, runtime libraries, and other tools that simplify the process of GPU programming.

FlexPod Datacenter

The FlexPod Datacenter is a reference architecture for hosting a wide range of enterprise workloads on both virtualized baremetal infrastructure in enterprise data centers. Cisco Validated Designs (CVDs) for FlexPod Datacenter solutions provide design and implementation guidance as well as Infrastructure as Code (IaC) automation using Red Hat Ansible to accelerate enterprise data center infrastructure deployments. The designs incorporate product, technology, and industry best practices to deliver a highly-available, scalable, and flexible architecture.

The key infrastructure components for compute, network, and storage in a FlexPod Data center solution are:

●    Cisco Unified Computing System (Cisco UCS) Infrastructure

●    Cisco Nexus 9000 switches

●    Cisco MDS 9000 SAN switches (when using Fibre Channel SAN)

●    NetApp AFF/FAS/ASA storage

FlexPod designs are flexible and can be scaled-up or scaled-out without sacrificing feature/functionality. FlexPod  solutions are built and validated in Cisco labs to ensure interoperability and minimize risk in customer deployments. CVD solutions enables enterprise IT teams to save valuable time that would otherwise be spent on designing and integrating the solution in-house.

All FlexPod CVDs are available in the Cisco Design Zone here: https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/flexpod-design-guides.html

FlexPod AI

The FlexPod AI solution in this document serves as a foundational infrastructure architecture to deliver a robust, scalable design for compute, networking, storage and GPU options for Enterprise AI initiatives. Using this solution, enterprises can quickly start on their AI journey and scale incrementally as the enterprise needs grow.

The FlexPod AI design adds the following components to the foundational architecture documented in the foundational FlexPod Datacenter Baremetal infrastructure solution. 

●    Cisco UCS X440p PCIe nodes, capable of hosting up to four GPUs (only some models). Each PCIe node is paired with a Cisco UCS compute node, specifically the Cisco UCS X210c M7 server but a Cisco UCS X410c can also be used with UCS. Connectivity between the compute node and PCIe node requires a PCIe mezzanine card on the compute node and a pair of X-Fabric modules on the Cisco UCS X9508 server chassis.

●    NVIDIA GPUs (L40S-48GB) for accelerating AI/ML workloads and model delivery pipeline.

●    Red Hat OpenShift for Kubernetes based container orchestration and management.

●    NetApp Trident for persistent storage (backed by NetApp ONTAP Storage).

●    Red Hat OpenShift AI for MLOps.

●    Object store provided by NetApp AFF storage

Figure 3 shows the components validated in this solution for hosting AI/ML workloads.

The following sections provide a brief overview of the new components added in this solution to support AI/ML workloads.

Cisco UCS-X 9508 Server Chassis

Cisco UCS X9508 is a 7RU chassis with eight slots that can support up 4 PCIe nodes, each paired with a compute node for a total of 4 compute nodes as shown in Figure 4.

PCIe or GPU nodes must be deployed in a slot adjacent to the compute node which will enable the compute node to automatically recognize and use the adjacent PCIe node as an extension of itself. To enable this connectivity between GPUs in PCIe nodes and the compute nodes, the following additional components are required to enable PCIe Gen4 connectivity between compute and GPU nodes.

●    PCIe mezzanine card on the compute node (UCSX-V4-PCIME)

●    Pair of Cisco UCS X9416 X-Fabric Modules deployed to the back of Cisco UCS X-series server chassis (UCSX-F-9416) - see Figure 5

The X-Fabric modules provide a redundant PCIe 4 fabric to enable PCIe connectivity between compute and PCIe/GPU nodes.

The Cisco UCS X9508 Chassis is midplane-less design, provides fewer obstructions for better airflow. The vertically oriented Cisco UCS X210c or X410c compute nodes and the PCIe nodes connect directly to horizontally oriented X-Fabric modules, located at the back of the chassis (see Figure 5). The innovate design enables Cisco UCS X-Series to easily upgrade to newer technologies and hardware without requiring forklift upgrades.

Cisco UCS X440p PCIe Node and GPUs

The Cisco UCS X440p PCIe node (UCSX-440P-U) is the first PCIe node supported on a Cisco UCS X-Series fabric. It is part of the Cisco UCS-X Series modular system, managed using Cisco Intersight and integrated to provide GPU acceleration for workloads running on Cisco UCS compute (X210c, X410c) nodes. GPUs can be installed on the PCIe node and then paired with a compute node in an adjacent slot to support AI/ML, VDI, and other workloads that require GPU resources. GPUs. The PCIe node requires riser cards to support different GPU form factors, either full height, full length (FHFL) or half height, half length (HHHL) GPUs as outlined below:

●    Riser Type A: Supports 1 x 16 PCIe connectivity for FHFL GPUs (UCSX-RIS-A-440P)

●    Riser Type B: Supports 1 x 8 PCIe connectivity for HHHL GPUs (UCSX-RIS-B-440P)

Each PCIe node supports a maximum of two riser cards, with each riser card capable of supporting up to:

●    1 x 16 FHFL dual slot PCIe cards, one per riser card for a total of two FHFL cards

●    1 x 8 HHHL single slot PCIe card, two per riser card for a total of four HHHL cards

The NVIDIA L40S-48GB GPU (UCSX-GPU-L40S) deployed in this solution is a FHFL GPU and uses the Type A riser card.

PCIe or GPU nodes must be deployed in a slot adjacent to the compute node and the compute node will automatically recognize and use the adjacent PCIe node as an extension of itself in Cisco Intersight as shown below. The compute node does require a mezzanine card for connecting to the PCIe fabric and node hosting GPUs as shown in the figure below. The figure also shows PCIe node is in slot 6 with two L40S GPUs, as indicated by the node name: PCIe-Node6-GPU-1, -2 with compute node in slot 5.

As stated earlier, each PCIe node allows you to add up to four HHHL GPUs to accelerate workloads running on either a Cisco UCS X210c or Cisco UCS X410c compute node. This provides up to 16 GPUs per chassis. As of the publishing of this document, the following GPU models are supported on a Cisco UCS X440p PCIe node.

Table 2.        GPU Options on Cisco UCS X-Series Server System

*Using the optional front mezzanine GPU adapter (UCSX-X10C-GPUFM-D) on Cisco UCS X210c compute node.

If additional GPUs are needed, up to two GPUs can be added using an optional GPU front mezzanine card on the Cisco UCS X210c or Cisco UCS X410c compute nodes. Only two GPU models are currently supported in this configuration but enables up to have up to 24 GPUs per chassis. Product IDs for enabling GPU acceleration components on Cisco UCS-X are summarized in Table 3.

Table 3.        Product IDs for GPU acceleration using PCIe Node

For NVIDIA GPUs, see NVIDIA AI Enterprise Software Licensing guide for up-to-date licensing and support information: https://resources.nvidia.com/en-us-nvaie-resource-center/en-us-nvaie/nvidia-ai-enterprise-licensing-pg?lb-mode=preview

NetApp Trident for Kubernetes Persistent Storage

NetApp Trident is an open-source, fully supported storage orchestrator for containers created by NetApp. It has been designed from the ground up to help you meet your containerized applications persistence demands using industry-standard interfaces, such as the Container Storage Interface (CSI). With Trident, microservices and containerized applications can take advantage of enterprise-class storage services provided by the full NetApp portfolio of storage systems. In a FlexPod environment, Trident is utilized to allow end users to dynamically provision and manage persistent volumes for containers backed by FlexVols and LUNs hosted on ONTAP-based products such as NetApp AFF and FAS systems.

Trident deploys (see Figure 6) as a single Trident Controller Pod and one or more Trident Node Pods on the Kubernetes cluster and uses standard Kubernetes CSI Sidecar Containers to simplify the deployment of CSI plugins. Kubernetes CSI Sidecar Containers are maintained by the Kubernetes Storage community.

Kubernetes node selectors and tolerations and taints are used to constrain a pod to run on a specific or preferred node. You can configure node selectors and tolerations for controller and node pods during Trident installation.

●    The controller plugin handles volume provisioning and management, such as snapshots and resizing.

●    The node plugin handles attaching the storage to the node.

In this solution, NetApp Trident dynamically provides K8s native persistent storage using both NFS file and iSCSI/NVMe-TCP block storage hosted on a NetApp AFF storage.

NetApp ONTAP S3

With the growing demand for S3-compatible storage, ONTAP extended its support to include an additional scale-out storage option for S3. Capitalizing on ONTAP's robust data management framework, ONTAP S3 provides S3-compatible object storage capabilities, allowing data to be represented as objects within ONTAP-powered systems, including AFF and FAS. Beginning with ONTAP 9.8, you can enable an ONTAP Simple Storage Service (S3) object storage server in an ONTAP cluster, using familiar manageability tools such as ONTAP System Manager to rapidly provision high-performance object storage for development and operations in ONTAP and taking advantage of ONTAP's storage efficiencies and security.

Solution Design

This chapter contains the following:

●    High-Level Design

●    Solution Topology

●    Design Overview

●    Connectivity Design

●    Logical Design

The FlexPod Datacenter for AI with Red Hat OpenShift AI solution aims to address the following design goals:

●    Best-practices based design for AI/ML workloads, incorporating product, technology, and industry best practices.

●    Simplify and streamline operations for AI/ML. Ease integration into existing deployments and processes.

●    Flexible design with options for tools, technologies and individual components and sub-systems used in the design can be modified to adapt to changing requirements (for example, storage access, network design)

●    Modular design where sub-system components (for example, links, interfaces, model, platform) can be expanded or upgraded as needed.

●    Scalable design: As deployments grow, FlexPod Datacenter can be scaled up or out to meet enterprise needs. Each FlexPod Datacenter deployment unit can also be replicated as needed to meet scale requirements.

●    Resilient design across all layers of the infrastructure with no single point of failure.

The following sections explain the solution architecture and design that meets these design requirements.

High-Level Design

A high-level design of the FlexPod Datacenter AI solution using Red Hat OpenShift AI is shown in Figure 9.

The solution provides a foundational infrastructure design for AI using NVIDIA GPUs, NetApp Storage, Red Hat OpenShift, and MLOps provided by Red Hat OpenShift AI.

Solution Topology

The high-level infrastructure design and topology built in Cisco labs for validating this FlexPod AI Datacenter infrastructure with Red Hat OpenShift AI is shown in Figure 10.

Design Overview

The FlexPod Datacenter infrastructure in this solution is an end-to-end 100Gb Ethernet design using  NFS file and block (iSCSI, NVMe-TCP) storage hosted on NetApp AFF C-series storage. The solution provides an OpenShift Kubernetes (K8s)  infrastructure for hosting cloud-native application workloads and ML workloads. The Openshift is running on Cisco UCS baremetal servers, specifically Cisco UCS X210C M7 servers with the latest Intel processors. Cisco UCS infrastructure and NetApp storage use multiple virtual port channels (VPCs) to connect to a pair of top-of-rack Cisco Nexus 9000 series switches. The access layer switches can use either multiple 100GbE or 400GbE uplinks to connect into the larger data center fabric. The solution incorporates design, technology, and product best practices to deliver a highly scalable and flexible architecture with no single point of failure.

To support Enterprise AI/ML initiatives and model delivery efforts, FlexPod AI extends the FlexPod Datacenter design as outlined in the upcoming sections to deliver a robust infrastructure design with GPU acceleration.

Cisco UCS servers with GPU nodes and NVIDIA GPUs

The design uses the Cisco UCS X-Series server chassis with NVIDIA GPUs to provide the compute and GPU resources in the solution. The Cisco UCS X9508 server chassis is a 7RU chassis with eight slots where Cisco UCS servers and PCIe nodes can be deployed with GPUs to provide a total of 8 FHFL GPUs or 16 HHHL GPUs per chassis. The Cisco UCS X9508 supports up to four PCIe nodes, with each PCIe node paired with a compute node, either a Cisco UCS X210c or X410c server. This design was validated using NVIDIA L40S-48GB GPUs, with two NVIDIA L40S GPUs installed on each GPU worker node. Alternatively, Cisco UCS C-Series Rackmount Servers can also be used. A Cisco UCS C-Series M7 server with Intel processors can support up to 3 x L40S GPUs.

Cisco UCS X-Fabric Technology

To support GPU acceleration on Cisco UCS X-Series systems, Cisco’s first generation UCS X-fabric technology, a pair of Cisco UCS X9416 Fabric modules are deployed to enable PCIe connectivity between UCS servers and UCS X440p PCIe nodes housing NVIDIA GPUs. The first-generation X-fabric supports 32 lanes of PCIe Gen 4 connectivity to each compute node, enabling each server to access a PCIe node, housing either 4 HHHL GPUs (for example, L4) or 2 FHFL GPUs (for example, L40S). Figure 11 shows the X-fabric modules and the connectivity between each compute node and the x440p PCIe node in the adjacent slot.

Connectivity Design

The hardware components and connectivity between them are shown in Figure 12.

●    A Cisco UCS 9508 chassis with at least 5 x Cisco UCS X210 M7 servers are connected using 100GbE IFM modules to Cisco UCS Fabric Interconnects, deployed in Intersight Managed Mode (IMM). 4 x 100GbE links from each IFM connect are bundled in a port-channel and connected to each Fabric Interconnect to provide an aggregate bandwidth of 800Gbps to the chassis with up to 8 UCS compute servers.

●    Two Cisco Nexus 93600CD-GX 100/400GbE Switches in Cisco NX-OS mode provide top-of-rack switching. The fabric interconnects use multiple 100GbE links to connect to the Nexus switches in a VPC configuration.

●    At least two 100 Gigabit Ethernet ports from each FI, in a port-channel configuration are connected to each Nexus 93600CD-GX switch.

●    One NetApp AFF C800 HA pair connects to the Cisco Nexus 93600CD-GX Switches using two 100 GE ports from each controller configured as a Port-Channel.

●    The high-performance servers are deployed as OpenShift compute nodes and booted using the Assisted Installer deployed RHCOS image on local M.2 boot drives in a RAID1 configuration. The persistent storage volumes are provisioned on the NetApp AFF C800 and accessed using NFS NAS storage,  iSCSI and NVMe-TCP storage.  ONTAP S3 object storage is also provisioned to store OpenShift AI models, artifacts, and other data.

●    Two of the Cisco UCS X210C M7 servers in Cisco UCS X9508 Chassis are paired with Cisco UCS X440p PCIe nodes, with each PCIe node housing 2 x NVIDIA L40S GPUs.

●    Each UCS M7 server is equipped with a Cisco VIC 15231 that provides 2 x 100GbE ports for 200Gbps of bandwidth from each server to the chassis.

Logical Design

Cisco UCS Server Networking Design

Each server is deployed with multiple virtual NICs (vNICs) and VLANs using Cisco Intersight server profiles as shown in Figure 13. Different vNIC configurations are used on OpenShift control and worker nodes.

 

The vNICs configuration on worked nodes are for power management, OpenShift cluster networking and storage access as outlined below: 

●    IB-MGMT (Optional): One vNIC and VLAN with fabric failover enabled on UCS Fabric Interconnects (FI) for in-band power management (for example, IPMI). Alternatively, you can CLUSTER-MGMT vNIC for this.

●    CLUSTER-MGMT: One vNIC and VLAN with fabric failover enabled on UCS Fabric Interconnects for all OpenShift cluster networking. This includes both pod and machine networks. The default cluster networking in OpenShift, i.e. Open Virtual Networking (OVN) is used in this solution.

●    iSCSI-A, iSCSI-B: Two vNICs and VLANs, one path through each UCS FI for iSCSI storage access.

●    NVMe-TCP-A, NVMe-TCP-B: Two VLANs, one path through each UCS FI for NVMe over TCP storage access. These are tagged VLANs using iSCSI-A and iSCSI-B vNICs.

●    NFS: One vNIC and VLAN for accessing NFS filesystems, with fabric failover enabled on UCS FIs.

●    OBJ: One vNIC and VLAN for accessing S3-compatible object store hosted on NetApp storage, with fabric failover enabled on UCS FIs.

The OpenShift Control node servers in the design are only provisioned for the first two vNICs in the above list. All vNICs are configured using vNIC templates. The vNIC Template used for control (-C) and worker (-W) nodes are shown in Figure 14.

The vNIC configuration on a given UCS server worker node, derived from the above worker node (-W) templates are shown in the figure below. The vNICs name includes the OpenShift interface numbering which starts with ‘eno5’.

Storage Design

The storage is set up with NFS, iSCSI, NVMe-TCP, and S3 protocols, each having VLAN and LIF (see Figure 15). NetApp Trident will handle dynamic storage orchestration for Red Hat OpenShift workloads. Trident can be configured with various storage backends based on requirements. For this solution, NAS, and SAN backends were used. Trident will provision PVCs based on the configured storage class and map them to the containers.

OpenShift AI requires S3-compatible object stores to store artifacts, logs, immediate results used by data science pipelines, and single- or multi-model serving platforms for deploying stored models. ONTAP S3 is configured to provide object stores and is directly presented to OpenShift AI for use in this solution. Following NetApp’s best practices, S3 LIFs are configured on both nodes.

OpenShift Design

Most machine learning (ML) models, frameworks, and test applications from popular sources like Hugging Face and NVIDIA GPU Cloud (NGC) are typically available as pre-packaged containers. The AI/ML ecosystem has also embraced Kubernetes, making containers the primary environment for the development and deployment of AI/ML workloads.

Given these factors and the need for an enterprise-class platform, this solution leverages OpenShift to offer enterprises a secure, robust Kubernetes orchestration and container management platform for developing, deploying, and managing cloud-native applications and ML workloads. By using OpenShift as the foundation for ML model delivery, including applications that integrate production ML models, enterprises can benefit from a unified platform for their AI/ML initiatives.

By combining OpenShift with OpenShift AI (see next section), OpenShift uniquely simplifies and accelerates projects with clear separation of functions. OpenShift administrators can continue to manage Kubernetes infrastructure administration, including provisioning GPUs, role-based access control, and resource utilization, while ML engineers can focus on ML delivery without worrying about the underlying infrastructure.

Red Hat offers multiple options for deploying OpenShift clusters, including both on-premises and SaaS solutions for connected and disconnected environments. These deployments can be managed using Advanced Cluster Management (ACM) or the cloud-based Red Hat Hybrid Cloud Console (HCC). In this solution, Red Hat OpenShift is deployed as a self-managed service using the Red Hat-recommended Assisted Installer from the Hybrid Cloud Console.

Red Hat recommends using the Assisted Installer for several reasons – one advantage being that it eliminates the need for a separate bootstrap machine during installation. Instead, the Assisted Installer uses one of the cluster nodes to manage the bootstrapping of other nodes in the cluster. Additionally, the Assisted Installer offers REST APIs that can be used to automate the installation process.

OpenShift is deployed on Cisco UCS-X baremetal servers, which are also managed from the cloud using Cisco Intersight. Red Hat Assisted Installer provides a discovery image (minimal image or full ISO) that must be downloaded and installed on the baremetal servers.  

Cisco and Red Hat provides an integration that allows the discovery ISO to be deployed directly on Cisco UCS servers provisioned in Cisco Intersight by clicking on the direct link to Intersight as shown below:

For a high-availability cluster, OpenShift typically requires 3 control nodes and two or more compute (or worker) nodes. The initial cluster used for validation is shown in Figure 17. Additional nodes can be added as needed to scale the cluster.

All network interfaces are provisioned using DHCP – DHCP requests are either routed through the gateway using DHCP relay or directly on the same segment (for storage vNICs) in this design. DNS, DHCP and NTP must all be setup prior to starting the installation.

Post-install, BMC access (for example, IPMI), NTP, etcd backup, storage and other post-install activities will need to be provisioned as needed. 

To support AI/ML efforts, the two compute/worker nodes are paired with Cisco UCS X440p nodes in adjacent slots, each equipped with NVIDIA L40S GPUs. These GPUs can be used by workloads running on OpenShift AI or directly on the OpenShift cluster.

When using GPUs, if not all worker nodes in the cluster are GPU-enabled, it is important to provision taints and tolerations on the nodes and workloads. This ensures that only workloads requiring GPUs are scheduled on the GPU-equipped nodes.

Workloads:

Nodes:

   

OpenShift Networking Design

OpenShift networking enables communication between various components both internal and external to the cluster.

Control Plane nodes and worker nodes, connect to two networks; OVN-Kubernetes that OpenShift manages and then the physical datacenter network.

For communication outside the cluster, the virtual NICs provisioned on UCS server nodes provide connectivity to the physical data center network.

Table 4.        Cisco UCS Server and OpenShift Network Mapping

For connectivity within the cluster, OpenShift uses Software-Defined Networking (SDN) to create overlay networks to interconnect pods and services across the cluster. The default networking in OpenShift is Open Virtual Networking – Kubernetes (or OVN-Kubernetes). OVN-Kubernetes is an open-source project that provides networking for Kubernetes clusters with OVN (Open Virtual Networking) and Open vSwitch (Open Virtual Switch) in its core architecture. It is plug-in, specifically designed for Kubernetes, and conforms to Kubernetes Container Network Interface (CNI) specifications.

Figure 18 illustrates the OpenShift networking on control and compute nodes in this solution.

CLUSTER-MGMT network is the cluster or machine network that all control and worker nodes are connected to. Overlay networks are created on this network for pods and services connectivity.

IB-MGMT network is included to provide independent management connectivity to the nodes. This network is used in this solution for management function, for example IPMI access and for loading Red Hat CoreOS to the servers during the install process.

Additional storage network interfaces and VLANs provide connectivity to NetApp storage – either directly or through NetApp Trident (file, block) and S3 compatible object store, also on NetApp storage.

For cluster networking (CLUSTER-MGMT), OVN architecture provides two key components, OVN controller and OVS virtual switch, which are deployed on each node to manage the networking, packet forwarding and policies.  OVN configures the OVS on each node to implement the declared network configuration. OVN uses the Geneve (Generic Network Virtualization Encapsulation) protocol to create overlay network between nodes.

The post-install and once DHCP has successfully provisioned the interfaces, each worker node should have a configuration as shown in Figure 19.

By default, Kubernetes (and OpenShift) allocates each pod an internal cluster-wide IP address that it can use for Pod-to-Pod communication using the Pod network. Within a Pod, all containers behave as if they’re on the same logical host and communicate with each other using localhost, using the ports assigned to the containers. For services, OpenShift, as in Kubernetes, exposes services using an internal stable IP address from within the cluster. This internal IP address, known as the ClusterIP, is type of service that allows other pods within the same cluster to communicate with the service without exposing it to the external network.

For communication outside the cluster, OpenShift provides services (node ports, load balancers) and API resources (Ingress, Route) to expose an application or a service outside cluster so that users can securely access the application or service running on the OCP cluster.

OpenShift Operators

Operators are a powerful tool in Kubernetes. It was designed to extend the capabilities of a Kubernetes cluster without changing the core Kubernetes code. Once a cluster is deployed, Red Hat OpenShift operators can be deployed to enable persistent storage, GPU acceleration and other services. A library of certified and community operators are available on Red Hat’s OperatorHub that is directly accessible from the cluster console. The operators deployed for this solution are shown in Figure 20.

NVIDIA GPU Operator

The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision and monitor GPUs. These components include:

●    NVIDIA drivers (to enable CUDA)

●    Kubernetes device plugin for GPUs

●    NVIDIA Container Runtime

●    Automatic node labeling

●    NVIDIA DCGM exporter

The GPU operator is responsible for enabling GPU acceleration on UCS-X worker nodes with NVIDIA GPUs on X440p GPU nodes. The NVIDIA GPU operator also requires Red Hat’s Node Feature Discovery Operator to detect the GPUs assigned to the worker node.

Red Hat Node Feature Discovery Operator

The Node Feature Discovery Operator (NFD) is responsible for detecting hardware capabilities and labeling the nodes with the hardware-specific information so that OpenShift cluster can use them. For NVIDIA GPUs on Cisco UCS worker nodes, the NFD Operator detects and labels it using the following label:

NetApp Trident Operator

NetApp Trident is an open-source storage provisioner and orchestrator maintained by NetApp. It enables you to create storage volumes for containerized applications managed by Docker and Kubernetes. It has been designed to help you meet your containerized application's persistence demands using industry-standard interfaces, such as the Container Storage Interface (CSI). For the release information, including patch release changes, see https://docs.netapp.com/us-en/trident/trident-rn.html.

Red Hat OpenShift AI Operator

Red Hat OpenShift AI operator deploys OpenShift AI on the OpenShift cluster that enables a fully supported environment for MLOps. The OpenShift AI environment deployed by the operator provides a core environment with built-in tools, libraries, and frameworks that ML engineers and data scientists need to train and deploy models. The GPU resources deployed on the OpenShift cluster are automatically available from the OpenShift AI UI and can be used as needed during various stages of model delivery ( for example, GPUs can be assigned to a Jupyter notebook for use in model experimentation). OpenShift AI includes project workspaces to enable multiple AI/ML efforts in parallel, Jupyter Notebooks with different built-in images to pick from (for example, PyTorch, TensorFlow, CUDA), Data Science Pipelines using OpenShift pipelines, model serving using ModelMesh (and Kserve) with Intel OpenVINO inferencing server. Customers can extend this environment by adding custom images, and other partner and open-source technologies. By using the operator framework, it is also simple to use the lifecycle OpenShift AI. 

MLOps using Red Hat OpenShift AI

Red Hat OpenShift includes key capabilities to streamline and scale machine learning operations (MLOps) in a consistent way. By applying DevOps and GitOps principles, organizations automate and simplify the iterative process of integrating ML models into software development processes, production rollout, monitoring, retraining, and redeployment to ensure continued prediction accuracy.

Red Hat OpenShift AI leverages OpenShift’s capabilities in application development and container infrastructure management to enable a robust, scalable, and secure environment for model delivery and MLOps. OpenShift Administrators manage all aspects of the underlying infrastructure, from GPU resources to storage to user access. This eases the operational burden on ML engineers and data scientists, enabling them to focus on model delivery and less time on managing the infrastructure. OpenShift also provides integration with DevOps capabilities (for example, OpenShift Pipelines, OpenShift GitOps, and Red Hat Quay). Also, projects and workbenches deployed in OpenShift AI are projects (or namespaces) in OpenShift, enabling Openshift administrator to monitor and manage the resources in their environment. These operational benefits make it significantly easier for Enterprise teams, enabling them to accelerate their ML efforts .

At a high level, a ML lifecycle can be summarized as follow. Red Hat OpenShift AI provides a unified platform for each of the above stages, along with the required AI/ML tools and applications.

●    Gather and prepare (or curate) data  to make sure the input data is complete, and of high quality

●    Develop model, including training, testing, and selection of the model with the highest prediction accuracy

●    Integrate models in application development process, and inferencing

●    Model monitoring and management, to measure business performance and address potential production data drift

Multiple efforts can run in parallel within OpenShift AI, from incubation projects to production serving of multiple models at scale. Red Hat OpenShift AI platform provides key capabilities to support this and accelerate model delivery as outlined below. 

●    Seamlessly leverage resources and capabilities from the underlying OpenShift cluster (for example, use OpenShift Identity provider to manage users).

●    Support for multiple Data Science Projects to enable parallel AI/ML efforts, including multiple workflows (known as workbenches) within a project.

●    Support for multiple work efforts or Workbenches within a given data science project to support parallel work efforts within the same projects. A workbench is an isolated area where you can work with models in your preferred IDE, such as a Jupyter notebook. You can add accelerators and data connections, create pipelines, and add cluster storage in your workbench. The workbenches can be launched with pre-built or custom images with necessary libraries and frameworks.

●    The pre-built image options available in the release of OpenShift AI used in this design include commonly used images such as:  Minimal Python, Standard Data Science, CUDA, PyTorch, TensorFlow, TrustyAI, Habana AI, and Code-server.

●    Other notebook options you can select from include:

◦     Container size (Small, Medium, Large,  and X Large) based on memory and CPU requirements for project

◦     Number of GPU accelerators (optional)

◦     Persistent Storage – new or existing (provided by NetApp Trident in this solution)

◦     Data Connection to access S3-compatible storage on NetApp ONTAP storage on-prem

●    If GPU acceleration is selected, OpenShift AI will detect and make the GPU available for use. The pre-built images that support GPU acceleration will also be updated to indicate that it is available s shown below. Otherwise, CPU resources will be used. Within a given data science project, the parallel efforts on different workbenches can individually select whether to use GPU or CPU resources.

●    For model serving using pre-integrated Intel OpenVINO inferencing server or use a custom server such as NVIDIA Triton. For model serving, you can specify the model repository where the model is stored, the format or framework the published model uses (for example, onnx, tensorflow, openvino_ir) as well as the number of GPU accelerators to use.

●    Simple drag and drop GUI based Pipeline Automation with options to schedule execution runs.

Red Hat OpenShift AI provides flexibility and scalable platform for an Enterprise AI/ML initiatives by providing pre-integrated both pre-integrate and customizable environments without compromising on flexibility, in an easy-to-use interface with automation, Jupyter notebooks, GitHub access, and multiple storage and database options  The scalability of the solution will primarily depend on underlying OpenShift infrastructure and environment.  

End-to-End Design

The FlexPod AI solution with MLOps is now capable of supporting a wide range of ML model delivery efforts, including both Predictive AI and Generative AI, single-modal and multi-modal, as well as CPU and GPU-based use cases with flexible inferencing engines and runtimes. The solution offers a foundational infrastructure platform to simplify, streamline and expedite an enterprise's AI/ML initiatives. The comprehensive design for the solution is shown in Figure 21.

Once the models are ready, they can be served and monitored through OpenShift AI. Enterprise applications can leverage this environment for everything from experimentation to production, seamlessly integrating their applications with the served models to support a variety of use cases as illustrated in Figure 22.

Solution Deployment

This chapter contains the following:

●    Deployment Overview

●    Deploy Networking – Cisco Nexus

●    Deploy Storage - NetApp ONTAP

●    Deploy Compute – Cisco UCS

●    Deploy Kubernetes – OpenShift on Baremetal UCS Servers

●    Deploy NetApp Trident Operator

●    Deploy GPU Operator - NVIDIA

●    Deploy Red Hat OpenShift AI for MLOps

●    Visibility and Monitoring – GPU

This chapter provides an high-level overview of the implementation steps for deploying the solution, followed by step-by-step guidance for implementing each step in the overall solution.

Deployment Overview

The AI/ML infrastructure leverages the latest FlexPod Datacenter CVD for Cisco UCS M7 baremetal servers running Red Hat OpenShift as the foundational design for containerized AI/ML workloads running on Red Hat OpenShift and MLOps for model deployment and maintenance using Red Hat OpenShift AI.

Check Cisco UCS Hardware Compatibility List for NVIDIA GPU support on Cisco UCS running Red Hat OpenShift and upgrade UCS server firmware as needed.

With the FlexPod Datacenter infrastructure in place, an overview of the remaining deployment steps to bring up the AI/ML infrastructure for model serving in production with MLOps are summarized in Table 5.

The detailed procedures for the steps listed in this table will be available on GitHub in the future: https://github.com/ucs-compute-solutions/FlexPod-OpenShift-AI

Table 5.        Deployment Overview

Deploy Networking – Cisco Nexus

The procedures detailed in this section will configure the necessary networking on Cisco Nexus 9000 series switches to enable connectivity between the different components in the solution. This section explains the following:

●    Initial setup of Cisco Nexus 9000 series Top-of-Rack (ToR) switches.

●    Provision Cisco Nexus switches to enable the connectivity that infrastructure and workloads running on the infrastructure need. This includes connectivity between UCS domains and NetApp storage directly connected to the switches, as well as upstream connectivity to other networks, both within the enterprise and externally.

Initial Setup of Cisco Nexus 9000 Series ToR Switches

This section describes the initial setup of a pair of Cisco Nexus 9000 series Top-of-Rack (ToR) switches used in the  solution.

Assumptions and Prerequisites

●    Assumes a greenfield deployment with new Nexus switches that have not been configured.

●    Console access to both Cisco Nexus switches.

●    Collect the setup information for your environment – see Table 6.

Setup Information

Table 6 lists the setup parameters and other information necessary for the procedures in this section. The information also includes access information for devices used in the configuration. 

Table 6.        Cisco Nexus: Initial Setup Parameters and Information

Deployment Steps

Use the setup information in Table 6 for the procedures detailed in this section.

Procedure 1.     Initial setup of the Nexus-A switch

Step 1.          Connect to the console port of the first Nexus switch (Nexus-A) .

Step 2.          Power on the switch.

Step 3.          Click Yes to abort PoAP and continue with normal setup.

Step 4.          Specify an admin password to access the switch.

Step 5.          Click Yes to enter the basic configuration dialog.

Step 6.          Specify a switch name, Mgmt0 IPv4 address, netmask, and default gateway. You can choose the default options for everything else or modify it as you see fit for your environment.

Step 7.          Review and Save to use the specified configuration.

Procedure 2.     Initial setup of the Nexus-B switch

Step 1.          Repeat the configuration steps in Procedure 1 to set up the Nexus-B switch.

Step 2.          Review the configuration. Confirm to use the configuration and Save it.

Provision Cisco Nexus Switches to enable workload and infrastructure connectivity

The procedures detailed in this section configure the Nexus 9000 series switches in the solution to provide the necessary infrastructure and workload connectivity. 

Assumptions and Prerequisites

●    The initial setup of the Nexus switches is complete.

●    Collect the setup information for your environment – see Table 7.

Setup Information

The setup parameters and other information necessary to configure the components in this section, including access information are listed in Table 7. 

Table 7.        Cisco Nexus: Setup Parameters and Information

Deployment Steps: Nexus-A Switch

Use the setup information listed in Table 7 to configure and deploy the Nexus-A switch.

Procedure 1.     Configure and deploy the Nexus-A switch

Step 1.          Log in and enable global features on Nexus-A switch.

Step 2.          SSH into Nexus-A switch and log in as admin.

Step 3.          Enter the configuration mode on Nexus-A switch:

Step 4.          Run the following commands to enable the following features. Some (for example, sftp-server) are optional:

Procedure 2.     Configure DNS, NTP, Clock, and other global configurations on the Nexus-A switch

Step 1.          From the configuration mode on the Nexus-A switch, run the following commands:

Procedure 3.     Configure the FlexPod VLANs on the Nexus-A switch

Step 1.          From the configuration mode on the Nexus-A switch, create the in-band management VLAN. Configure the interface VLAN and Gateway IP if you’re using this switch as default GW:

Step 2.          From the configuration mode on Nexus-A, provision all remaining FlexPod VLANs. This typically includes the native VLAN, OpenShift Cluster management VLAN, and storage VLANs:

Procedure 4.     Configure the virtual Port Channel (vPC) domain and peer-links on the Nexus-A switch

Step 1.          Configure vPC domain and peer keepalives:

Step 2.          Configure vPC peer-links:

Procedure 5.     Configure the upstream connectivity from Nexus-A switch to the enterprise data center network

Step 1.          Configure the VPCs from the Nexus-A switches to upstream switches in the enterprise data center network for connectivity to other parts of the enterprise and external networks:

Procedure 6.     Configure the vPC connectivity from Nexus-A to Cisco UCS compute infrastructure

Step 1.          Configure the first vPC to Cisco UCS compute infrastructure (Fabric Interconnect A). Use (show cdp | lldp neighbors) to verify interface and neighbor connectivity as needed:

Step 2.          Configure the second vPC to Cisco UCS compute infrastructure (Fabric Interconnect B). Use (show cdp | lldp neighbors) to verify interface and neighbor connectivity as needed. See the Note in Step 1 regarding udld.

Procedure 7.     Verify the vPC is up and operational

Step 1.          Run the following command to verify the vPC is operational:

Procedure 8.     Configure the interfaces from Nexus-A to NetApp Storage

Step 1.          Configure the first interface to NetApp Storage:

Step 2.          Configure the second interface to NetApp Storage:

Step 3.          Run the following to save the configuration:

Deployment Steps: Nexus-B Switch

Use the setup information found here to configure and deploy the Nexus-B switch.

Procedure 1.     Log in and enable global features on the Nexus-B switch

Step 1.          SSH into Nexus-B switch and log in as admin.

Step 2.          Enter configuration mode on Nexus-B switch.

Step 3.          Enter configuration mode on Nexus-A switch.

Step 4.          Run the following commands to enable the following features. Some (for example, sftp-server) are optional:

Procedure 2.     Configure the DNS, NTP, Clock, and other global configurations on the Nexus-B switch

Step 1.          From the configuration mode on the Nexus-B switch, run the following commands:

Procedure 3.     Configure FlexPod VLANs on the Nexus-B switch

Step 1.          From the configuration mode on the Nexus-B switch, create the in-band management VLAN. Configure the interface VLAN and Gateway IP if you’re using this switch as the default GW:

Step 2.          From the configuration mode on the Nexus-A switch, provision all remaining FlexPod VLANs. This includes the native VLAN, OpenShift Cluster management VLAN, and storage VLANs:

Procedure 4.     Configure the virtual Port Channel (vPC) domain and peer-links on the Nexus-B switch

Step 1.          Configure the vPC domain and peer keepalives:

Step 2.          Configure the vPC peer-links:

Procedure 5.     Configure the upstream connectivity from the Nexus-B switch to the enterprise data center network

Step 1.          Configure the VPCs from the Nexus-A switches to upstream switches in the enterprise data center network for connectivity to the other parts of the enterprise and external networks.

Procedure 6.     Configure the vPC connectivity from the Nexus-B switch to the Cisco UCS compute infrastructure

Step 1.          Configure the first vPC to Cisco UCS compute infrastructure (Fabric Interconnect A). Use (show cdp | lldp neighbors) to verify interface and neighbor connectivity as needed:

Step 2.          Configure the second vPC to Cisco UCS compute infrastructure (Fabric Interconnect B). Use (show cdp | lldp neighbors) to verify the interface and neighbor connectivity as needed. See the Note in Step 1 regarding udld.

Procedure 7.     Verify the vPC is up and operational

Step 1.          Run the following command to very the vPC is operational:

Procedure 8.     Configure interfaces from Nexus-B to NetApp Storage

Step 1.          Configure the first interface to NetApp Storage:

Step 2.          Configure the second interface to NetApp Storage:

Step 3.          Run the following to save the configuration:

Deploy Storage - NetApp ONTAP

The procedures detailed in this section configure the NetApp ONTAP Storage for the OpenShift cluster.

Assumptions and Prerequisites

●    Assumes that the base storage setup is completed and the ONTAP license has been added to the cluster. To do the base storage setup, follow the steps in section NetApp ONTAP Storage Configuration of the FlexPod Datacenter Base Manual Configuration with Cisco IMM and NetApp ONTAP Deployment Guide.

●    Access to NetApp storage.

●    Collect the setup information for your environment – see Table 8.

Table 8.        NetApp Storage: Setup parameters and information

Deployment Steps

Procedure 1.     Create interface groups and set MTU

Step 1.          Open an SSH connection using the cluster IP and log in as admin user.

Step 2.          To create LACP interface groups for the 100GbE data interfaces, run the following commands:

Step 3.          To change the MTU size on the interface-group ports, run the following commands:

Procedure 2.     Configure ONTAP Storage for the OpenShift Cluster

Step 1.          Create an IPspace for the OpenShift tenant:

Step 2.          Create the AC10-OCP-MGMT, AC10-OCP-NFS, AC10-OCP-iSCSI, AC10-OCP-NVMe-TCP and AC10-OCP-S3 broadcast domains with the recommended maximum transmission unit (MTU):

Step 3.          Create the OCP management VLAN ports and add them to the OCP management broadcast domain:

Step 4.          Create the OCP NFS VLAN ports and add them to the OCP NFS broadcast domain:

Step 5.          Create the OCP iSCSI VLAN ports and add them to the OCP iSCSI broadcast domains:

Step 6.          Create the OCP NVMe-TCP VLAN ports and add them to the OCP NVMe broadcast domains:

Step 7.          Create the OCP S3 VLAN ports and add them to the OCP S3 broadcast domain:

Step 8.          Create the SVM (Storage Virtual Machine) in IPspace. Run the vserver create command:

Step 9.          Add the required data protocols to the SVM and remove the unused data protocols from the SVM:

Step 10.       Add the two data aggregates to the AC10-OCP-SVM aggregate list and enable and run the NFS protocol in the SVM:

Step 11.       Create a service policy for the S3 object store:

Step 12.       Create a Load-Sharing Mirror of the SVM Root Volume. Create a volume to be the load-sharing mirror of the infrastructure SVM root volume only on the node that does not have the Root Volume:

Step 13.       Create the 15min interval job schedule:

Step 14.       Create the LS mirroring relationship:

Step 15.       Initialize and verify the mirroring relationship:

Step 16.       (Optional) To create the log in banner for the SVM, run the following command:

Step 17.       Create a new rule for the SVM NFS subnet in the default export policy and assign the policy to the SVM’s root volume:

Step 18.       Create and enable the audit log in the SVM:

Step 19.       Run the following commands to create NFS Logical Interfaces (LIFs):

Step 20.       Run the following commands to create iSCSI LIFs:

Step 21.       Run the following commands to create NVMe-TCP LIFs:

Step 22.       Run the following commands to create S3 LIFs:

Step 23.       Run the following command to create the SVM-MGMT LIF:

Step 24.       Run the following command to verify LIFs:

Step 25.       Create a default route that enables the SVM management interface to reach the outside world:

Step 26.       Set a password for the SVM vsadmin user and unlock the user:

Step 27.       Add the OCP DNS servers to the SVM:

Procedure 3.     Configure ONTAP S3 Bucket

Step 1.          Go to Storage > Storage VMs> and click the SVM and go to Settings.

Step 2.          Go to Protocol and click the Setting icon to create the S3 server and provide the details, including certificate expiration period and then click Save.

Step 3.          Go to Storage > Buckets and click Add.

Step 4.          Provide the name of the bucket, choose SVM and the size of the bucket. Click Save.

Step 5.          Create one more bucket for storing pipeline artifacts.

Procedure 4.     Configure S3 user and group

Step 1.          Go to Storage > Storage VMs. Choose the AC10-OCP-SVM, click Settings and then click  under S3.

Step 2.          To add a user, click Users > Add.

Step 3.          Enter a name for the user.

Step 4.          Click Save. The user is created and an access key and a secret key are generated for the user.

Step 5.          To add a group, click Groups and click Add.

Step 6.          Enter a group name and choose the user from the list created earlier.

Step 7.          Choose the FullAccess policy from the list and click Save.

Step 8.          Repeat this procedure to create additional S3 buckets for Pipeline Artifacts, and so on.

Deploy Compute – Cisco UCS

The procedures detailed in this section provision the Cisco UCS compute infrastructure in the solution. This section details the following:

●    Initial setup of Cisco UCS Fabric Interconnects in Intersight Managed Mode (IMM).

●    Initial setup of Cisco Intersight for managing the UCS servers in this solution.

●    Configure UCS domain using a Domain Profile derived from a UCS Domain Profile Template. A UCS domain is defined by a pair of Fabric Interconnects and all servers connected to it.

●    Configure UCS X-Series chassis using a Chassis Profile derived from a UCS Chassis Profile Template.

●    Configure UCS Servers in the chassis using Server Profiles derived from a UCS Server Profile Template.

Initial Setup of Cisco UCS Fabric Interconnects

This section describes the initial setup of a pair of Cisco UCS Fabric Interconnects used in the solution.

Assumptions and Prerequisites

●    Assumes a greenfield deployment with new Cisco UCS Fabric Interconnects that have not been configured.

●    Networking and Storage should be set up prior to deploying the UCS compute infrastructure.

●    Console access to Cisco UCS Fabric Interconnects.

●    Collect the setup information for your environment – see Table 9.

Setup Information

Table 9 lists the setup parameters and other information necessary to do the configuration in this section. The information includes access information for management systems and devices being configured in addition to the parameters.

Table 9.        Cisco UCS Fabric Interconnects: Setup Parameters and Information

Deployment Steps

Use the setup information provided in Table 9 to do the initial setup of Cisco UCS Fabric Interconnects.

Procedure 1.     Initial setup of the first UCS Fabric Interconnect in the UCS Domain

Step 1.          Connect to the console port on the first Cisco UCS Fabric Interconnect.

Step 2.          Power on the Fabric Interconnect. You will see the power-on self-test messages as the Fabric Interconnect boots.

Step 3.          When the unconfigured system boots, it prompts you for the setup method to be used. Enter console to continue the initial setup using the console CLI.

Step 4.          Enter the management mode for the Fabric Interconnect as intersight to manage the Fabric Interconnect strictly through Cisco Intersight. Also known as Intersight Managed Mode (IMM). Alternate option is ucsm to manage the Fabric Interconnects using Cisco UCS Manager (and Cisco Intersight) but this is different from managing it in IMM mode.

Step 5.          Enter y to confirm that you want to continue the initial setup.

Step 6.          Enter the password for the admin account. To use a strong password enter y.

Step 7.          To confirm re-enter the password for the admin account.

Step 8.          Enter yes to continue the initial setup for a cluster configuration.

Step 9.          Enter the Fabric Interconnect fabric (either A or B ).

Step 10.       Enter system name (hostname for the FIs without -A or -B in the name since it will automatically get added to distinguish between FI-A and FI-B).

Step 11.       Enter the IPv4 or IPv6 address for the management port of the Fabric Interconnect.

Step 12.       Enter the IPv4 subnet mask or IPv6 network prefix.

Step 13.       Enter the IPv4 or IPv6 default gateway.

Step 14.       Enter the IPv4 or IPv6 address for the DNS server. The address matches the address type of the management IP.

Step 15.       Enter yes if you want to specify the default Domain name, or no if you do not.

Step 16.       (Optional) Enter the default Domain name.

Step 17.       Review the setup summary and enter yes to save and apply the settings or enter no to go through the setup again to change some of the settings.

Procedure 2.     Initial setup of the second UCS Fabric Interconnect in the UCS Domain

Step 1.          Connect to the console port on the second Cisco UCS Fabric Interconnect.

Step 2.          Power on the Fabric Interconnect. You will see the power-on self-test messages as the Fabric Interconnect boots.

Step 3.          When the unconfigured system boots, it prompts you for the setup method to be used. Enter console to continue the initial setup using the console CLI.

Step 4.          Enter y to add Fabric Interconnect-A to the cluster.

Step 5.          Enter the admin password for Fabric Interconnect A.

Step 6.          Enter the IPv4 or IPv6 address for the management port of the local Fabric Interconnect A.

Step 7.          Review the setup summary and enter yes to save and apply the settings or enter no to go through the setup again to change some of the settings.

Initial Setup of Cisco Intersight to manage the Cisco UCS Servers in this solution

The procedures described in this section will provision the following: 

●    Create a Cisco Intersight account.

●    Setup Licensing in Cisco Intersight.

●    Configure a Resource Group for the servers in the solution.

●    Configure an Organization for the servers in the solution.

●    Claim Cisco UCS Fabric Interconnects in Cisco Intersight.

●    Upgrade Firmware on Cisco UCS Fabric Interconnects.

Assumptions and Prerequisites

●    You should have a valid cisco.com account.

●    You should have Smart Licensing enabled with licenses available in a virtual account for Intersight.

●    IP Access to Cisco UCS Fabric Interconnects.

●    Assumes that a new organization and resource group is being provisioned in Intersight for this solution.

Setup Information

Table 10 lists the setup parameters and other information necessary to do the configuration in this section.

Table 10.     Cisco Intersight: Setup Parameters and Information

Deployment Steps

Use the setup information listed in Table 10 to configure Intersight to manage the UCS compute infrastructure in the solution.

Procedure 1.     Create a Cisco Intersight account 

Step 1.          Use a web browser to navigate to https://intersight.com/.

Step 2.          Click Create an account.

Step 3.          From the drop-down list and choose the Region (for example, US East | EU Central ) to specify where to host account.

Step 4.          Click Next.

Step 5.          Read and accept the General Terms license agreement. Click Next.

Step 6.          Provide an Account Name. Click Create.

Step 7.          In the Licensing window, choose Register Smart Licensing if you have purchased Intersight licenses.

Step 8.          Using a separate browser window, collect the Product Instance Registration Token from Smart Software Licensing Portal on cisco.com. Once logged into the portal, click the Inventory tab, and then choose your Virtual Account from the drop-down list. 

Step 9.          Click New Token and copy the newly generated token or copy an existing token from the list.

Step 10.       Paste the token into the Product Instance Registration Token box. Intersight will now attempt to license and register the newly created account with Cisco Smart Licensing Portal. This may take a few minutes. A window displays stating it was successful.

Step 11.       Click Next.

Step 12.       In the Subscription Information window, choose Enable Subscription Information.

Step 13.       Click Next.

Step 14.       In the Products window, choose the radio buttons for the Intersight services you want to enable.

Step 15.       (Optional) Choose a Default Tier from the drop-down list and check the box to Set Default Tier to all existing servers.

Step 16.       Click Proceed. In the window, click Confirm to proceed with the selected services.

The Intersight account is now ready for provisioning and management of Cisco UCS and other data center infrastructure.

Procedure 2.     Setup Licensing in Cisco Intersight

Step 1.          Log into the Cisco Smart Licensing portal: https://software.cisco.com/software/smart-licensing/inventory.

Step 2.          Confirm that you’re using the right account. The account name should be displayed in the top-right corner of the window.

Step 3.          Click the Inventory tab.

Step 4.          Choose the Virtual Account from the drop-down list.

Step 5.          From the General tab, click New Token to generate a new token in the Product Instance Registration Tokens section. You can also choose a pre-generated token from the list and copy that to license the Cisco Intersight service.

Step 6.          Review and click Proceed in the pop-up window regarding Cisco’s Product export control and sanction laws.

Step 7.          In the Create Registration Token pop-up window, provide a Description and click Create. Token.

Step 8.          Click the  to copy this newly created token and copy to clipboard.

Step 9.          Log into Intersight (intersight.com) and go to System > Licensing.

Step 10.       Click Smart Licensing Details and then click Register Smart Licensing.

Step 11.       In Register Smart Licensing wizard, paste the previously copied token into the Product Instance Registration Token box. Click Next. This process will take a few minutes to complete.

Step 12.        For Subscription Information, keep the default settings (Enable Subscription Information) and click Next.

Step 13.       For Products, click Infrastructure Service & Cloud Orchestrator. Choose a Default Tier from the drop-down list. Choose Set Default Tier to all existing servers. Click Proceed.

Step 14.       In the Confirm Products window, click Confirm.

Synchronizing Smart Licensing starts and takes a few minutes to complete.

If licensing completes successfully, you will see the following:

Procedure 3.     Create a Resource Group in Intersight

Step 1.          Go to System > Resource Groups.

Step 2.          Click the + Create Resource Group box in the top-right corner of the window.

Step 3.          Specify a Name and Description for the Resource Group (for example, FPB-OAI_RG).

Step 4.          Under Resources click Custom.

Step 5.          If the Cisco UCS Fabric Interconnects or servers have already been claimed, they can be added to the resource group by checking the box next to the claimed device – see the second screenshot below. If you’re adding a subset of servers managed by the UCS Fabric Interconnects, click the Edit icon in the Sub-Target column and choose the specific servers as shown in the first screenshot and then click Select. 

Step 6.          Click Create.

Procedure 4.     Create an Organization in Intersight

Step 1.          Go to System > Organizations.

Step 2.          Click the + Create Organization box in the top-right corner of the window.

Step 3.          In the wizard, under General, specify a Name and Description for the Organization (for example, FPB-OAI_ORG).

Step 4.          Click Next.

Step 5.          In the Configuration section, check the box for the previously created Resource Groups.

Step 6.          Click Next.

Step 7.          In the Summary window, verify the settings and click Create.

Procedure 5.     Claim Cisco UCS Fabric Interconnects in Cisco Intersight

Step 1.          Use a web browser to log in as admin to the management IP address of Fabric Interconnect A.

Step 2.          Click the Device Connector tab. The status should be Not Claimed.

Step 3.          Copy the Device ID and Claim Code to claim the device in Cisco Intersight.

Step 4.          Go back or log back into Cisco Intersight.

Step 5.          Go to System > Targets.

Step 6.          Click Claim a New Target.

Step 7.          Choose Cisco UCS Domain (Intersight Managed) and click Start.

Step 8.          In the Claim Cisco UCS Domain (UCSM Managed) Target window, paste the Device ID and Claim Code copied from Cisco UCS FI-A. Choose the previously created Resource Group (see the Note above).

Step 9.          Click Claim to claim the UCS FIs as a target in Cisco Intersight.

If the claim was successful, the Cisco UCS Fabric Interconnects display as Connected.

The Device Connector status on both FIs show the status as Claimed.

Procedure 6.     (Optional) Upgrade Firmware on Cisco UCS Fabric Interconnects

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Operate > Fabric Interconnects.

Step 3.          To upgrade the Fabric Interconnect pair, choose one of the fabric interconnects and click the elipses at the end of the row and choose Upgrade Firmware from the list.

Step 4.          In the Upgrade Firmware wizard, under General verify the Fabric Interconnect information.

Step 5.          Click Next.

Step 6.          Under Version, choose Advanced Mode to see the options available, such as enabling Fabric Interconnect Traffic Evacuation.

Step 7.          Choose a version from the list to upgrade and click Next.

Step 8.          Under Summary, verify the information and click Upgrade to start the upgrade process.

Step 9.          Click Upgrade again in the Upgrade Firmware pop-up window to confirm.

Step 10.       Monitor the upgrade from the Requests panel ( icon) since you will be prompted for permission to upgrade each FI. Follow the prompts on screen to grant permission.

The upgrade upgrades both FIs in the cluster.

Step 11.       Wait for the upgrade to complete before proceeding to the next step.

Configure the UCS Domain using a Domain Profile derived from a UCS Domain Profile Template

The procedures described in this section will create all policies that will be part of the UCS Domain Profile Template that will be used to derive the UCS Domain Profile. This includes:

●    Create VLAN Multicast Policy

●    Create VLAN Policies for FI-A and FI-B (policies for FI-A and FI-B are the same in this CVD)

●    Create VSAN Policies for FI-A and FI-B (not used in this CVD)

●    Create Ethernet Network Group Policies for FI-A and FI-B (policies for FI-A and FI-B are the same in this CVD)

●    Create Flow Control Policy

●    Create Link Aggregation Policy

●    Create Link Control Policy

●    Create Port Policies for FI-A and FI-B

●    Create NTP Policy

●    Create Syslog Policy

●    Create Network Connectivity Policy

●    Create SNMP Policy

●    Create LDAP Policy (Optional)

●    Create Certificate Management Policy (Optional)

●    Create System QoS Policy

●    Create Switch Control Policy

The policies are used to:

●    Create a UCS Domain Profile Template using the above policies

●    Derive a UCS Domain Profile from the UCS Domain Profile Template

●    Deploy the UCS Domain Profile to provision the Cisco UCS domain

Assumptions and Prerequisites

●    Complete the initial setup of Cisco UCS Fabric Interconnects with IP Management access.

●    Valid cisco.com and Intersight accounts.

●    Enable Cisco Smart Software Licensing on the Intersight account.

●    Name of Intersight Organization that the UCS domain will be part of.

●    Collect the setup information for your environment – see Table 11 for the required setup information.

Setup Information

Table 11.     Cisco Intersight: UCS Domain Profile Template

Deployment Steps

Use the setup information provided in this section to configure the policies for the UCS domain consisting of the two Cisco UCS Fabric Interconnects. The policies will be part of the UCS Domain Profile Template which will be used to derive the UCS Domain Profile to configure this specific UCS Domain.

The following procedures configure policies that are used in the VLAN & VSAN Configuration section of Create UCS Domain Profile Template wizard.

Procedure 1.     Create (VLAN) Multicast Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Platform Type – either All or UCS Domain.

Step 4.          Choose Multicast Policy.

Step 5.          Click Start.

Step 6.          From the Policies > Multicast Policy window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, choose the default settings:

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 2.     Create a VLAN Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Click VLAN.

Step 5.          Click Start.

Step 6.          From the Policies > VLAN window, in the General section for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, click Add VLANs.

Step 12.       In the Create VLAN window, specify a Name/Prefix and VLAN ID for the native VLAN. Keep the remaining defaults.

Step 13.       For Multicast Policy*, click Select Policy and choose the previously configured multicast policy.

Step 14.       Click Add to add the VLAN.

Step 15.       Repeat steps 11 – 14 to add the remaining VLANs including the Native VLAN if it is different from the default (vlan=1).

Step 16.       If using a non-default native VLAN, scroll down and check the box for Set Native VLAN ID and specify the VLAN ID for the native VLAN. This VLAN should be one of the previously created VLANs.

Step 17.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 3.     Create FI-A Ethernet Network Group Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Ethernet Network Group Policy.

Step 5.          Click Start.

Step 6.          From the Policies > Ethernet Network Group window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, click Add VLANs and choose From Policy from the drop-down list.

Step 12.       From the Select Policy window, choose the previously created VLAN policy.

Step 13.       Click Next.

Step 14.       You should now see all VLANs from the VLAN policy listed. Check the box to select all VLANs.

Step 15.       Click Select.

Step 16.       In the Policy Details window, choose the VLAN ID for native VLAN, click the elipses and choose Set Native VLAN.

Step 17.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 4.     Create FI-B Ethernet Network Group Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Ethernet Network Group.

Step 5.          Click Start.

Step 6.          From the Policies > Ethernet Network Group window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, click Add VLANs and choose From Policy from the drop-down list.

Step 12.       From the Select Policy window, choose the previously created VLAN policy.

Step 13.       Click Next.

Step 14.       You should now see all VLANs from the VLAN policy listed. Check the box for all VLANs.

Step 15.       Click Select.

Step 16.       In the Policy Details window, choose VLAN ID for native VLAN, click the elipses and choose Set Native VLAN.

Step 17.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 5.     Create Flow Control Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Flow Control.

Step 5.          Click Start.

Step 6.          From the Policies > Flow Control window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, keep the default settings.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 6.     Create Link Aggregation Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Link Aggregation.

Step 5.          Click Start.

Step 6.          From the Policies > Link Aggregation window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, keep the default settings.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 7.     Create Link Control Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Link Control.

Step 5.          Click Start.

Step 6.          From the Policies > Link Control window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, keep the default settings.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 8.     Create Port Policy for FI-A

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Port.

Step 5.          Click Start.

Step 6.          From the Policies > Port window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          For Fabric Interconnect Model, choose a model from the drop-down list.

Step 9.          (Optional) For Set Tags, specify value in key:value format.

Step 10.       (Optional) For Description, specify a description for this policy.

Step 11.       Click Next.

Step 12.       In the Unified Ports section, leave the slider as is since you are not configuring the Fibre Channel (FC) ports in this CVD. All ports in this FI will be Ethernet ports.

Step 13.       Click Next.

Step 14.       In the Breakout Options section, if using breakout ports, configure it now otherwise keep the default settings.

Step 15.       Click Next.

Step 16.       In the Port Roles section, click the Port Roles.

Step 17.       Scroll down and choose the server ports to configure. When the checkboxes for the relevant ports are selected, scroll up and click Configure.

Step 18.       In the Configure Ports window, for the Role, choose Server from the drop-down list.

Step 19.       For the remaining settings, keep the default settings.

Step 20.       Click Save.

Step 21.       In the Port Roles section, click the Port Channels.

Step 22.       Click Create Port Channel.

Step 23.       In the Create Port Channel window, choose Ethernet Uplink Port Channel from the drop-down list.

Step 24.       For the Port Channel ID, specify a unique ID; the ID is local to the FI. It does not need to match the ID on the switch to which it connects.

Step 25.       For Admin Speed, keep the default settings.

Step 26.       For FEC, keep the default settings.

Step 27.       For Ethernet Network Group, keep the default settings.

CAUTION!: If using Disjoint L2, click Select Policies and choose the previously configured policy.Only use this in a Disjoint L2 deployment.

Step 28.       For Flow Control, click Select Policy and choose the previously configured policy.

Step 29.       For Link Aggregation, click Select Policy and choose the previously configured policy.

Step 30.       For Link Control, click Select Policy and choose the previously configured policy.

Step 31.       In the Select Member Ports section, scroll down and check the box next to the uplink ports that should be part of this port channel.

Step 32.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 9.     Create Port Policy for FI-B

Step 1.          Repeat the steps in Procedure 8. Create Port Policy for FI-A to configure a port policy for FI-B.

 

Procedure 10.  Create NTP Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose NTP.

Step 5.          Click Start.

Step 6.          From the Policies > NTP window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, choose Enable NTP. 

Step 12.       Specify NTP server IP addresses and add more as needed. 

Step 13.       Choose a Time zone from the drop-down list.

Step 14.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 11.  Create a Syslog Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Syslog.

Step 5.          Click Start.

Step 6.          From the Policies > Syslog window, in the General section for Organization, choose previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, expand Local Logging > File. For Minimum Severity to Report, choose from the options in the drop-down list. 

Step 12.       For Remote Logging, enable Syslog Server 1 and provision the IP address, Port, Protocol and Minimum Severity To Report for the syslog server. Repeat this step if you’re using a second syslog server.

Step 13.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 12.  Create a Network Connectivity Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Network Connectivity.

Step 5.          Click Start.

Step 6.          From the Policies > Network Connectivity window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, specify Preferred IPv4 DNS Server and Alternate IPv4 DNS Server IP addresses. Keep the default setting for all fields.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 13.  Create a SNMP Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose SNMP.

Step 5.          Click Start.

Step 6.          From the Policies > SNMP window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, enable SNMP.

Step 12.       Choose the SNMP version and in the Configuration section, configure the parameters as needed: 

Step 13.       In the Add SNMP User section, add SNMP Users as needed.

Step 14.       In the Add SNMP Trap Destination section, add SNMP Trap destination as needed.

Step 15.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 14.  Create a System QoS Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose System QoS.

Step 5.          Click Start.

Step 6.          From the Policies > System QoS window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, specify the QoS Policies for Best Effort policy as shown below:

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 15.  Create a Switch Control Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Choose Platform Type – either All or UCS Domain.

Step 4.          Choose Switch Control.

Step 5.          Click Start.

Step 6.          From the Policies > Switch Control window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In Policy Details section, verify that Switching Mode > Ethernet is set to End Host Mode as shown below. Kepp the default settings.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 16.  Create a UCS Domain Profile Template: General

Step 1.          Use a web browser to navigate to intersight.com and log in to your account.

Step 2.          Go to Configure > Templates.

Step 3.          Choose UCS Domain Profile Templates.

Step 4.          Click Create UCS Domain Profile Template.

Step 5.          From the Create UCS Domain Profile Template window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 6.          For Name, specify a name for this policy.

Step 7.          (Optional) For Set Tags, specify value in key:value format.

Step 8.          (Optional) For Description, specify a description for this policy.

Step 9.          Click Next.

Procedure 17.  Create a UCS Domain Profile Template: VLAN & VSAN Configuration for FI-A and FI-B

Step 1.          For VLAN & VSAN Configuration, under Fabric Interconnect A, for VLAN Configuration, click Select Policy.

Step 2.          In the Select Policy window, choose the previously created VLAN Policy for FI-A (same as FI-B VLAN policy in this CVD).

Step 3.          For VLAN & VSAN Configuration, under Fabric Interconnect B, for VLAN Configuration, click Select Policy.

Step 4.          In the Select Policy window, choose the previously created VLAN Policy for FI-B (same as FI-A VLAN policy in this CVD).

Step 5.          Click Next.

Procedure 18.  Create a UCS Domain Profile Template: Ports Configuration for FI-A and FI-B

Step 1.          For Ports Configuration, under Fabric Interconnect A, click Select Policy.

Step 2.          In the Select Policy window, choose the previously created Port Policy for FI-A.

Step 3.          For Ports Configuration, under Fabric Interconnect B, click Select Policy.

Step 4.          In the Select Policy window, choose the previously created Port Policy for FI-B.

Step 5.          Click Next.

Procedure 19.  Create a UCS Domain Profile Template: UCS Domain Configuration

Step 1.          For the UCS Domain Configuration, under Management, choose the previously configured policies for NTP, Syslog, Network Connectivity, SNMP, LDAP (not used in this CVD), Certificate Management (not used in this CVD) policies by clicking Select Policy for each policy.

Step 2.          Under Network, choose the previously configured policies for System QoS and System Control policies by clicking Select Policy for each policy.

Step 3.          Click Next.

Procedure 20.  Create a UCS Domain Profile Template : Summary

Step 1.          In the Summary window, verify the settings across the Ports Configuration, VLAN & VSAN Configuration, and UCS Domain Configuration tabs.

Step 2.          Click Close to save the Domain Profile Template and derive the profiles later.

Procedure 21.  Derive and Assign UCS Domain Profile Configuration for UCS domain

To generate a UCS domain Profile to configure the UCS domain, complete the following steps to derive a domain profile using the previously configured domain profile template.

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates.

Step 3.          Click UCS Domain Profile Templates.

Step 4.          Choose a previously configured UCS Domain Profile Template from the list.

Step 5.          Click the elipses and choose Derive Profiles from the drop-down list.

Step 6.          From the Derive window, in the General section for Domain Assignment, choose Assign Now or Assign Later to deploy the UCS domain profile (now or later). For the former, choose the UCS domain from the list.

Step 7.          Click Next.

Step 8.          From the Derive window, in the Details section, specify the profile names for the derived profile(s).

Step 9.          Click Next.

Step 10.       From the Derive window, in the Summary section, verify the settings.

Step 11.       Click Derive.

Procedure 22.  Deploy the Cisco UCS Domain Profile to UCS Domain

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Profiles.

Step 3.          Click UCS Domain Profiles.

Step 4.          Choose the previously derived UCS Domain Profile from the list. You can also create/derive new UCS Domain Profiles here.

Step 5.          Click the elipses and choose Deploy from the list.

Step 6.          In the pop-up window, click Deploy.

Step 7.          For Name, specify a new name for the profile or us the previously specified one.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          Click Next.

Step 10.       For UCS Domain Assignment, choose the UCS FI to apply the configuration/profile.

Step 11.       Click Next.

Step 12.       Step through the remaining options (VLAN & VSAN Configuration, Ports Configuration, UCS Domain Configuration, and Summary) and verify the configuration.

Step 13.       Click Deploy.

Procedure 23.  Verify the Cisco UCS Domain Profile Deployment

Step 1.          Go to Configure > Profiles > UCS Domain Profiles and verify that the domain profile was successfully deployed – it should have a status of OK.

Step 2.          Go to Operate > Chassis to verify that the chassis has been discovered and is visible.

Step 3.          Go to Operate > Servers to verify that the servers have been discovered and are healthy.

Configure the UCS Chassis using a Chassis Profile derived from a UCS Chassis Profile Template

The procedures described in this section will create the pools and policies that are part of the UCS Chassis Profile Template. The template is then used to derive the UCS Chassis Profile for configuring a given Cisco UCS X-Series chassis. This includes:

●    Create IP Pool for In-Band Management.

●    Create IP Pool for Out-of-Band Management.

●    Create IMC Access Policy: IP configuration for the in-band chassis connectivity. This setting is independent of Server IP connectivity and only applies to communication to and from the chassis.

●    Create SNMP Policy to configure SNMP trap settings.

●    Create Power Policy to enable power management and power supply redundancy mode.

●    Create Thermal Policy to control the speed of FANs. 

These policies will be used to:

●    Create a UCS Chassis Profile Template.

●    Derive a UCS Chassis Profile from the UCS Domain Profile Template.

●    Deploy the UCS Chassis Profile to provision a given UCS X-Series chassis.

Assumptions and Prerequisites

●    Complete the initial setup of the Cisco UCS Fabric Interconnects with IP Management access.

●    Valid cisco.com and Intersight accounts.

●    Enable Cisco Smart Software Licensing on the Intersight account.

●    Collect the setup information for your environment – see Table 12 for the required setup information.

Setup Information

Table 12.     Cisco Intersight: UCS Chassis Profile Template

Deployment Steps

Use the setup information provided in this section to configure pools and policies for the Cisco UCS X-Series chassis. The pools and policies are part of the Cisco UCS Chassis Profile Template which is used to derive the UCS Chassis Profile to configure a given Cisco UCS X-Series chassis.

Procedure 1.     Create IP Pools – Out-of-Band Management

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Pools.

Step 3.          Click Create Pool.

Step 4.          Choose IP.

Step 5.          Click Start.

Step 6.          From the Pools > IP Pool window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the IPv4 Pool Details section, specify the Netmask, Gateway, Primary DNS and Secondary DNS.

Step 12.       Click Add IP Blocks.

Step 13.       Click Next.

Step 14.       Disable Configure IPv6 Pool.

Step 15.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 2.     Create IP Pools – In-Band Management

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Pools.

Step 3.          Click Create Pool.

Step 4.          Choose IP.

Step 5.          Click Start.

Step 6.          From the Pools > IP Pool window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the IPv4 Pool Details section, specify the Netmask, Gateway, Primary DNS and Secondary DNS.

Step 12.       Click Add IP Blocks. Add two IP blocks, one for UCS Fis and another for servers connected to the UCS FIs.

Step 13.       Click Next.

Step 14.       Disable Configure IPv6 Pool.

Step 15.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 3.     Create IMC Access Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose IMC Access.

Step 5.          Click Start.

Step 6.          From the Policies > IMC Access window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, enable In-Band Configuration.

Step 12.       For the VLAN ID, specify the ID for In-Band Management.

Step 13.       Check the box for IPv4 Address Configuration.

Step 14.       For IP Pool, click Select IP Pool to choose the previously configured In-Band Management IP Pool.

Step 15.       Enable Out-of-Band Configuration.

Step 16.       For IP Pool, click Select IP Pool to choose the previously configured Out-of-Band Management IP Pool.

Step 17.       Click Next.

Step 18.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 4.     Create a Power Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Power.

Step 5.          Click Start.

Step 6.          From the Policies > Power window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, for Power Restore state, use the drop-down list to change the state to Last State. Keep the remaining defaults.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 5.     Create a SNMP Policy

Procedure 6.     Create a Thermal Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Thermal.

Step 5.          Click Start.

Step 6.          In the Policies > Thermal window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, use the defaults for FAN Control Mode.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 7.     Create a UCS Chassis Profile Template

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates.

Step 3.          Choose UCS Domain Profile Templates.

Step 4.          Click Create UCS Domain Profile Template.

Step 5.          In the Create UCS Domain Profile Template window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 6.          For Name, specify a name for this policy.

Step 7.          (Optional) For Set Tags, specify value in key:value format.

Step 8.          (Optional) For Description, specify a description for this policy.

Step 9.          Click Next.

Step 10.       In the Chassis Configuration section, for IMC Access, click Select Policy and choose the policy provisioned earlier.

Step 11.       For Power policy, click Select Policy and choose the policy provisioned earlier.

Step 12.       For SNMP policy, skip this step. This policy was already attached to Cisco UCS Domain Profile.

Step 13.       For Thermal policy, click Select Policy and choose the policy provisioned earlier.

Step 14.       Click Next.

Step 15.       In the Summary section, verify the settings.

Step 16.       Click Close to exit and save the chassis profile template. You can also click Derive Profiles to immediately derive a chassis profile.

Procedure 8.     Derive UCS Chassis Profile(s) to configure UCS X-Series chassis

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates.

Step 3.          Choose UCS Chassis Profile Template.

Step 4.          Click the UCS Chassis Profile Template to use from the list.

Step 5.          Click the elispses and choose Derive Profiles from the drop-down list.

Step 6.          Click Next.

Step 7.          In the UCS Chassis Profile Template > FPB-OAI-Chassis-Profile_Template window, in the General section, choose Assign Later, and specify the Number of Profiles to derive.

Step 8.          Click Next.

Step 9.          In the Details section, specify the Name and Organization for the profile.

Step 10.       Click Next.

Step 11.       In the Summary section, verify the information.

Step 12.       Click Derive to derive a chassis profile to configure a given Cisco UCS X-Series chassis.

Procedure 9.     Assign and Deploy Cisco UCS Chassis Profile to configure the Cisco UCS X-Series chassis

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Profiles.

Step 3.          Choose UCS Chassis Profiles.

Step 4.          Choose the previously derived UCS Chassis Profile from the list. You can also create/derive a new UCS Chassis Profiles from here.

Step 5.          Click the elipses and choose Edit from the drop-down list.

Step 6.          Click the elipses and choose Deploy from the list.

Step 7.          Choose the UCS X-Series chassis to deploy the profile. 

Step 8.          Click Next.

Step 9.          Click Deploy.

Procedure 10.  Verify Cisco UCS Chassis Profile Deployment

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Profiles.

Step 3.          Choose UCS Chassis Profiles.

Step 4.          Verify that the chassis profile was successfully deployed – it should have a Status of OK with the profile Name and Chassis ID that was used to deploy the profile on the chassis.

Step 5.          Verify that the chassis has been discovered and is visible under Operate > Chassis.

Configure the UCS Server(s) using the Server Profile(s) derived from a UCS Server Profile Template

The procedures described in this section create the pools and policies necessary to configure a UCS Server Profile Template.

●    Pools

◦     UUID Pool

◦     iSCSI Pools

◦     IQN Pools (not used in this CVD)

◦     MAC Pool

◦     IP Pools (if different from ones created for UCS Chassis

◦     FC WW Pools (not used in this CVD)

●    Policies

◦     Adapter Configuration Policy (Only for Standalone UCS Servers – not used in this CVD)

◦     BIOS Policy

◦     Boot Order Policy (Local boot used in this CVD)

◦     Ethernet Adapter Policy

◦     Ethernet Network Control Policy

◦     Ethernet Network Group (for vNICs/interfaces on servers) Policy

◦     Ethernet QoS Policy

◦     FC Adapter Policy (not used in this CVD)

◦     FC NVMe Initiator Adapter Policy (not used in this CVD)

◦     FC Network Policy (not used in this CVD)

◦     FC QoS Policy (not used in this CVD)

◦     Firmware Policy

◦     IPMI over LAN Policy

◦     iSCSI Adapter Policy (not used in this CVD)

◦     iSCSI Boot Policy (not used in this CVD)

◦     iSCSI Target Policy (not used in this CVD)

◦     Local User Policy

◦     SAN Connectivity Policy (not used in this CVD)

◦     Storage Policy

◦     vKVM Policy

◦     vMedia Policy

Some of the pools and policies defined above are used to create vNIC Templates and LAN Connectivity Policies as summarized below:

●    vNIC Templates for OpenShift Control Nodes

●    vNIC Templates for OpenShift Worker Nodes (with storage NICs for storage access)

●    LAN Connectivity Policies for:

◦     FC Boot (not used in this CVD)

◦     ISCSI Boot (not used in this CVD)

◦     OpenShift Control Nodes using Local Boot

◦     OpenShift Worker Nodes using Local Boot (+ iSCSI/NVMe Storage, + NFS/Object Storage)

These pools and policies are used for the following:

●    Create UCS Server Profile Templates for OpenShift Control and Worker Nodes

●    Derive UCS Server Profiles from UCS Server Profile Templates for OpenShift Control and Worker Nodes

●    Deploy the UCS Server Profiles to provision UCS servers as OpenShift Control and Worker Nodes

Assumptions and Prerequisites

●    Complete the initial setup of Cisco UCS Fabric Interconnects with IP Management access

●    Valid cisco.com and Intersight accounts

●    Enable Cisco Smart Software Licensing on the Intersight account

●    Name of the Intersight Organization of which the UCS Server is comprised

●    Collect setup information for your environment - see Table 13 for the required setup information

Setup Information

Table 13.     Cisco Intersight: UCS Server Profile Template

Deployment Steps - Pools and Policies

Use the setup information provided above to configure pools and policies for the UCS server. The pools and policies are part of the UCS Server Profile Template which are used to derive the UCS Server Profile to configure Cisco UCS servers.

Procedure 1.     Create MAC Pools

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Pools.

Step 3.          Click Create Pool.

Step 4.          Choose MAC.

Step 5.          Click Start.

Step 6.          In the Pools > MAC Pool window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Pool Details section, specify the MAC Blocks consisting of Starting MAC Address and Pool Size.

Step 12.       Click + to add additional MAC pools as needed.

Step 13.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 2.     Create UUID Pools

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Pools.

Step 3.          Click Create Pool.

Step 4.          Choose UUID.

Step 5.          Click Start.

Step 6.          In the Pools > UUID window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Pool Details section, specify the Prefix and UUID Blocks with starting value and size.

Step 12.       Click + to add additional pools as needed.

Step 13.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 3.     Create a BIOS Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose BIOS.

Step 5.          Click Start.

Step 6.          In the Policies > BIOS window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, expand Processor.

Step 12.       From the Intel® VT drop-down list, click disabled.

Step 13.       In the Policy Details section, expand Processor > Processor C6 Report and from the drop-down list, click enabled.

Step 14.       Go to Processor > Workload Configuration and from the drop-down list, click enabled.

Step 15.       In the Policy Details section, expand Server Management > Consistent Device Naming and from the drop-down list, click enabled.

Step 16.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 4.     Create a Boot Order Policy

 

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Boot Order.

Step 5.          Click Start.

Step 6.          In the Policies > Boot Order window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, leave UEFI enabled and Secure Boot disabled.

Step 12.       Click Add Boot Device and choose Virtual Media from the drop-down list.

Step 13.       Configure the Device Name and Sub-Type to mount the ISO using virtual KVM session to the server.

Step 14.       Click Add Boot Device and choose Virtual Media from the drop-down list.

Step 15.       Configure the Device Name and Sub-Type to mount the ISO using the server’s CIMC.

Step 16.       Click Add Boot Device and choose Local Disk from the drop-down list.

Step 17.       Specify the Device Name and Slot to enable boot from local disk. The Local Disk boot option should be at the top to ensure that the nodes always boot from the M.2 disks once after Red Hat CoreOS is installed on the servers.

Step 18.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 5.     Create Ethernet Adapter Policies

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Ethernet Adapter to create the first policy.

Step 5.          Click Start.

Step 6.          In the Policies > Ethernet Adapter window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for the first policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       Click Select Cisco Provided Configuration and choose Linux-v2 from the drop-down list.

Step 12.       Click Select.

Step 13.       Click Next.

Step 14.       Review the Settings for the policy.

Step 15.       Click Create. A pop-up message displays stating the policy was created successfully.

Step 16.       To create the second ethernet adapter policy, go to Configure > Policies.

Step 17.       Click Create Policy.

Step 18.       Choose Ethernet Adapter.

Step 19.       Click Start.

Step 20.       In the Policies > Ethernet Adapter window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 21.       For Name, specify a name for the first policy.

Step 22.       (Optional) For Set Tags, specify value in key:value format.

Step 23.       (Optional) For Description, specify a description for this policy using the settings in the Setup Information section.

Step 24.       Click Next.

Step 25.       Use the settings in Setup Information to configure this policy as shown below:

Step 26.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 6.     Create a Ethernet Network Control Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Ethernet Network Control.

Step 5.          Click Start.

Step 6.          In the Policies > Ethernet Network Control window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, enable CDP and LLDP.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 7.     Create Ethernet Network Group Policies – for In-Band, Cluster Management, and Storage vNICs

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Ethernet Network Group.

Step 5.          Click Start.

Step 6.          In the Policies > Ethernet Network Group window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, click Add VLANs and from the drop-down list, choose From Policy.

Step 12.       In the Select Policy window, choose the previously created VLAN policy.

Step 13.       Click Next.

Step 14.       Choose the VLAN to add to the first vNIC.

Step 15.       Click Select.

Step 16.       In the Policy Details window, choose the VLAN and right-click the elipses and click Set Native VLAN.

Step 17.       Click Create. A pop-up message displays stating the policy was created successfully.

Step 18.       Repeat this procedure to create the Ethernet Network Group Policies for the remaining vNICs using the Setup Information for this section.

Procedure 8.     Create Ethernet QoS Policies – Default and Jumbo

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Ethernet QoS.

Step 5.          Click Start.

Step 6.          In the Policies > Ethernet QoS window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for the first policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, keep all default settings.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Step 13.       Repeat the procedures to create the second policy using the Setup Information for this section.

Step 14.       In the Policy Details section, keep the defaults for everything except for the MTU. Set MTU to 9000.

Step 15.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 9.     Create the Firmware Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Firmware.

Step 5.          Click Start.

Step 6.          In the Policies > Firmware window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          Choose Target Platform.

Step 9.          (Optional) For Set Tags, specify value in key:value format.

Step 10.       (Optional) For Description, specify a description for this policy.

Step 11.       Click Next.

Step 12.       In the Policy Details section, choose the Server Model from the drop-down list and Firmware Version to use for the server model.

Step 13.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 10.  Create the IPMI over LAN Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose IPMI over LAN.

Step 5.          Click Start.

Step 6.          In the Policies > IPMI over LAN window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, keep all the default settings.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 11.  Create a Local User Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Local User.

Step 5.          Click Start.

Step 6.          In the Policies > Local User window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, click Add Users to add an admin user. Keep the default settings for everything else.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 12.  Create a Storage Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Storage.

Step 5.          Click Start.

Step 6.          In the Policies > Storage window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, enable M.2 RAID Configuration. Keep the defaults settings for everything else.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 13.  Create Virtual KVM Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Power.

Step 5.          Click Start.

Step 6.          In the Policies > Power window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, enable Allow Tunneled vKVM. Keep the default settings for everything else.

Step 12.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 14.  Create a Virtual Media Policy

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose Virtual Media.

Step 5.          Click Start.

Step 6.          In the Policies > Virtual Media window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          (Optional) For Set Tags, specify value in key:value format.

Step 9.          (Optional) For Description, specify a description for this policy.

Step 10.       Click Next.

Step 11.       In the Policy Details section, keep the default settings.

Step 12.       Click Add Virtual Media.

Step 13.       In the Add Virtual Media window, use the settings in Setup Information to configure the policy.

Step 14.       Click Add.

Step 15.       Click Create. A pop-up message displays stating the policy was created successfully.

Deployment Steps – vNIC Templates

Use the Setup Information to configure vNIC templates for the UCS server. The vNIC templates will be used by the LAN connectivity policy that will be part of the UCS Server Profile Template which will be used to derive the UCS Server Profile to configure Cisco UCS servers.

Procedure 1.     Create a vNIC Template for In-Band Management

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates from the left navigation pane.

Step 3.          Choose the vNIC Templates tab and click Create vNIC Template.

Step 4.          In the Create vNIC Template window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 5.          For Name, specify a name for this policy.

Step 6.          (Optional) For Set Tags, specify value in key:value format.

Step 7.          (Optional) For Description, specify a description for this policy.

Step 8.          Click Next.

Step 9.          In the Configuration section, for MAC Pool, click Select Pool to choose the MAC pool for Fabric-A (FI-A).

Step 10.       Enable Failover.

Step 11.       For Ethernet Network Group, click Select Policies to choose the policy for this vNIC template.

Step 12.       For Ethernet Network Control, click Select Policy to choose the policy for this vNIC template.

Step 13.       For Ethernet QoS, click Select Policy to choose the policy for this vNIC template.

Step 14.       For Ethernet Adapter, click Select Policy to choose the policy for this vNIC template.

Step 15.       Keep the default settings for everything else.

Step 16.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 2.     Create a vNIC Template for OpenShift Cluster Management

Step 1.          Repeat the deployment steps in the previous procedure to deploy a vNIC template for OpenShift Cluster. Use the settings from the Setup Information for this section.

Procedure 3.     Create a vNIC Template for iSCSI-A storage

Step 1.          Repeat the deployment steps in the previous procedure to deploy a vNIC template for iSCSI-A. Use the settings from the Setup Information for this section.

Procedure 4.     Create a vNIC Templates for iSCSI-B storage

Step 1.          Repeat the deployment steps in the previous procedure to deploy a vNIC template for iSCSI-B. Use the settings from the Setup Information for this section.

Procedure 5.     Create a vNIC Templates for NFS Storage access

Step 1.          Repeat the deployment steps in the previous procedure to deploy a vNIC template for NFS storage. Use the settings from the Setup Information for this section.

Procedure 6.     Create a vNIC Templates for Object Store access

Step 1.          Repeat the deployment steps in the previous procedure to deploy a vNIC template for Object store. Use the settings from the Setup Information for this section.

Deployment Steps – LAN Connectivity Policy

Use the Setup Information to configure LAN Connectivity policy for the UCS server. This policy is part of the UCS Server Profile Template which is used to derive the UCS Server Profile to configure Cisco UCS servers.

Procedure 1.     Create LAN Connectivity Policy for OpenShift Control Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose LAN Connectivity.

Step 5.          Click Start.

Step 6.          In the Policies > LAN Connectivity window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          Choose Target Platform.

Step 9.          (Optional) For Set Tags, specify value in key:value format.

Step 10.       (Optional) For Description, specify a description for this policy.

Step 11.       Click Next.

Step 12.       In the Policy Details section, under vNIC Configuration, click Add and choose vNIC from Template from the drop-down list.

Step 13.       In the Add vNIC from Template window, click Select vNIC Template.

Step 14.       Choose the vNIC Template from the list.

Step 15.       Click Select.

Step 16.       Specify a Name and PCI Order for this NIC.

Step 17.       Click Add.

Step 18.       To add additional vNICs, repeat steps 1 – 17 of this procedure. In the Policy Details section, under vNIC Configuration, click Add and choose vNIC from Template from the drop-down list to add additional vNICs. Provide a unique Name and PCI Order for each new vNIC.

Step 19.       Click Create. A pop-up message displays stating the policy was created successfully.

Procedure 2.     Create a LAN Connectivity Policy for OpenShift Worker Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Policies.

Step 3.          Click Create Policy.

Step 4.          Choose LAN Connectivity.

Step 5.          Click Start.

Step 6.          In the Policies > LAN Connectivity window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 7.          For Name, specify a name for this policy.

Step 8.          Choose Target Platform.

Step 9.          (Optional) For Set Tags, specify value in key:value format.

Step 10.       (Optional) For Description, specify a description for this policy.

Step 11.       Click Next.

Step 12.       In the Policy Details section, under vNIC Configuration, click Add and choose vNIC from Template from the drop-down list.

Step 13.       In the Add vNIC from Template window, click Select vNIC Template.

Step 14.       Choose the vNIC Template from the list.

Step 15.       Click Select.

Step 16.       Specify a Name and PCI Order for this NIC.

Step 17.       Click Add.

Step 18.       Repeat steps 1 – 17 in this procedure to add the remaining vNICs. In the Policy Details section, under vNIC Configuration, click Add and then choose vNIC from Template from the drop-down list to add additional vNICs. Provide a unique Name and PCI Order for each new vNIC.

Step 19.       Click Create. A pop-up message displays stating the policy was created successfully.

Deployment Steps – Server Profile Template

Use the Setup Information to create a Cisco UCS Server Profile Template. This template is used to derive the UCS Server Profile(s) to provision individual UCS servers.

Procedure 1.     Create UCS Server Profile Template for OpenShift Control Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates.

Step 3.          Choose the UCS Server Profile Templates tab. Click UCS Server Profile Template.

Step 4.          In the Create UCS Server Profile Template window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 5.          For Name, specify a name for this template.

Step 6.          For Target, choose a UCS platform.

Step 7.          (Optional) For Set Tags, specify value in key:value format.

Step 8.          (Optional) For Description, specify a description for this policy.

Step 9.          Click Next.

Step 10.       In the Compute Configuration section, choose the previously created pools and policies.

Step 11.       Click Next.

Step 12.       In the Management Configuration section, choose the previously created pools and policies.

Step 13.       Click Next.

Step 14.       In the Storage Configuration section, choose the previously created pools and policies.

Step 15.       Click Next.

Step 16.       In the Network Configuration section, choose the previously created Connectivity Policies.

Step 17.       Click Next.

Step 18.       In the Summary section, review the selections made.

Step 19.       Click Close – the profile will be derived at a later time.

Procedure 2.     Create UCS Server Profile Template for OpenShift Worker Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates from the left navigation pane.

Step 3.          Choose the UCS Server Profile Templates tab. Click UCS Server Profile Template.

Step 4.          In the Create UCS Server Profile Template window, in the General section, for Organization, choose the previously created organization from the drop-down list.

Step 5.          For Name, specify a name for this template.

Step 6.          For Target Platform, choose a UCS platform.

Step 7.          (Optional) For Set Tags, specify value in key:value format.

Step 8.          (Optional) For Description, specify a description for this policy.

Step 9.          Click Next.

Step 10.       In the Compute Configuration section, choose the previously created pools and policies.

Step 11.       Click Next.

Step 12.       In the Management Configuration section, choose the previously created pools and policies.

Step 13.       Click Next.

Step 14.       In the Storage Configuration section, choose the previously created pools and policies.

Step 15.       Click Next.

Step 16.       In the Network Configuration section, choose the previously created Connectivity Policies.

Step 17.       Click Next.

Step 18.       In the Summary section, review the selections made.

Step 19.       Click Close – the profile will be derived at a later time.

Deployment Steps – Derive Server Profile(s)

The procedures in this section derive the UCS Server Profile(s) to configure UCS servers from a previously provisioned UCS Server Profile Template.

Procedure 1.     Derive the UCS Server Profile(s) for OpenShift Control Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates.

Step 3.          Choose UCS Server Profile Templates.

Step 4.          Choose a UCS Server Profile Template to use.

Step 5.          Click the elipses and choose Derive Profiles from the drop-down list.

Step 6.          Click Next.

Step 7.          In the UCS Server Profile Template > FPB-OAI-M7-OCP-C-Server-Profile_Template window, in the General section, choose Assign Later, and specify the Number of Profiles to derive in the box provided.

Step 8.          Click Next.

Step 9.          In the Details section, specify the naming format for the profiles.

Step 10.       Click Next.

Step 11.       In the Summary section, review the information.

Step 12.       Click Derive to derive server profiles to configure the OpenShift Control Nodes.

Procedure 2.     Derive UCS Server Profile(s) for OpenShift Worker Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Templates.

Step 3.          Choose UCS Server Profile Templates.

Step 4.          Choose UCS Server Profile Template to use.

Step 5.          Click the elipses and choose Derive Profiles from the drop-down list.

Step 6.          Click Next.

Step 7.          In the UCS Server Profile Template > FPB-OAI-M7-OCP-W-Server-Profile_Template window, in the General section, choose Assign Later, and specify the Number of Profiles to derive in the box provided.

Step 8.          Click Next.

Step 9.          In the Details section, specify the naming format for the profiles.

Step 10.       Click Next.

Step 11.       In the Summary section, review the information.

Step 12.       Click Derive to derive server profiles to configure the OpenShift Control Nodes.

Deployment Steps – Deploy Server Profile(s)

The procedures in this section deploy server profiles that provision the servers.

Procedure 1.     Deploy and Assign Cisco UCS Server Profiles to configure OpenShift Control  and Worker Nodes

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Profiles.

Step 3.          Choose UCS Server Profiles.

Step 4.          Choose a previously derived UCS Server Profiles from the list.

Step 5.          Click the elipses and choose Assign Server from the list.

Step 6.          Choose the UCS server to deploy the profile. 

Step 7.          Click Assign.

Step 8.          Repeat steps 1 – 7  to assign the remaining server profiles.

Step 9.          To deploy the server, click the elipses and click Deploy from the drop-down list.

Step 10.       In the Deploy UCS Server Profile pop-up window, make the selections to activate and reboot the server.

Step 11.       Click Deploy. The system will validate and provision the settings on the selected servers.

Step 12.       Repeat this procedure to assign and deploy server profiles to all OpenShift control and worker nodes in the cluster being built.

Procedure 2.     Verify that  UCS Server Profiles were deployed successfully

Step 1.          Use a web browser to navigate to intersight.com and log into your account.

Step 2.          Go to Configure > Profiles.

Step 3.          Choose UCS Server Profiles.

Step 4.          Verify that all server profiles have a Status of OK with the profile Name and the server on which it was deployed.

Deploy Kubernetes – OpenShift on Baremetal UCS Servers

This section provides the detailed procedures for deploying a Red Hat OpenShift Kubernetes (K8s) cluster in an Enterprise data center. The cluster will be used for hosting AI/ML workloads managed using Red Hat OpenShift AI serving as the MLOps platform for this solution. The OpenShift cluster is deployed from the cloud using Red Hat Hybrid Cloud Console using the Red Hat recommended Assisted Installer. Red Hat provides other installation options depending on the level of customization required.

Prerequisites

●    Complete compute, network and storage setup which includes:

◦     Intersight managed Cisco UCS servers that serve as control and worker nodes in the OpenShift cluster

◦     Setup of NetApp storage for providing NFS, S3-compatible Object and IP based block storage access using iSCSI/NVMe over TCP.

◦     Networking enabled through Top-of-Rack Cisco Nexus switches with access from compute to storage as well as external access with reachability to quay.io, Red Hat’s Hybrid Cloud Console, and so on. The IP subnet for the OpenShift cluster should be reachable from the Installer and should have access to network services (see prerequisites below)

●    Valid Red Hat account on Red Hat’s Hybrid Cloud Console (console.redhat.com) to access and deploy the cluster using Red Hat’s Assisted installer

●    OpenShift requires the following components to be in place before the installation:

◦     Installer workstation or machine for OpenShift cluster management – Rocky Linux is used in this CVD. This installer provides CLI access to the cluster. More importantly, it provides secure SSH access to nodes in the cluster post-deployment. This requires prerequisite setup (see below) before the cluster is deployed.

◦     To enable SSH Access to the OpenShift cluster, public keys must be provided to the OpenShift installer. Installer will pass the keys to the nodes through the initial configuration (ignition) files during installation. The nodes will add the keys to the ~/.ssh/authorized_keys list to enable password-less secure authentication as user: core.

◦     IP subnet for OpenShift cluster. Two static IP addresses from this subnet will need to be allocated for use as API and Ingress Virtual IP (VIP).

◦     NTP: IP address for an NTP source in your network.

◦     DHCP Server – Windows AD server enabled for this service is used in this CVD. DHCP is used to dynamically provision the IP address on all interfaces on OpenShift control and worker baremetal nodes – total of two and six interfaces.

◦     DNS Server - Specific DNS entries/records for OpenShift. See definitions from Red Hat documentation:

-      Base Domain

-      OpenShift Cluster Name

-      API Virtual IP

-      Ingress Load Balancer Virtual IP

Setup Information

Table 14.     Red Hat OpenShift Deployment: Setup Information

Deployment Steps – Setup Prerequisites

This section details the prerequisite setup to install Openshift cluster. 

Procedure 1.     Deploy Installer workstation to manage the OpenShift cluster

Step 1.          Deploy a workstation with Linux (for example, Rocky Linux, RHEL) to manage the Openshift cluster using CLI.

Step 2.          On the OpenShift Installer machine, create a new directory for storing all data related to the new cluster being deployed. For example: ocp-ac10-cvd is used in this CVD.

Procedure 2.     Enable SSH access to OpenShift cluster

Step 1.          Go to the newly created directory and run the following commands to generate a SSH key pair to enable SSH access to the OpenShift cluste:.

 

Step 2.          Verify that the ssh-agent process is running and if not, start it as a background task by running the following:

Step 3.          Add the SSH private key identity to the SSH agent for your local user.

Step 4.          Assisted Installer adds the SSH keys to the ignition files that are used to do the initial configuration of the OpenShift nodes. Once the OpenShift cluster is deployed, you will be able to access the cluster as user core without the need for password.

Step 5.          Verify the connectivity from OpenShift cluster subnet to all NTP sources.

Procedure 3.     Add the DNS records for the OpenShift cluster’s API and Ingress VIP 

Step 1.          On the DNS server, create a domain (for example, ac10-ocp) and sub-domain (for example, apps) under the parent/base domain (for example, fpb.local).

For this CVD, the DNS service is enabled on a Windows AD server. The DNS configuration on this server for this cluster is shown below: 

Procedure 4.     Add DHCP Pools for various networks and configure the DHCP options for NTP, DNS, Gateway

Step 1.          On the DHCP server, create DHCP scopes for OpenShift control and worker node subnets.

For this CVD, the DHCP service is enabled on a Windows AD server. The DHCP configuration on the server for this cluster is shown below:

For each scope, the following DHCP options are configured:

Deployment Steps – Install the OpenShift Cluster

This section details the post-deployment verification. 

Procedure 1.     Install OpenShift cluster using Assisted Installer from Red Hat Hybrid Cloud Console

Step 1.          Use a web browser to go to console.redhat.com and log into your account.

Step 2.          Go to Containers > Clusters.

Step 3.          Go to the Red Hat OpenShift Container Platform (OCP) tile and click Create Cluster.

Step 4.          Click the Datacenter tab.

Step 5.          Choose Assisted Installer > Create Cluster. Alternatively, you can choose another infrastructure and installer option from the list below.

Step 6.          For Cluster Details, provide the necessary inputs to the Assisted Installer as shown. Use the settings provided in Table 14.

Step 7.          Scroll down to the end of the page and choose the settings shown below:

Step 8.          Click Next. 

Step 9.          For Operators, skip all options.

Step 10.       Scroll-down to the end of the page.

Step 11.       Click Next.

Step 12.       For Host Discovery, click Add Hosts.

Step 13.       In the Add Hosts pop-up window, for the Provisioning Type, choose Minimal Image File from the drop-down list.

Step 14.       For the SSH public key, upload the SSH keys previously generated on the Installer workstation. Keep the default settings.

Step 15.       Click Generate Discovery ISO.

Step 16.       In the Add hosts window, click Add Hosts from Cisco Intersight, to select and deploy the Discovery ISO on the selected servers to start the installation process.

Step 17.       Now you will be directed to Cisco Intersight. Log in using the account where the servers were deployed.

After you log in, you will see the Execute Workflow: Boot Servers from ISO URL window:

Step 18.       For Organization, choose the Intersight organization for the UCS servers (for example, FPB-OAI).

Step 19.       For Servers, click No Servers Selected and choose from the list of available servers.

Step 20.       Click Save.

Step 21.       Click Execute to initiate the download of the Discovery ISOs to the list of servers that will form the OpenShift cluster with control and worker nodes.

The following screenshot displays showing that the workflow was triggered:

Step 22.       Click the Requests icon to monitor the booting of the UCS servers using the discovery ISO provided Assisted Installer running on the Hybrid Cloud Console.

Step 23.       When the process completes (~5 minutes), return to the Hybrid Cloud Console and Assisted Installer.

Step 24.       Click Close.

You will see the nodes show up one by one in the Assisted Installer under Host Discovery > Host Inventory.

Step 25.       Wait for all nodes to display. This may take several minutes.

Step 26.       Click View Cluster events to view the activities.

Step 27.       Click Next.

When all five servers have booted from the Discovery ISO, they will appear in the Assisted Installer:

Step 28.       For each server, under Role, from the drop-down list choose whether it is a control plane node or a worker node.

Step 29.       Edit the hostname of each node by clicking the current hostname.

Step 30.       Expand each node and verify NTP is synced.

Step 31.       Scroll down and click Next.

Step 32.       Expand each node and confirm the role of the M.2 disk is set to Installation disk.

Step 33.       Click Next.

Step 34.       For Networking, choose Cluster-Managed Networking. For Machine network, choose the OpenShift cluster management subnet from the drop-down list. Specify the API and Ingress IP in the corresponding fields.

Step 35.       Scroll down and check that all nodes have a Ready status.

Step 36.       When all nodes are in a Ready status, click Next.

Step 37.       Review the information.

Step 38.       Click Install cluster to begin the cluster installation.

Step 39.       On the Installation progress page, expand the Host inventory. The installation will take 30-45 minutes.

When the installation is complete, the status of all nodes display as completed.

You can expand a node and verify its configuration as shown below. For example, interface names, speeds, DHCP provided IP addresses and GPU specific information, such as the GPU model and Vendor:

Deployment Steps – Post-Deployment

This section details the post-deployment steps and other verifications required.

Procedure 1.     Download and save the kubeconfig and kubeadmin password from the Red Hat Hybrid Cloud Console

Step 1.          When the install is complete, download and save the kubeconfig file in a safe location as instructed.

Step 2.          To download the kubeconfig file, click Download kubeconfig. Copy the file (if needed) to the OpenShift Installer machine and save it in the location specified in the next step.

Step 3.          On the installer machine, in a terminal window, after you’ve downloaded the file to the installer machine, run the following commands to copy the file (from Downloads) to the location in the cluster directory as specified:

Step 4.          Return to the Assisted Installer cluster installation page on Red Hat Hybrid Cloud Console and click the icon next to kubeadmin password to copy the password.

Step 5.          On the installer machine, in a terminal window, run the following commands to copy and save the kubeadmin password in a location specified below:

Procedure 2.     Download kubeconfig, kubeadmin password, and oc tools

Step 1.          Return to the Assisted Installer cluster installation page on Red Hat Hybrid Cloud Console and click or copy the Web Console URL and go to the URL to launch the OpenShift Console for the newly deployed cluster. Log in using the kubeadmin and the kubeadmin password.

Step 2.          Click the ? icon and choose Command Line Tools from the drop-down list. Links for various tools are provided in this page.

Step 3.          Click Download oc for Linux for x86_64.

Step 4.          Copy the file (if needed) to the OpenShift Installer machine and save it in the cluster directory location as specified below:

Step 5.          To enable oc tab completion for bash, run the following:

Procedure 3.     Set up Power Management for Bare Metal Hosts

Step 1.          Log into the OpenShift Cluster Console.

Step 2.          Go to Compute > Nodes to see the status of the OpenShift nodes.

Step 3.          In the Red Hat OpenShift console, click Compute > Bare Metal Hosts.

Step 4.          For each Bare Metal Host, click the elipses and choose Edit Bare Metal Host and check the box to Enable power management.

Step 5.          Click Save.

Step 6.          Repeat steps 1 - 5 for all baremetal hosts in the cluster.

 

Procedure 4.     (Optional) Reserve Resources for System Components

Step 1.          Log into the OpenShift Cluster console.

Step 2.          Click the + to generate the following YAML file with the necessary configuration for worker nodes:

Step 3.          Click Create.

Step 4.          Repeat steps 1 and 2 for control node and then run the following:

Step 5.          Click Create.

Procedure 5.     Setup NTP on control-plane and worker nodes

Step 1.          Log into the OpenShift Installer machine and create a new directory for storing machine configs in the previously created cluster directory as shown below:

Step 2.          Create the following files in the machine-configs directory with the correct NTP Ips:

File: 99-control-plane-chrony-conf-override.bu

File: 99-worker-chrony-conf-override.bu

Step 3.          Create the .yaml files from the butane files with butane:

Step 4.          Apply the configuration to the OpenShift cluster:

Deployment Steps – Setup Storage interfaces on Kubernetes Nodes

This section details the following:

●    Setup networking for accessing storage using iSCSI – enable iSCSI multipathing

●    Setup networking for accessing storage using NVMe-TCP

●    Setup networking for accessing storage using NFS

●    Setup networking for accessing storage using S3-compatible Object Store

●    Verify that the NetApp array is reachable from worker nodes on the storage access methods

To setup the storage interfaces on Kubernetes nodes to access the NetApp storage using access iSCSI, NVMe-TCP, and NFS storage, complete the following procedures.

Procedure 1.     Setup iSCSI Multipathing on iSCSI interfaces

Step 1.          Create the following YAML configuration file to enable iSCSI multipathing:

Step 2.          Apply the configuration:

Procedure 2.     Configure the worker node interface to enable the NVMe-TCP Path A interface

Step 1.          Run the following configuration file to configure the NVMe-TCP-A interface – it will be a tagged VLAN on the iSCSI-A interface.

Step 2.          Convert the configuration files to .yaml format:

Step 3.          Apply the configuration:

Procedure 3.     Configure the worker node interface to enable the NVMe-TCP Path B interface

Step 1.          Run the following configuration file to configure the NVMe-TCP-B interface – it will be a tagged VLAN on the iSCSI-B interface.

Procedure 4.     Enable NVMe-TCP discovery

Step 1.          Enable NVMe-TCP discovery – it uses the previously configured interfaces:

Step 2.          Convert the configuration files to .yaml format:

Step 3.          Apply the configuration:

Procedure 5.     Configure the worker node interface to enable NFS storage access

Step 1.          Run the following configuration file to configure the NFS interface:

Step 2.          Convert the configuration files to .yaml format:

Step 3.          Apply the configuration:

Procedure 6.     Configure the worker node interface to enable the S3 Object storage access

Step 1.          Run the following configuration file to configure S3 Object interface:

Step 2.          Convert the configuration files to .yaml format:

Step 3.          Apply the configuration:

Step 4.          Over the next 10-20 minutes each of the nodes will go through the “Not Ready” state and reboot. You can monitor this by going to Compute > MachineConfigPools in the OCP Console.

Procedure 7.     Verify the connectivity to NetApp Storage

Step 1.          From the OpenShift installer machine, SSH into the first worker node.

Step 2.          For each storage access interface, verify connectivity to NetApp storage by pinging the LIFs.

Step 3.          Repeat step 1 and 2 for each worker node.

Deploy NetApp Trident Operator

NetApp Trident is an open-source, fully supported storage orchestrator for containers and Kubernetes distributions. It was designed to help meet the containerized applications’ persistence demands using industry-standard interfaces, such as the Container Storage Interface (CSI). With Trident, microservices and containerized applications can take advantage of enterprise-class storage services provided by the NetApp portfolio of storage systems. More information about Trident can be found here: NetApp Trident Documentation. There are various methods to install NetApp Trident. In this solution, we will cover the installation of NetApp Trident version 25.2.0 using the Trident Operator, which is installed using OperatorHub.

The Trident Operator is designed to manage the lifecycle of Trident. It streamlines the deployment, configuration, and management processes. The Trident Operator is compatible with OpenShift version 4.10 and later.

Procedure 1.     Install the NetApp Trident Operator from OpenShift Operator Hub

Step 1.          Log into the OCP web console and create a project with the name trident.

Step 2.          Go to Operators > OperatorHub.

Step 3.          Type Trident in the filter box and click Certified NetApp Trident Operator.

Step 4.          Click Install.

Step 5.          For Update Approval, choose Manual.

Step 6.          For Installed Namespace, click Create project and specify project name as trident.

Step 7.          Click Create.

Step 8.          Click Install.

Step 9.          Click Approve.

Step 10.       Wait for the installation to complete. Verify the installation completes successfully.

Step 11.       Click View Operator.

Step 12.       Go to Trident Orchestrator.

Step 13.       Click Create TridentOrchestrator.

Step 14.       Click Create.

Step 15.       Wait for the Status to become Installed.

Step 16.       SSH into the installer machine and verify the status of the Trident pods. All pods should be running.

Procedure 2.     Get Trident Utility Tool

Step 1.          Create a directory on the management VM, then download and untar the file to obtain trident-installer folder:

Step 2.          Copy tridentctl to /usr/local/bin:

Step 3.          Verify the trident version:

Procedure 3.     Configure the Storage Backends in Trident

Step 1.          Configure the connections to the SVM on the NetApp storage array created for the OpenShift installation. For more options regarding storage backend configuration, go to https://docs.netapp.com/us-en/trident/trident-use/backends.html.

Step 2.          Create a backends directory and create the following backend definition files in that directory:

Step 3.          Create the storage backends for all storage protocols in your FlexPod:

Step 4.          Create the following Storage Class files:

Step 5.          Create storage classes:

Step 6.          Verify the neylu created storage classes:

Step 7.          Create VolumeSnapshotClass file:

Step 8.          Verify the VolumeSnapshotClass:

Deploy GPU Operator - NVIDIA

Procedure 1.     Deploy NVIDIA Feature Discovery Operator

To deploy NVIDIA's GPU Operator in Red Hat OpenShift, Red Hat's Node Feature Discovery (NFD) Operator must first be deployed. NFD is Kubernetes add-on capability, deployed as an Operator that discover hardware-level features and expose them for use.  For nodes with NVIDIA GPUs, NFD will label the worker nodes indicating that a NVIDIA GPU is installed on that node. 

Step 1.          Log into Red Hat OpenShift's cluster console.

Step 2.          Go to Operators > Operator Hub and search for Red Hat’s Node Feature Discovery Operator.

Step 3.          Choose Node Feature Discovery Operator provided by Red Hat.

Step 4.          In the Node Feature Discovery Operator window, click Install.

Step 5.          Keep the default settings (A specific namespace on the cluster). The operator is deployed in the openshift-nfd namespace.

Step 6.          Click Install.

Step 7.          When the NFD Operator installation completes, click View Operator.

Step 8.          From the top menu, choose Node Feature Discovery.

Step 9.          Click Create NodeFeatureDiscovery.

Step 10.       Keep the default settings and click Create.

Step 11.       Confirm that the nfd-instance has a status of: Available, Upgradeable

Step 12.       To confirm that NFD labelled the worker nodes with NVIDIA GPUs correctly, go to Compute > Nodes and a choose a worker node with GPU. 

Step 13.       Go to Details and verify that the worker node has the label:

Step 14.       You can also use the following CLI commands to verify (from OpenShift installer workstation) this across all nodes:

 

Procedure 2.     Deploy the NVIDIA GPU Operator on Red Hat OpenShift

Step 1.          Log into Red Hat OpenShift's cluster console.

Step 2.          Go to Operators > Operator Hub and search for NVIDIA GPU Operator.

Step 3.          Choose NVIDIA GPU Operator tile and click Install.

Step 4.          Keep the default settings for namespace (nvidia-gpu-operator) and click Install.

Step 5.          When the installation completes, click View Operator.

Step 6.          Choose the ClusterPolicy tab, then click Create ClusterPolicy. The platform assigns the default name of gpu-cluster-policy.

Step 7.          Keep the default settings and click Create.

Step 8.          Wait for the gpu-cluster-policy status to become Ready.

Step 9.          Log into the OpenShift Installer machine and check the status of the servers with GPUs by running the following:

 

Step 10.       Connect to one of the nvidia-driver-daemonset containers and view the GPU status:

 

Procedure 3.     Enable NVIDIA GPU (vGPU) DCGM Monitoring on Red Hat OpenShift

Step 1.          Go to https://docs.nvidia.com/datacenter/cloud-native/openshift/latest/enable-gpu-monitoring-dashboard.html, enable GPU Monitoring Dashboard to monitor the GPUs from the OpenShift cluster console.

Step 2.          Log into the OpenShift cluster console.

Step 3.          Go to Observe > Dashboards.

Step 4.          Under Dashboard, from the drop-down list and choose NVIDIA DCGM Exporter Dashboard.

You can now use the OpenShift console to monitor the NVIDIA GPUs.

Procedure 4.     Setup Taints and Tolerations

Taints and Tolerations enable nodes to control which Pods are scheduled/not scheduled on them. Node affinity is a node characteristic that makes Pods prefer a set of nodes, while taints are the exact opposite, such as it repels pods from certain nodes.

Taint allows a node to refuse a pod to be scheduled on it unless it has a matching toleration. Tolerations are applied to Pods and allow Pods to be scheduled on nodes that have a matching taint. One or more taints can be applied to a node. You apply a taint using a node specification and toleration using a pod specification.

For more information, see: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/nodes/controlling-pod-placement-onto-nodes-scheduling#nodes-scheduler-taints-tolerations-about_nodes-scheduler-taints-tolerations

Step 1.          Log into the OpenShift Cluster console.

Step 2.          Go to Compute > Nodes.

Step 3.          Choose a worker node with GPU from the list.

Step 4.          Click the YAML tab.

Step 5.          Click Actions and choose Edit node from the drop-down list.

Step 6.          Paste the following in the spec: section of the configuration:

Step 7.          Click Save.

Step 8.          Go to Workloads > Pods.

Step 9.          For Project, choose nvidia-gpu-operator from the drop-down list.

Step 10.       Search and find the pod name that starts with: nvidia-driver-daemonset that is running on the worker node where you deployed the taint.

Step 11.       From the Details tab, click the pencil icon add the following if it doesn’t already exist:

Step 12.       Click Save.

Deploy Red Hat OpenShift AI for MLOps

Red Hat OpenShift AI is a complete platform for the entire lifecycle of your AI/ML projects. In this section, you will deploy Red Hat OpenShift AI as an MLOPs platform in the solution to accelerate your AI/ML projects.

Deployment Steps – Prerequisites for KServe Single-Model Serving platform

This section details the prerequisite setup required for using KServe single-model serving platform to serve large models such as Large Language Models (LLMs) in Red Hat OpenShift AI. If you’re only using OpenShift AI for multi-model serving, then you can skip this section. KServe orchestrates model serving for different types of models and includes model-serving runtimes that support a range of AI frameworks.

In this CVD, KServe is deployed in advanced deployment mode which uses Knative serverless, deployed using OpenShift Serverless Operator. Automated Install of KServe is deployed on the OpenShift cluster by configuring OpenShift AI Operator to configure KServe and its dependencies. KServe requires a cluster with a node that has at least 4 CPUs and 16GB of memory.

Procedure 1.     Deploy Red Hat OpenShift Service Mesh Operator on a OpenShift Cluster

To support KServe for single-model serving, deploy Red Hat OpenShift Service Mesh Operator on OpenShift cluster as detailed below.

Step 1.          Log into Red Hat OpenShift cluster’s web console.

Step 2.          Go to Operators > Operator Hub and search for OpenShift Service Mesh.

Step 3.          Choose and click the Red Hat OpenShift Service Mesh 2 tile.

Step 4.          Click Install.

Step 5.          Keep the default settings. The operator will be deployed in the openshift-operators namespace.

Step 6.          Click Install.

Step 7.          Click View Operator and verify that the operator deployed successfully.

Procedure 2.     Deploy Red Hat OpenShift Serverless on OpenShift cluster

To support KServe for single-model serving, deploy the Red Hat OpenShift Serverless Operator on the OpenShift cluster as detailed below.

Step 1.          Log into Red Hat OpenShift cluster’s web console.

Step 2.          Go to Operators > Operator Hub and search for OpenShift Serverless.

Step 3.          Choose and click the Red Hat OpenShift Serverless tile.

Step 4.          Click Install.

Step 5.          Keep the default settings. The operator will be deployed in a new openshift-serverless namespace.

Step 6.          Click Install.

Step 7.          Click View Operator and verify that the operator deployed successfully.

Procedure 3.     Deploy Red Hat Authorino on OpenShift Cluster

To support KServe for single-model serving, deploy the Red Hat Authoring Operator on the OpenShift cluster to add an authorization provider as detailed below.

Step 1.          Log into the Red Hat OpenShift cluster’s web console.

Step 2.          Go to Operators > Operator Hub and search for Authorino.

Step 3.          Choose and click the Red Hat – Authorino Operator tile.

Step 4.          Click Install.

Step 5.          Keep the default settings. The operator will be deployed in the openshift-operators namespace.

Step 6.          Click Install.

Step 7.          Click View Operator and verify that the operator deployed successfully.

Deployment Steps – Deploy Red Hat OpenShift AI Operator

This section details the procedures for deploying Red Hat Openshift AI on a Red Hat OpenShift cluster to enable an MLOps platform to develop and operationalize AI/ML use cases. 

Prerequisites

●    OpenShift cluster deployed with 2 worker nodes, each with at least 8 CPUs and 32 GiB RAM available for OpenShift AI to use. Additional cluster resources maybe required depending on the needs of the individual AI/ML projects supported by OpenShift AI.

●    OpenShift cluster is configured to use a default storage class that can be dynamically provisioned to provide persistent storage. This is provided by NetApp Trident in this solution. 

●    Access to S3-compatible object store with write access. In this solution, NetApp storage provides this. The object store is used by OpenShift AI for use as:

◦     Model Repo to store models that will used for model serving in inferencing use cases

◦     Pipeline Artifacts to store data science pipeline runs logs, results and other artifacts or metadata.

◦     Data storage to store large data sets that maybe used by data scientists to test or experiment with.

◦     Input or Output data for Distributed Workloads 

●    Identity provider configured for OpenShift AI (same as Red Hat OpenShift Container Platform). You cannot use OpenShift administrator (kubeadmin) for OpenShift AI. You will need to define a separate user with cluster-admin role to access OpenShift AI.

●    Internet Access, specifically access to the following locations. cdn.redhat.com

◦     subscription.rhn.redhat.com

◦     registry.access.redhat.com

◦     registry.redhat.io

◦     quay.io

●    If using NVIDIA GPUs and other NIVIDA resources, then above access should include:

◦     ngc.download.nvidia.cn

◦     developer.download.nvidia.com

●    Verify that the following perquisites from the previous section have been successfully deployed. The following are required to support the different uses cases that were validated as a part of this solution. See Solution Validation section of this document for more details on these use cases.

◦     Red Hat OpenShift Serverless Operator to support single-model serving of large models using Kserve.

◦     Red Hat OpenShift Service Mesh to support single-model serving.

◦     Red Hat Authorino Operator to add an authorization provider to support single-model serving.

Procedure 1.     Deploy Red Hat OpenShift AI Operator on the OpenShift Cluster

Step 1.          Log into the Red Hat OpenShift cluster console.

Step 2.          Go to Operators > Operator Hub and search for OpenShift AI.

Step 3.          Click the Red Hat OpenShift AI tile.

Step 4.          Click Install.

Step 5.          Keep the default settings. The operator will be deployed in the redhat-ods-operator namespace.

Step 6.          Click Install.

Step 7.          When the installation completes, click Create DataScienceCluster.

Step 8.          For Configure via, choose YAML view.

Step 9.          Review the OpenShift AI components under spec > components. Verify that kserve component’s managementState is Managed.

Step 10.       Click Create.

Step 11.       When the installation completes, view the operator status by clicking the All instances tab. It should have a status of Ready.

Procedure 2.     Set up the OpenShift AI admin user

Before you can use OpenShift AI, you will need to setup an OpenShift AI admin user in order to log in and setup the environment as an OpenShift AI administrator.

Step 1.          Log into the OpenShift Installer machine and go to the cluster directory.

Step 2.          Run the following command to create an user with administrator privileges:

 

Step 3.          Copy the contents of the admin.htpasswd and log into the OpenShift cluster console.

Step 4.          You will see a message as shown below, indicating that you’re logged in as a temporary administrative user. Click the link to update the cluster OAuth configuration.

Step 5.          In the Cluster OAuth configuration window, for IDP, choose HTPasswd from the drop-down list.

Step 6.          In the Add Identify Provider:HTPasswd window, paste the contents of the admin.htpasswd file as shown below:

Step 7.          Click Add.

Step 8.          Go to User Management > Users.

Step 9.          Choose the user that was previously created using htpasswd. Click the username.

Step 10.       In the User > User Details window, go to the RoleBindings tab.

Step 11.       Click Create binding.

Step 12.       In the Create Rolebinding window, click Cluster-wide role binding (ClusterRoleBinding).

Step 13.       Specify a name, such as oai-admin.

Step 14.       For Role Name, choose cluster-admin from the drop-down list.

Step 15.       Click Create.

Step 16.       Log into the OpenShift AI. If you were previously logged in, you may want to open a browser window in Incognito mode to log in using the new account.

Step 17.       To log into OpenShift AI, click the square tile and choose Red Hat OpenShift AI from the drop-down list.

Step 18.       Click htpasswd.

Step 19.       Log in using the new account and now you’ll see the Settings since you’re an administrator. You can now start setting up and use the environment for your AI/ML projects.

Visibility and Monitoring – GPU

The GPUs deployed on Cisco UCS systems in the solution can be observed and monitored using the tools detailed in this section.

Visibility

Cisco Intersight allows you to manage Cisco UCS servers, including the GPU or PCIe (X440p) nodes with GPUs. Each GPU node is paired with an adjacent compute server as shown below:

 

In addition to centralized provisioning and orchestration that Cisco Intersight provides, it also provides visibility across all sites and locations that an enterprise has. Enterprises can use either built-in or custom dashboards. For example, power and energy consumption is a critical consideration in AI/ML deployments and a dashboard such as the one shown below can help Enterprises understand their consumption pattern more efficiently.

Power and energy statistics on a per server is also possible and important to track for GPU workloads.

Monitoring

To monitor the GPU utilization, memory, power and metrics, the solution uses the following tools to get a consolidated view. Alternatively, a Grafana dashboard can be set up to enable a consolidated view – this is outside the scope of this solution.

●    Red Hat OpenShift observability dashboard available from the OpenShift cluster console

●    nvidia-smi CLI tool that NIVIDA provides for Red Hat OpenShift

GPU Monitoring from Red Hat OpenShift Dashboard

The OpenShift dashboard uses Prometheus metrics. NVIDIA GPU Operator exposes DCGM metrics to Prometheus that the dashboard uses to display GPU metrics available to OpenShift. NVIDIA GPU Operator, when deployed will expose DCGM metrics to the OpenShift that can be viewed from the integrated dashboard.

To view the metrics exposed by DCGM exporter in OpenShift, see the following file available here. When creating custom dashboards using Grafana, the exact metric and query to use can be found here. Also, a JSON file with the metrics is available for Grafana from the same repo. Grafana community operator is available on Red Hat Operator Hub and can be deployed as needed for a customized view.

The OpenShift dashboard currently displays the following default metrics for a GPU (Table 15).

Table 15.     OpenShift Metrics

NVIDIA CLI tool for Red Hat Openshift 

NVIDIA provides the nvidia-smi CLI tool to collect GPU metrics and other details from the GPU in OpenShift and as outlined below:

In Red Hat OpenShift, execute the following commands from OpenShift installer workstation:

oc exec -it nvidia-driver-daemonset-<version> -- nvidia-smi (-q)

Solution Validation

This chapter contains the following:

●    Hardware and Software Matrix

●    Interoperability Matrices

●    GPU Functional/Load Tests

●    Validated AI/ML Use Cases

●    Link to GitHub Repo

Hardware and Software Matrix

Table 16 lists the hardware and software components that were used to validate the solution in Cisco labs.

Table 16.     Hardware/Software Matrix

* Red Hat OpenShift AI versions were upgraded as we progressed through the testing. The version shown in the table is the version running at the time of the writing of this guide.

Interoperability Matrices

The interoperability information for the different components in the solution are summarized in Table 17.

Table 17.     Interoperability

GPU Functional/Load Tests

The following GPU focused validation was completed:

●    GPU Functional Validation – Sample CUDA Application.

●    GPU Stress/Load Test using GPU Burn Tests from: https://github.com/wilicc/gpu-burn. The test iterates up to max. GPU utilization to ensure that the GPU is performing (Tflop/s) as it should before we add AI/ML workloads to Red Hat OpenShift.

●    Same PyTorch script executed from Jupyter Notebooks on Red Hat OpenShift – see the Sample GPU Tests folder in https://github.com/ucs-compute-solutions/FlexPod-OpenShift-AI

The following sections show the results of the above-mentioned sanity tests.

Sample CUDA Application Test

Configuration YAML file:

Sample GPU Burn Test

The specifics of this test can be found in the GitHub repo provided earlier. Results of executing the test are provided in this section.  

Validated AI/ML Use Cases

A summary of the AI/ML use cases and other testing that were validated for this effort are listed below. The code for these use cases will be made available on GitHub repo (link below) in the near future.

●    Fraud Detection - Basic validation of a simple model using TensorFlow across the MLOps lifecycle. This model is an example in which transactions are analyzed using previous labeled data as either fraudulent or valid. This model would generally be called as part of real time transaction processing in financial institutions.

●    Object Detection - Validation of a more advanced predictive AI Model. In this case we used PyTorch and YOLOv8 object detection. Starting from an open-source model, we retrain that model on new labeled data giving the ability for the model to detect car accidents. While not necessary, we see the benefits of using GPUs to reduce training time. As a service API, this model could be consumed by applications using traffic cameras to detect accidents or other uses.

●    Text to Image - Used to determine viability of Generative AI on the platform. Throughout the MLOps lifecycle, GPU accelerators are required.  We started stable diffusion for image generation using PyTorch and CUDA 11.8. The demo involves fine-tuning the model using a small amount of custom data, exporting, and saving model in ONNX format to a model repo, andoperationalizing the model into production using an inferencing server for use by application teams.

●    NVIDIA Inference Microservices (NIM) – Validation of NVIDIA NIM on OpenShift AI represents a powerful integration aimed at optimizing AI inferencing and accelerating the deployment of generative AI applications. The integration supports performance-optimized foundation and embedding models, enabling faster time to value for AI applications.

●    RAG Chatbot – Validation of an Enterprise chatbot using RAG. Milvus Vector store will be first deployed. A sample collection of Enterprise data will then be ingested into a Milvus Vector store with LangChain and Hugging Face embedding model. A vLLM Inferencing server will be deployed with Mistral-7b-Instruct or an equivalent LLM. We will then use input from the chatbot (built using Gradio) to generate a query to vector store and query output, along with original Chatbot input will be used to generate an inferencing request to the LLM running on the inferencing server and the response deployed on the chatbot GUI

●    Fine-Tune pre-trained Llama3.1 models – Validation of fine-tuning LLMs using Ray on OpenShift AI. Ray is a distributed computing framework, and the Kubernetes operator for Ray makes it easy to provision resilient and secure Ray clusters that can leverage the compute resources available on any infrastructure. The model will be fine-tuned using HF transformers, Accelerate, PEFT (LoRA), DeepSpeed and a training example from Ray.

Link to GitHub Repo

The use case code for the validated use cases and automation used in this solution will be made available on GitHub in the future: https://github.com/ucs-compute-solutions/FlexPod-OpenShift-AI

Conclusion

Taking AI/ML projects from proof-of-concept to production is a significant challenge for organizations due to the complexity involved. Even with a production-ready ML model, integrating it into an enterprise application and data pipelines are challenging. For an Enterprise, scaling these efforts across multiple applications requires an operational framework that is sustainable.

A critical strategic decision that enterprises can make from the get-go is to have a plan for operationalizing their AI/ML efforts with consistency and efficiency. Instead of ad-hoc efforts that add to the technical debt, adopting processes, tools, and best-practices is essential for delivering models quickly and efficiently. Implementing MLOps is a crucial step toward this goal. MLOps integrates successful DevOps practices into machine learning, promoting collaboration, automation, and CI/CD to speed up model delivery.

The FlexPod AI solution using Red Hat OpenShift AI offers a comprehensive, flexible, and scalable platform for supporting an Enterprise’s AI/ML efforts.  Red Hat OpenShift AI provides both pre-integrated and custom tools and technologies to accelerate AI/ML efforts and operationalize AI in a repeatable manner, with consistency and efficiency.

The FlexPod infrastructure is a proven platform in enterprise data centers, delivering a high-performance and flexible architecture for demanding applications such as SAP, Oracle, HPC and graphics-accelerated VDI. Cisco Intersight, a SaaS platform, simplifies IT operations with comprehensive management and visibility across the FlexPod datacenter, including GPUs and sustainability dashboards. FlexPod AI extends existing FlexPod datacenter capabilities to streamline AI infrastructure deployments and accelerate AI/ML efforts by reducing complexity.

This CVD provides a comprehensive solution for hosting AI/ML workloads in enterprise data centers, enabling enterprises to accelerate and operationalize AI/ML efforts quickly and at scale.

About the Authors

Archana Sharma, Technical Marketing, Cisco UCS Compute Solutions, Cisco Systems Inc.

Archana Sharma is a Technical Marketing Engineer with over 20 years of experience at Cisco on a variety of technologies that span Data Center, Desktop Virtualization, Collaboration, and other Layer2 and Layer3 technologies. Archana currently focusses on the design and deployment of Cisco UCS based solutions for Enterprise data centers, specifically Cisco Validated designs and evangelizing the solutions through demos and industry events such as Cisco Live. Archana holds a CCIE (#3080) in routing and switching and a bachelor’s degree in electrical engineering from North Carolina State University.

Abhinav Singh, Sr. Technical Marketing Engineer, Hybrid Cloud Infra & OEM Solutions, NetApp

Abhinav Singh is a Senior Technical Marketing Engineer for the Converged Infrastructure Solutions team at NetApp, who has over 15 years of expertise in Data Center Virtualization, Networking, and Storage. Abhinav specializes in designing, validating, implementing, and supporting Converged Infrastructure solutions, encompassing Data Center Virtualization, Hybrid Cloud, Cloud Native, Database, Storage, and Gen AI. Abhinav holds a bachelor's degree in electrical and electronics engineering.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

●    John George, Technical Marketing Engineer, Cisco Systems, Inc.

●    Karl Eklund, Principal Consultant – AI, Red Hat

●    Roberto Carratalá, Principal AI Platform Architect, Red Hat

Appendix

This appendix contains the following:

●    Appendix A – References used in this guide

Appendix A – References used in this guide

FlexPod

Cisco Design Zone for FlexPod CVDs: https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/data-center-design-guides-all.html#FlexPod

Cisco Unified Computing System (UCS)

Cisco UCS Hardware Compatibility Matrix: https://ucshcltool.cloudapps.cisco.com/public/  

Cisco Intersight: https://www.intersight.com

Cisco Intersight Managed Mode: https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Intersight_Managed_Mode_Configuration_Guide.html 

Cisco Unified Computing System: http://www.cisco.com/en/US/products/ps10265/index.html

Cisco UCS 6536 Fabric Interconnects: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs6536-fabric-interconnect-ds.html

Cisco Network

Cisco Nexus 9000 Series Switches: http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html

NVIDIA GPUs

Licensing: https://resources.nvidia.com/en-us-ai-enterprise/en-us-nvidia-ai-enterprise/nvidia-ai-enterprise-licensing-guide?pflpid=5224&lb-mode=preview

Certification: https://www.nvidia.com/en-us/data-center/products/certified-systems/

NetApp Storage

NetApp IMT: https://mysupport.netapp.com/matrix/#welcome

NetApp AFF C-Series: https://www.netapp.com/data-storage/aff-c-series/

NetApp AFF A-Series: https://www.netapp.com/data-storage/aff-a-series

NetApp ASA: https://www.netapp.com/data-storage/all-flash-san-storage-array/

NetApp FAS: https://www.netapp.com/data-storage/fas/

NetApp ONTAP: https://docs.netapp.com/ontap-9/index.jsp

NetApp Trident: https://docs.netapp.com/us-en/trident/

NetApp Active IQ UM: https://docs.netapp.com/us-en/active-iq-unified-manager/

Red Hat OpenShift

Red Hat OpenShift Operators: https://www.redhat.com/en/technologies/cloud-computing/openshift/what-are-openshift-operators

Red Hat OpenShift Ecosystem catalog: https://catalog.redhat.com/software/search?deployed_as=Operator

Automation

GitHub repository for Cisco UCS solutions: https://github.com/ucs-compute-solutions/

Feedback

For comments and suggestions about this guide and related guides, join the discussion on Cisco Community here: https://cs.co/en-cvds.

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_P1)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)