FlexPod Datacenter with SUSE Rancher for AI Workloads

 

Published Date: May 2024

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: http://www.cisco.com/go/designzone.

Executive Summary

The FlexPod Datacenter solution is a validated approach for deploying Cisco^® and NetApp technologies and products to build shared private and public cloud infrastructure. Cisco and NetApp have partnered to deliver a series of FlexPod solutions that enable strategic data-center platforms. The success of the FlexPod solution is driven by its ability to evolve and incorporate both technology and product innovations in the areas of management, computing, storage, and networking. To help organizations with their digital transformation and application modernization practices, Cisco and NetApp have partnered to produce this Cisco Validated Design (CVD) for the FlexPod™ Datacenter for SUSE Rancher and Rancher Kubernetes Engine (RKE2) solution. As the hybrid-cloud operation is the new de-facto default for many companies, the network connection to the public cloud, the Kubernetes cluster management, and the workload management across on-premises and public clouds are covered as part of this solution design.

FlexPod delivers an integrated architecture that incorporates compute, storage, and network design best practices, thereby minimizing IT risks by validating the integrated architecture to ensure compatibility between various components. The solution also addresses IT pain points by providing documented design guidance, deployment guidance, and support that can be used in various stages (planning, designing, and implementation) of a deployment. FlexPod delivered as Infrastructure as Code (IaC) further eliminates error-prone manual tasks, allowing quicker and more consistent solution deployments.

SUSE® Rancher Enterprise Container Management is an enterprise-ready Kubernetes container platform management with full-stack automated operations to manage hybrid cloud and multi-cloud deployments. SUSE Rancher is optimized to improve developer productivity and promote innovation. The SUSE Rancher gives developers a self-service platform to build and run containerized applications. With SUSE Rancher, you can quickly start creating new cloud-native applications or cloud-enabling existing applications and spawning an environment for a new microservice in minutes.

Combining SUSE Rancher with the FlexPod solution can simplify the deployment and management of the container infrastructure. The Ansible integration with the FlexPod solution automates the deployment of the FlexPod infrastructure enabling you to take advantage of programming and automating the infrastructure at scale with agility, extending the benefits of automation to the entire stack.

Some of the key advantages of integrating Cisco FlexPod Datacenter as a workload domain into SUSE Rancher are:

·     Simpler and programmable infrastructure: FlexPod infrastructure delivered as infrastructure-as-a-code through a single partner integrable open API.

·     Latest hardware and software compute innovations: policy-based configurations, delivered using Cisco Intersight, to deploy and manage the latest processor, memory, network, and power/cooling improvements.

·     Storage Modernization: deliver high-speed, consistent, low latency, multi-tenant storage using a range of NetApp all-flash arrays.

·     Innovative cloud operations: continuous feature delivery and no need for maintaining on-premises physical or virtual machines supporting management functions.

·     Built for investment protections: design ready for future technologies such as liquid cooling and high-Wattage CPUs; CXL-ready.

The FlexPod solution includes integration of the Cisco Intersight with NetApp Active IQ Unified Manager, and if required VMware vCenter to deliver monitoring, orchestration, and workload optimization capabilities for different layers (virtualization and storage) of the FlexPod infrastructure. The modular nature of the Cisco Intersight platform also provides an easy upgrade path to additional services, such as Intersight Workload Optimization.

The solution provided in this document was validated on distinct types of Cisco UCS servers to ensure functionality in heterogeneous landscapes, like expansions of existing Cisco UCS B-Series blade-based systems with Cisco UCS X-Series nodes or using M5, M6 and M7 generation servers in the same setup. The diagrams shown in this document are focused on X-Series based deployments. The configuration principles are valid for any type of Cisco UCS servers.

If you’re interested in understanding the FlexPod design and deployment details, including the configuration of various elements of design and associated best practices, refer to the Cisco Validated Designs for FlexPod, here: https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/flexpod-design-guides.html.

Solution Overview – FlexPod Datacenter with SUSE Rancher

This chapter contains the following:

·     Introduction

·     Audience

·     Purpose of this Document

·     What’s New in this Release?

Introduction

The featured FlexPod Datacenter for SUSE Rancher Enterprise Container Manager solution is a pre-designed, integrated, and validated architecture for the data center that combines Cisco UCS servers, the Cisco Nexus family of switches, and NetApp AFF A-series storage into a single, flexible architecture. FlexPod is designed for high availability (HA), with no single point of failure, while maintaining cost-effectiveness and flexibility in the design to support a wide variety of workloads. The SUSE Rancher software provides multiple ways of deployment, this document is focused on only one. The FlexPod solution with SUSE Rancher in this deployment guide was tested and validated with Cisco UCS M5, M6 and M7 servers managed by Intersight Managed Mode (IMM) and leveraging local disks for the SUSE Linux Enterprise (SLE) and SLE Micro operating systems.

Integration between the SUSE Rancher/RKE2 and the NetApp storage and data management services occurs at several levels. The main storage integration is based on Container Storage Interface (CSI) Astra Trident for Kubernetes Driver for NetApp storage systems, which enables container orchestrators such as Kubernetes to manage the life cycle of persistent storage.

The focus of the FlexPod Datacenter with SUSE Rancher solution is on a bare metal cluster with the RKE2 nodes running SUSE Linux Enterprise on Cisco UCS servers. Virtualized RKE2 clusters and single-node deployments are validated to better support smaller deployments and Test/Dev installations.

The following deployment steps for the FlexPod Datacenter for the SUSE Rancher solution are explained in this document:

·     Hardware deployment.

·     SUSE Linux Enterprise (SLE) 15

·     SUSE Rancher Kubernetes Engine (RKE2) cluster

·     SUSE Rancher Enterprise Container Manager

·     CSI Astra Trident for Kubernetes

·     Optional deployment with SUSE SLE Micro and K3s

·     NVIDIA GPU Driver and CUDA Toolkit

·     NetApp DataOps Toolkit

Note:      The infrastructure deployment is the same as documented in the FlexPod Datacenter with End-to-End 100G, Cisco Intersight Managed Mode, VMware 7U3, and NetApp ONTAP 9.11 Deployment Guide.

All configurations on the Cisco UCS, Nexus switches, and NetApp AFF storage for SUSE Rancher are in addition to the documented configuration for VMware. This configuration simplifies the deployment of different use-cases on the same FlexPod and in parallel help Cisco TAC support customers in case of an issue.

Audience

The intended audience of this document includes but is not limited to IT architects, sales engineers, field consultants, professional services, IT managers, partner engineering, and customers who want to take advantage of an infrastructure built to deliver IT efficiency and enable IT innovation.

Purpose of this Document

This document provides deployment guidance around incorporating the Cisco Intersight—managed UCS platform within FlexPod Datacenter infrastructure to run SUSE Rancher Enterprise Container Manager and SUSE Rancher Kubernetes Engine (RKE) cluster and NVIDIA GPUs to enable AI/ML workloads. The document covers various considerations and best practices for a successful deployment. The document also highlights the product requirements for integrating virtualization and storage systems to Cisco Intersight to deliver a true cloud-based integrated approach for infrastructure management.

What’s New in this Release?

The following elements distinguish this version of FlexPod from previous models:

·     Cisco Intersight Infrastructure Manager

·     NetApp ONTAP 9.13.1 P6

·     SUSE Rancher Enterprise Container Manager

·     SUSE Rancher Kubernetes Engine (RKE)

·     NetApp Astra Trident

·     NVIDIA Datacenter GPUs

·     NVIDIA CUDA-Toolkit

·     NetApp DataOps Toolkit

Hardware Deployment

This chapter contains the following:

·     Requirements

·     Physical Topology

·     Ansible Automation for Solution Deployment

·     Switch Configuration

·     Storage Configuration

Requirements

This section explains the key design requirements and various prerequisites for delivering this new solution.

The FlexPod Datacenter with SUSE Rancher solution closely aligns with all FlexPod CVDs and meets the following general design requirements:

·     Resilient design across all infrastructure layers with no single point of failure.

·     Scalable design with the flexibility to add compute capacity, storage, or network bandwidth as needed.

·     Modular design that can be replicated to expand and grow as the needs of the business grow.

·     Flexible design that can support components beyond what is validated and documented in this guide.

·     Simplified design with the ability to automate and integrate with external automation and orchestration tools.

For SUSE Rancher integration into a traditional FlexPod solution, the following specific design considerations are observed:

·     Deployment option for one or three master nodes, where the opportunity with one master is only recommended for non-production installations.

·     A minimum of 2 worker nodes with the ability to increase the nodes as the load requirements increase.

·     Automating the FlexPod infrastructure deployment and RKE2 installation by utilizing Ansible Playbooks to simplify the installation and reduce the deployment time.

·     Present persistent storage (volumes) to the containerized applications using the NetApp Astra Trident CSI framework.

·     Dedicated Cisco UCS vNICs for different traffic needs with UCS Fabric Failover for high availability.

This solution uses the same hardware components and initial configuration as the solution documented here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html.

Note:      This document refers to the required information to finish a task. Some VLAN IDs and used names differ between the two documents, as those information are used/defined as variables in both documents, there is no impact to the deployment of the solution.

Physical Topology

This FlexPod design utilizes Cisco UCS servers connected and managed through Cisco UCS Fabric Interconnects and the Intersight Infrastructure Manager (IMM) to manage the servers. These high-performance servers are configured as compute nodes where SUSE Linux Enterprise Micro is loaded using local disk boot leveraging a RAID1 volume provided by M.2 controller and two SATA SSDs. The persistent storage volumes for containers are provisioned on the NetApp AFF A400 using NFS NAS storage. Optionally, you can use iSCSI storage.

IP- and FC-based Storage Access: FC, iSCSI, and NFS

A typical topology for the FlexPod Datacenter is shown in Figure 1.

Figure 1.       FlexPod Physical Topology

To build an IP only storage access in a FlexPod configuration, the components are set up as follows:

·     Cisco UCS Fabric Interconnects provide chassis and network connectivity.

·     The Cisco UCS X9508 Chassis connects to fabric interconnects using Cisco UCS X9108 Intelligent Fabric Module, where four 25 Gigabit Ethernet ports are used on each IFM to connect to the appropriate Fabric Interconnect. If additional bandwidth is required, all eight 25G ports can be utilized.

·     Cisco UCS X210c M6 Compute Nodes contain fourth-generation Cisco 14425 virtual interface cards (VICs).

·     Cisco UCS C220 or C240 Servers with fourth generation VICs 1457 VICs connect to the fabric interconnects with 25GE.

·     Cisco Nexus 93180YC-FX3 Switches in Cisco NX-OS mode provide the switching fabric.

·     Optional: Cisco MDS 9132T FC Switches provide the SAN switching fabric.

·     Cisco UCS 6454 Fabric Interconnect 25-Gigabit Ethernet uplink ports connect to Cisco Nexus 93180YC-FX3 Switches in a Virtual Port Channel (vPC) configuration. Optional uplink connection to Cisco MDS 9132T FC Switches via Port Channel (PC) configuration.

·     The NetApp AFF A400 controllers connect to the Cisco Nexus 93180YC-FX3 Switches using two 100 GE ports from each controller configured as a vPC. Optional connection to the Cisco MDS 9132T SAN Switches.

·     SUSE software is installed on Cisco UCS Compute Nodes with local disks to validate the infrastructure.

Note:      Since Cisco UCS C-series is being managed and configured by Cisco Intersight Managed Mode, the nodes must satisfy the software and hardware requirements outlined here: https://intersight.com/help/saas/supported_systems

VLAN Configuration

Table 1 lists VLANs configured for setting up the FlexPod environment along with their usage.

Table 1.          VLAN Usage

* iSCSI is listed only for the VMware part.
**Infrastructure and VMware related VLANs are listed as reference as they are used to setup the infrastructure with this CVD:  https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html

Some of the key highlights of VLAN usage are as follows:

·     VLAN 3072 allows you to manage and access out-of-band management interfaces of various devices and is brought into the infrastructure to allow CIMC access to the UCS servers and is also available to infrastructure virtual machines (VMs). Interfaces in this VLAN are configured with MTU 1500.

·     VLAN 17 is used for in-band management of VMs, hosts, and other infrastructure services. Interfaces in this VLAN are configured with MTU 1500.

·     VLAN 3017 provides ESXi hosts, management systems, and other infrastructure services access to the NFS storage hosted on the NetApp Controllers. Interfaces in this VLAN are configured with MTU 9000.

◦    Optional: A pair of iSCSI VLANs (3117 and 3217) is configured to provide access to iSCSI based storage for data access. Interfaces in these VLANs are configured with MTU 9000.

·     VLAN 1043 is used as an access network for RKE2 cluster 1 to access all RKE2 hosts, and services deployed on top. Interfaces in this VLAN are configured with MTU 1500.

·     VLAN 1047 provides services deployed on top of RKE2 cluster 1 access to the NFS storage hosted on the NetApp Controllers managed by NetApp Astra Trident CSI. Interfaces in this VLAN are configured with MTU 9000.

·     VLAN 2043 is used as an access network for RKE2 cluster 2 to access all RKE2 hosts, and services deployed on top. Interfaces in this VLAN are configured with MTU 1500.

·     VLAN 2047 provides services deployed on top of RKE2 cluster 2 access to the NFS storage hosted on the NetApp Controllers managed by NetApp Astra Trident CSI. Interfaces in this VLAN are configured with MTU 9000.

Physical Components

Table 2 lists the hardware components used to build the solution in the lab. You are encouraged to review your requirements and adjust the various components as needed.

Table 2.          FlexPod Hardware Components

Software Revisions

Table 3 lists software versions of the components, drivers, and software (for example, various NetApp software tools, Cisco Intersight Assist, and so on) used to build this solution.

Table 3.          Software Components and Versions

Virtual Machines

Table 4 lists the required components deployed as virtual machines to build the validated solution. You are encouraged to review the list and adjust the deployment type, the size or quantity of various components as needed.

Table 4.                Virtual Machines were used for this solution

Ansible Automation for Solution Deployment

This section provides information about setting up and running Ansible playbooks to configure the infrastructure for the SUSE Rancher and RKE solution. As the focus of this solution validation was on the software and workload, the infrastructure setup is based on this document: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_imm_m7_iac.html#AnsibleAutomationWorkflowandSolutionDeployment

For the deployment process, see: https://developer.cisco.com/codeexchange/github/repo/ucs-compute-solutions/FlexPod-IMM-VMware/. This document will guide you through the deployment process and highlight the additions required to make the SUSE Rancher and RKE software stack work. Ansible automation requires a management workstation (control machine) to run Ansible playbooks for configuring Cisco Nexus, NetApp ONTAP Storage, Cisco UCS, and SUSE Rancher/RKE.

Management Workstation

A management workstation is a VM where Ansible is installed and has access to the Internet to download various packages and clone the playbook repository. Instructions for installing the workstation Operating System (OS) or complete setup of Ansible are not included in this document; however, basic installation and configuration of Ansible are provided as a reference. A guide for installing and getting started with Ansible can be found at: https://docs.ansible.com/ansible_community.html.

Procedure 1.             Prepare Management Workstation

In this procedure, the installation steps are performed on a virtual machine running openSUSE Leap 15.5 to prepare the host for automated solution deployment. The automation includes the configuration of Cisco UCS, Cisco Nexus, NetApp Storage, and SUSE Rancher using Ansible Playbooks. The following steps were performed as root user.

Step 1.                         Install Python 3.11:

Step 2.                         Verify Python version:

Step 3.                         Install Ansible:

Step 4.                         Verify Ansible version:

Step 5.                         Install git:

Step 6.                         Update pip and setup tools:

Step 7.                         Install NetApp specific modules:

Step 8.                         Install ansible-galaxy collections for Cisco UCS, NX-OS, NetApp, and VMware as follows:

 

Ansible Playbooks

To download the Ansible playbooks for configuring the infrastructure, the management workstation needs a working installation of Git and access to the public GitHub repository. You can also manually download the repository and copy the files to the management workstation. The Ansible playbooks used in this document can be found at the following links:

Cisco DevNet: https://developer.cisco.com/codeexchange/github/repo/ucs-compute-solutions/FlexPod-IMM-VMware/

GitHub repository: https://github.com/ucs-compute-solutions/FlexPod-IMM-VMware

The Cisco Nexus Switches, NetApp Storage and Cisco UCS must be physically racked, cabled, powered, and configured with management IP addresses before the Ansible-based installation procedure can begin. Upgrade the Cisco UCS, Nexus Switches to appropriate software versions listed in Table 3.

Before executing the Ansible playbooks to setup various devices, several variables must be updated based on your specific implementation. These variables contain values such as the interfaces, interface numbering, VLANs, pools, policies and ports on Cisco UCS, IP addresses and interfaces for storage and so on.

Note:      Day 2 Configuration tasks such as adding additional VLAN, datastores, Virtual Machines and so on, can be performed manually or with Cisco Intersight Cloud Orchestrator (ICO).

Procedure 1.             Clone GitHub Collection

To copy the github repository for the project, clone the collection to a new (empty) folder on the management workstation. Cloning the repository creates a local copy, which is then used to modify and run the playbooks.

Step 1.                         From the management workstation, create a new folder for the project. The GitHub collection will be cloned to this new folder.

Step 2.                         Open a command-line or console interface on the management workstation and change directories to the newly created folder.

Step 3.                         Clone the GitHub collection using the following command:

Switch Configuration

This section provides the procedure for configuring the Cisco Nexus 93180YC-FX3 switches used for ethernet LAN switching in this solution. The switch configuration for this validated design is based on the switching configuration covered in FlexPod Datacenter with Cisco UCS X-Series Cisco Validated Design (CVD): https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#NetworkSwitchConfiguration therefore this section only explains the changes from the base CVD.

Physical Connectivity

Follow the physical connectivity guidelines for FlexPod as explained in section FlexPod Topology.

Initial Configuration

For setting up the initial switch configuration, complete the steps explained here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#InitialConfiguration

Manual Configuration Steps

The required configuration steps for the FlexPod solution are documented here:
https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#CiscoNexusSwitchManualConfiguration.

Procedure 1.             Create VLANs

Cisco Nexus A and Cisco Nexus B

In addition to the existing VLANs, follow this step on both switches. Refer to VLAN information in VLAN Configuration for setting up all the required VLANs.

Step 1.                         From the global configuration mode, run the following commands:

Procedure 2.             Create Port Channel Parameters

Cisco Nexus A and Cisco Nexus B

To configure port channel parameters, follow the steps on both Cisco Nexus switches.

Step 1.                         From the global configuration mode, run the following commands to setup VPC Peer-Link port-channel:

Step 2.                         From the global configuration mode, run the following commands to setup port-channels for UCS FI 6454 connectivity:

Step 3.                         From the global configuration mode, run the following commands to setup port-channels for NetApp A400 connectivity:

Step 4.                         From the global configuration mode, run the following commands to setup port-channels for connectivity to existing management switch:

Step 5.                         From the global configuration mode, run the following commands to setup port-channels for connectivity to existing datacenter switches:

Procedure 3.             Configure IP Gateways

To enable internet access for the installation and the workloads we have enabled IP routing and HSRP function on the Nexus Switches. If IP gateways for the VLANs covered below are present on the upstream switches, the configuration in this step can be skipped. If some or all the gateways are not configured, use Hot Standby Router Protocol (HSRP) and Switched Virtual Interface (SVI) on the Nexus switches to set up gateways for Out-of-band management*, in-band management*, RKE2 access networks.

Note:      *Gateways for management networks will be pre-configured in your existing environments.

Configure Nexus-A Switch

Step 1.                         From the global configuration mode, run the following commands to setup IP addresses and HSRP:

Configure Nexus-B Switch

Step 1.                         From the global configuration mode, run the following commands to setup IP addresses and HSRP:

Storage Configuration

NetApp AFF A400 Controllers

See the following section (NetApp Hardware Universe) for planning the physical location of the storage systems:

·        Site Preparation

·        System Connectivity Requirements

·        Circuit Breaker, Power Outlet Balancing, System Cabinet Power Cord Plugs, and Console Pinout Requirements

·        AFF Series Systems

NetApp Hardware Universe

The NetApp Hardware Universe (HWU) application provides supported hardware and software components for any specific ONTAP version. It also provides configuration information for all the NetApp storage appliances currently supported by ONTAP software and a table of component compatibilities.

To confirm that the hardware and software components that you would like to use are supported with the version of ONTAP that you plan to install, follow these steps at the NetApp Support site:

1.    Access the HWU application to view the System Configuration guides.

2.    Click the Products tab to select Platforms menu to view the compatibility between different versions of the ONTAP software and the NetApp storage appliances with your desired specifications.

3.    Alternatively, to compare components by storage appliance, click Utilities and select compare Storage Systems.

Controllers

Follow the physical installation procedures for the controllers found here: https://docs.netapp.com/us-en/ontap-systems/index.html.

Disk Shelves

NetApp storage systems support a wide variety of disk shelves and disk drives. The complete list of disk shelves that is supported by the AFF A400 is available at the NetApp Support site.

When using SAS disk shelves with NetApp storage controllers, refer to: https://docs.netapp.com/us-en/ontap-systems/sas3/install-new-system.html for proper cabling guidelines.

When using NVMe drive shelves with NetApp storage controllers, refer to: https://docs.netapp.com/us-en/ontap-systems/ns224/hot-add-shelf.html for installation and servicing guidelines.

NetApp ONTAP Configuration

Complete the NetApp AFF A400 initial cluster setup explained here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_imm_m7_iac.html#NetAppONTAPStorageConfiguration

FC, NVMe, iSCSI, and FC Boot have not been explained in the CVD since the current solution deployment contains Boot from Local Disk and implements NFS services to provide IP-based storage access for FlexPod. At the completion of this step, NetApp A400 management connectivity and ONTAP cluster setup is ready.

Procedure 1.             Log into the Cluster

Step 1.                         Open an SSH connection to either the cluster IP or the host name.

Step 2.                         Log into the admin user with the password provided during cluster setup.

Procedure 2.             Verify Storage Failover

To confirm that storage failover is enabled, run the following commands for a failover pair:

Step 1.                         Verify the status of the storage failover:

Note:      Both <st-node01> and <st-node02> must be capable of performing a takeover. Continue with Step 2 if the nodes can perform a takeover.

Step 2.                         Enable failover on one of the two nodes if it was not completed during the installation:

Note:      Enabling failover on one node enables it for both nodes.

Step 3.                         Verify the HA status for a two-node cluster.

Note:      This step is not applicable for clusters with more than two nodes.

Step 4.                         If HA is not configured use the below commands. Only enable HA mode for two-node clusters.

Note:      Do not run this command for clusters with more than two nodes because it causes problems with failover.

Step 5.                         Verify that hardware assist is correctly configured:

Procedure 3.             Set Auto-Revert on Cluster Management Interface

To set the auto-revert parameter on the cluster management interface, follow this step:

Step 1.                         Run the following command:

Note:      A storage virtual machine (SVM) is referred to as a Vserver or vserver in the GUI and CLI.

Procedure 4.             Zero All Spare Disks

Step 1.                         To zero all spare disks in the cluster, run the following command:

Note:      Advanced Data Partitioning creates a root partition and two data partitions on each SSD drive in an AFF configuration. Disk auto assign should have assigned one data partition to each node in an HA pair. If a different disk assignment is re-quired, disk auto assignment must be disabled on both nodes in the HA pair by running the disk option modify command. Spare partitions can then be moved from one node to another by running the disk removeowner and disk assign commands.

Procedure 5.             Set Up Service Processor Network Interface

Step 1.                         Assign a static IPv4 address to the Service Processor on each node by running the following commands:

Note:      The Service Processor IP addresses should be in the same subnet as the node management IP addresses.

Procedure 6.             Create Manual Provisioned Aggregates 

An aggregate containing the root volume is created during the ONTAP setup process. To manually create additional aggregates, determine the aggregate name, the node on which to create it, and the number of disks it should contain.

Step 1.                         To create new aggregates, run the following commands:

Note:      You should have the minimum number of hot spare disks for the recommended hot spare disk partitions for their aggregate.

Note:      For all-flash aggregates, you should have a minimum of one hot spare disk or disk partition. For non-flash homogenous aggregates, you should have a minimum of two hot spare disks or disk partitions. For Flash Pool aggregates, you should have a minimum of two hot spare disks or disk partitions for each disk type.

Note:      In an AFF configuration with a small number of SSDs, you might want to create an aggregate with all, but one remaining disk (spare) assigned to the controller.

Note:      The aggregate cannot be created until disk zeroing completes. Run the storage aggregate show command to display the aggregate creation status. Do not proceed until both aggr1_node01 and aggr1_node02 are online.

Procedure 7.             Remove Default Broadcast Domains

By default, all network ports are included in separate default broadcast domain. Network ports used for data services (for example e0e, e0f, and so on) should be removed from their default broadcast domain and that broadcast domain should be deleted.

Step 1.                         To perform this task, run the following commands:

Note:      Delete the Default broadcast domains with Network ports.

Procedure 8.             Disable Flow Control on 25/100GbE Data Ports

To disable flow control on 25 and 100GbE data ports, follow these steps:

Step 1.                         Run the following command to configure the ports on node 01:

Step 2.                         Run the following command to configure the ports on node 02:

Step 3.                         Run the following command to verify:

Procedure 9.             Enable Cisco Discovery Protocol

Step 1.                         To enable the Cisco Discovery Protocol (CDP) on the NetApp storage controllers, run the following command to enable CDP in ONTAP:

Procedure 10.         Enable Link-layer Discovery Protocol on all Ethernet Ports

Step 1.                         Enable Link-layer Discovery Protocol (LLDP) on all ports of all nodes in the cluster:

Procedure 11.         Enable FIPS Mode on the NetApp ONTAP Cluster (Optional)

NetApp ONTAP is compliant in the Federal Information Processing Standards (FIPS) 140-2 for all SSL connections. When SSL FIPS mode is enabled, SSL communication from NetApp ONTAP to external client or server components outside of NetApp ONTAP will use FIPS compliant crypto for SSL.

Step 1.                         To enable FIPS on the NetApp ONTAP cluster, run the following commands:

Note:      If you are running NetApp ONTAP 9.8 or earlier manually reboot each node in the cluster one by one. Beginning in NetApp ONTAP 9.9.1, rebooting is not required.

Note:      If FIPS is not enabled on the NetApp ONTAP cluster, you will see a warning in AIQUM stating “FIPS Mode Disabled.”

Procedure 12.         Configure Timezone

To configure time synchronization on the cluster, follow this step:

Step 1.                         Set the time zone for the cluster.

Note:      For example, in the eastern United States, the time zone is America/New_York

Procedure 13.         Configure Simple Network Management Protocol

Note:      If you have enabled FIPS then please look at the following points while configuring SNMP.

·     The SNMP users or SNMP traphosts that are non-compliant with FIPS will be deleted automatically. “Configure SNMP traphosts” configuration will be non-compliant with FIPS.

·     The SNMPv1 user, SNMPv2c user (After configuring SNMP community) or SNMPv3 user (with none or MD5 as authentication protocol or none or DES as encryption protocol or both) is non-compliant with FIPS.

Step 1.                         Configure basic SNMP information, such as the location and contact. When polled, this information is visible as the sysLocation and sysContact variables in SNMP.

Step 2.                         Configure SNMP traps to send to remote hosts, such as an Active IQ Unified Manager server or another fault management system.

Note:      This step works when FIPS is disabled.

Note:      An SNMPv1 traphost or SNMPv3 traphost (configured with an SNMPv3 user non-compliant to FIPS) is non-compliant to FIPS.

Step 3.                         Configure SNMP community.

Note:      This step works when FIPS is disabled.

Note:      SNMPv1 and SNMPv2c are not supported when cluster FIPS mode is enabled.

Procedure 14.         Configure SNMPv3 Access

SNMPv3 offers advanced security by using encryption and passphrases. The SNMPv3 user can run SNMP utilities from the traphost using the authentication and privacy settings that you specify.

Note:      When FIPS is enabled, the following are the supported/compliant options for authentication and privacy protocol:

·     Authentication Protocol: sha, sha2-256

·     Privacy protocol: aes128

Step 1.                         To configure SNMPv3 access, run the following commands:

Note:      Refer to the SNMP Configuration Express Guide for additional information when configuring SNMPv3 security users.

Procedure 15.         Remove insecure ciphers from the NetApp ONTAP Cluster

Note:      If you do not perform this procedure, you will see the warning in AIQUM “SSH is using insecure ciphers.”

Step 1.                         Ciphers with the suffix CBC are considered insecure. To remove the CBC ciphers, run the following NetApp ONTAP command:

Procedure 16.         Create Management Broadcast Domain

Step 1.                         If the management interfaces are required to be on a separate VLAN, create a new broadcast domain for those interfaces by running the following command:

Procedure 17.         Create NFS Broadcast Domain

Step 1.                         To create an NFS data broadcast domain with a maximum transmission unit (MTU) of 9000, run the following commands to create a broadcast domain for NFS in ONTAP:

Procedure 18.         Create 2 Default Broadcast Domains and Add ifgroups a0a interface on each node to Default Broadcast Domain (applicable for 2-node cluster only)

Step 1.                         Create default broadcast domain.

Step 2.                         To add ports to default broadcast domains, run the following commands:

Procedure 19.         Create Interface Groups

Step 1.                         To create the LACP interface groups for the 25GbE data interfaces, run the following commands:

Procedure 20.         Create VLANs

Step 1.                         Create the management VLAN ports and add them to the management broadcast domain:

Step 2.                         To verify, issue the following command:

Step 3.                         Create the NFS VLAN ports and add them to the Infra-NFS broadcast domain:

Procedure 21.         Create SVM (Storage Virtual Machine)

The SVM (AA04-RKE2-BM-SVM) is used to configure NFS services for storage access by NetApp Astra Trident.

Step 1.                         Run the vserver create command:

Step 2.                         Add the required data protocols to the SVM:

Step 3.                         Remove the unused data protocols from the SVM:

Note:      It is recommended to remove fcp and iscsi protocols if they are not used:

Step 4.                         Add the two data aggregates to the AA04-RKE2-BM-SVM aggregate list for the NetApp ONTAP Tools:

Step 5.                         Enable and run the NFS protocol in the AA04-RKE2-BM-SVM:

Note:      If the NFS license was not installed during the cluster configuration, make sure to install the license before starting the NFS service.

Procedure 22.         Vserver Protocol Verification

Step 1.                         Verify the required protocols are added to the AA04-RKE2-BM-SVM vserver:

Procedure 23.         Create Load-Sharing Mirrors of SVM Root Volume

Step 1.                         Create a load-sharing mirror volume of the “AA04-RKE2-BM-SVM” SVM root volume on the node that does not have the Root Volume:

Step 2.                         Create a job schedule to update the root volume mirror relationships every 15 minutes:

Step 3.                         Create mirroring relationships:

Step 4.                         Initialize the mirroring relationship:

Procedure 24.         Configure HTTPS Access

To configure secure access to the storage controller, follow these steps:

Step 1.                         Increase the privilege level to access the certificate commands:

Step 2.                         A self-signed certificate is already in place. Verify the certificate and obtain parameters (for example, the <serial-number>) by running the following command:

Step 3.                         For each SVM shown, the certificate common name should match the DNS fully qualified domain name (FQDN) of the SVM. Delete the two default certificates and replace them with either self-signed certificates or certificates from a certificate authority (CA). To delete the default certificates, run the following commands:

Note:      Deleting expired certificates before creating new certificates is best practice. Run the security certificate delete command to delete the expired certificates. In the following command, use TAB completion to select and delete each default certificate.

Step 4.                         To generate and install self-signed certificates, run the following commands as one-time commands. Generate a server certificate for the AA04-RKE2-BM-SVM and the cluster SVM. Use TAB completion to aid in the completion of these commands.

Step 5.                         To obtain the values for the parameters required in step 5 (<cert-ca> and <cert-serial>), run the security certificate show command.

Step 6.                         Enable each certificate that was just created by using the –server-enabled true and –client-enabled false parameters. Use TAB completion to aid in the completion of these commands.

Step 7.                         Disable HTTP cluster management access.

Note:      It is normal for some of these commands to return an error message stating that the entry does not exist.

Step 8.                         Change back to the normal admin privilege level and verify that the system logs are available in a web browser.

Procedure 25.         Set password for SVM vsadmin user and unlock the user

Step 1.                         Set a password for the SVM vsadmin user and unlock the user using the following commands:

Procedure 26.         Configure login banner for the SVM

Step 1.                         To create login banner for the SVM, run the following command:

Note:      If the login banner for the SVM is not configured, users will observe a warning in AIQUM stating “Login Banner Disabled.”

Procedure 27.         Remove insecure ciphers from the SVM

Ciphers with the suffix CBC are considered insecure.

Note:      If you do not perform the procedure, you will see a warning in AIQUM saying “SSH is using insecure ciphers.”

Step 1.                         To remove the CBC ciphers from the SVM, run the following NetApp ONTAP command:

Procedure 28.         Configure Export Policy Rule

Note:      This step is crucial when using ontap-nas (NFS) storage driver in Astra Trident since this SVM is added as a Trident Backend.

To configure NFS on the SVM, follow these steps:

Step 1.                         Create a new rule for the infrastructure NFS subnet in the default export policy:

Note:      For more information on configuring NFS Export Policy for Trident, go to: https://docs.netapp.com/us-en/trident/trident-use/ontap-nas-prep.html#requirements.

Step 2.                         Assign the FlexPod export policy to the infrastructure SVM (AA04-RKE2-BM-SVM) root volume:

Procedure 29.         Create FlexVol Volumes

The following information is required to create a NetApp FlexVol® volume:

·     The volume name

·     The volume size

·     The aggregate on which the volume exists

Step 1.                         Run the following command to create a volume for storing SVM audit log configuration:

Step 2.                         Update set of load-sharing mirrors using the command below:

Procedure 30.         Create NFS LIFs

Step 1.                         To create NFS LIFs, run the following commands:

Step 2.                         To verify, run the following commands:

Procedure 31.         Create SVM Management LIF (Add Infrastructure SVM Administrator)

To add the SVM administrator and SVM administration LIF in the in-band management network, follow these steps:

Step 1.                         Run the following commands:

Step 2.                         Create a default route that enables the SVM management interface to reach the outside world.

Step 3.                         To verify, run the following commands:

Step 4.                         Add route to reach the corresponding RKE2 master/worker network:

Note:      A cluster serves data through at least one and possibly several SVMs. With these steps, you’ve created a single data SVM. You can create additional SVMs depending on your requirement.

Procedure 32.         Configure Auto Support

NetApp AutoSupport® sends support summary information to NetApp through HTTPS.

Step 1.                         To configure AutoSupport, run the following command:

Procedure 33.         Configure DNS for Infrastructure SVM (AA04-RKE2-BM-SVM)

Step 1.                         To configure DNS for the Infra-SVM, run the following command:

Procedure 34.         Create and enable auditing configuration for the SVM

Note:      If you do not perform this procedure for the SVM, you will see a warning in AIQUM stating “Audit Log is disabled.”

Step 1.                         To create auditing configuration for the SVM, run the following command:

Step 2.                         Run the following command to enable audit logging for the SVM:

Note:      It is recommended that you enable audit logging so you can capture and manage important support and availability information. Before you can enable auditing on the SVM, the SVM’s auditing configuration must already exist.

Procedure 35.         Test Auto Support

Step 1.                         To test the Auto Support configuration by sending a message from all nodes of the cluster, run the following command:

Now the setup of the NetApp storage is completed. Since SLES servers are booted with local disks, storage configuration for boot is not required. The configuration required for NetApp Astra Trident will be done later.

Cisco Intersight Managed Mode – Initial Setup

This chapter contains the following:

·     Set up Cisco Intersight Managed Mode on Cisco UCS Fabric Interconnects

·     Set up Cisco Intersight Account

·     Initial Cisco Intersight Configuration

The Cisco Intersight Managed Mode (also referred to as Cisco IMM or Intersight Managed Mode) is a new architecture that manages Cisco Unified Computing System™ (Cisco UCS®) fabric interconnect–attached systems. Cisco Intersight managed mode standardizes both policy and operation management for Cisco UCS B-series M5 and M6 servers, Cisco UCS X210c M6 and M7 compute nodes used in this deployment guide. For a complete list of supported platforms, go to: https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Intersight_Managed_Mode_Configuration_Guide/b_intersight_managed_mode_guide_chapter_01010.html

During the initial setup, Cisco UCS FIs are configured in Intersight Managed Mode and added to a newly created Intersight account. Intersight organization creation, resource group definition and license setup are also part of the initial setup. At the end of this section, you can start creating various chassis and server level policies and profiles to deploy Cisco UCS compute nodes.

Set up Cisco Intersight Managed Mode on Cisco UCS Fabric Interconnects

To set up Cisco UCS 6454 Fabric Interconnects in Intersight Managed Mode, complete the steps explained here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#CiscoIntersightManagedModeSetUp  

Note:      If a software version that supports Intersight Managed Mode (4.1(3) or later) is already installed on Cisco UCS Fabric Interconnects, do not upgrade the software to a recommended recent release using Cisco UCS Manager. The software upgrade will be performed using Cisco Intersight to make sure Cisco UCS X-series firmware is part of the software upgrade.

Set up Cisco Intersight Account

To set up a new Cisco Intersight Account, complete the steps explained here: https://www.intersight.com/help/saas/getting_started/create_cisco_intersight_account

Note:      Setting up a new Intersight account is not necessary if you plan to add the Cisco UCS FIs to an existing account.

Initial Cisco Intersight Configuration

When setting up a new Cisco Intersight account (as described in this document), the account needs to be enabled for Cisco Smart Software Licensing. It is also required to configure Organizations, Resource Groups, and to claim the Cisco UCS hardware.

Cisco Intersight Managed Mode – Domain Profile Setup

This chapter contains the following:

·     General Configuration

·     VLAN Configuration

·     Review and Deploy the Domain Profile

·     Configure Cisco UCS Chassis Profile (optional)

A Cisco UCS domain profile configures a fabric interconnect pair through reusable policies, allows configuration of the ports and port channels, and configures the VLANs and VSANs in the network. It defines the characteristics of and configured ports on fabric interconnects. The domain-related policies can be attached to the profile either at the time of creation or later. One Cisco UCS domain profile can be assigned to one fabric interconnect domain.

Domain profile setup has the following steps:

·     General configuration – name and organization assignment

·     UCS Domain Assignment – assign previously claimed UCS Fabric Interconnects to the domain profile

·     VLAN configuration – define required VLANs

·     Port configuration – configure server and uplink ports and port-channels for Ethernet traffic

·     UCS domain configuration – policies such as NTP, DNS and QoS

·     Review and deploy – review the configuration and deploy the UCS domain profile

General Configuration

To configure the name, description, and organization for the UCS domain profile, complete the steps from Procedure 9 and following, explained here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#CiscoIntersightManagedModeSetUp

VLAN Configuration

To define the VLANs, complete the steps explained here:
https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#VLANandVSANConfiguration before continuing with the RKE2 specific configurations.

Figure 2.       UCS Domain Profile VLAN Policy Mapping

Procedure 1.             Add SUSE Rancher/RKE2 VLANs to the Domain Profile

To add the RKE2 VLANs to the Domain Profile, follow the steps shown in the screenshot below:

Review and Deploy the Domain Profile

With a successful deployment of the UCS domain profile, the ethernet port channels should be enabled and the Cisco UCS rack servers and compute nodes should be successfully discovered.

Figure 3.       Example of discovered compute nodes and rack server

Configure Cisco UCS Chassis Profile (optional)

Cisco UCS Chassis profile in Cisco Intersight allows you to configure various parameters for the chassis, including:

·     IMC Access Policy: IP configuration for the in-band chassis connectivity. This setting is independent of Server IP connectivity and only applies to communication to and from chassis.

·     SNMP Policy, and SNMP trap settings.

·     Power Policy to enable power management and power supply redundancy mode.

·     Thermal Policy to control the speed of FANs.

A chassis policy can be assigned to any number of chassis profiles to provide a configuration baseline for a chassis. In this deployment, no chassis profile was created or attached but you can configure policies to configure SNMP or Power parameters and attach them to the chassis as needed. For more information about configuring UCS chassis policies, go to: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#CiscoIntersightManagedModeSetUp

Cisco Intersight Managed Mode – Server Profile Template

This chapter contains the following:

·     vNIC Placement for Server Profile Templates

·     Server Profile Template Creation

In Cisco Intersight Managed Mode, a server profile enables resource management by simplifying policy alignment and server configuration. The server profiles are derived from a server profile template. Server profile template and its associated policies can be created using the server profile template wizard.

If VMware ESXi hosts will be deployed on the same UCS Domain, complete the steps explained here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_ucs_xseries_e2e_ontap_manual_deploy.html#CiscoUCSIMMManualConfiguration

In this document, one server profile template is created for SUSE Rancher and RKE2 hosts deployed bare metal.

vNIC Placement for Server Profile Templates

This section explains the vNIC definitions and placement for server profile templates.

SUSE RKE2 Host vNIC Placement

Two vNICs are configured and manually placed as listed in Table 5.

Table 5.          vNIC placement for Management Domain hosts

Server Profile Template Creation

Procedure 1.             Configure a Server Profile Template

Step 1.                         Log in to Cisco Intersight.

Step 2.                         Go to Infrastructure Service > Configure > Templates and in the main window click Create UCS Server Profile Template.

Procedure 2.             General Configuration

Step 1.                         Select the organization from the drop-down list (for example RTP4-AA04).

Step 2.                         Provide a name for the server profile template. The name used in this deployment is AA04-RKE-Local

Step 3.                         Select UCS Server (FI-Attached).

Step 4.                         Provide an optional description.

Step 5.                         Click Next.

Procedure 3.             Compute Configuration – UUID Pool

Follow these steps to configure UUID pool under the Compute Configuration.

Note:      For additional granularity, you can choose to create different UUID pools for different workloads or departments.

Step 1.                         Click Select Pool under UUID Pool and then in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example RTP4-AA04) and provide a name for the UUID Pool (for example, AA04-UUID-Pool).

Step 3.                         Provide an optional Description and click Next.

Step 4.                         Provide a UUID Prefix (for example, a random prefix of AA040000-0000-0001 was used).

Step 5.                         Add a UUID block.

Step 6.                         Click Create.

Procedure 4.             Compute Configuration – BIOS policy

Follow these steps to configure BIOS policies under the Compute Configuration.

Step 1.                         Click Select Policy next to BIOS and in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-RKE-BIOS or AA04-RKE-BIOS-M6).

Step 3.                         Click Next.

Step 4.                         On the Policy Details screen, select appropriate values for the BIOS settings. In this deployment, the default BIOS values were selected.

Step 5.                         Click Create.

Procedure 5.             Compute Configuration - Boot Order policy for Management Domain hosts

Follow these steps to configure Boot Order policy for RKE2 hosts booting from local disk.

Step 1.                         Click Select Policy next to Boot Order and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AAO4-RKE-Local-Boot).

Step 3.                         Click Next.

Step 4.                         For Configured Boot Mode option, select Unified Extensible Firmware Interface (UEFI).

Step 5.                         Turn on Enable Secure Boot.

Step 6.                         Click Add Boot Device drop-down list and select Virtual Media.

Step 7.                         Provide a device name (for example, KVM-Mapped-ISO) and then, for the subtype, select KVM Mapped DVD.

Step 8.                         From the Add Boot Device drop-down list, select Local Disk.

Step 9.                         Provide the Device Name: M2-RAID1 and the Slot: MSTOR-RAID.

Step 10.                     Verify the order of the boot policies and adjust the boot order as necessary using arrows next to the delete button.

Step 11.                     Click Create.

Procedure 6.             Compute Configuration - Virtual Media Policy

Follow these steps to configure Virtual Media Policy to allow mapping an ISO files as installation source for operating system.

Step 1.                         Click Select Policy next to Virtual Media and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-SLE-15).

Step 3.                         Turn on Enable Virtual Media, Enable Virtual Media Encryption, and Enable Low Power USB.

Step 4.                         Do not Add Virtual Media at this time.

Step 5.                         Click Create.

Step 6.                         Click Next to move to Management Configuration.

Management Configuration

These policies will be added to the management configuration:

·     IMC Access to define the pool of IP addresses for compute node KVM access.

·     IPMI Over LAN to allow Intersight to manage IPMI messages.

·     Local User to provide local administrator to access KVM.

·     Virtual KVM to allow the Tunneled KVM.

Procedure 1.             Management Configuration - Cisco IMC access policy

Follow these steps to configure Cisco IMC access policy:

Step 1.                         Click Select Policy next to IMC Access and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-IMC-Access).

Step 3.                         Click Next.

Note:      You can select in-band management access to the compute node using an in-band management VLAN (for example, VLAN 17) or out-of-band management access via the Mgmt0 interfaces of the FIs. Policies like SNMP, vMedia and Syslog are currently not supported via Out-Of-Band and will require an In-Band IP to be configured. In-band management access was configured in this deployment guide.

Step 4.                         Enable In-Band Configuration and provide the in-band management VLAN (for example, 17).

Step 5.                         Make sure IPv4 address configuration is selected.

Step 6.                         Under IP Pool, click Select IP Pool and then, in the pane on the right, click Create New.

Step 7.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the pool (for example, AA04-Mgmt-IP-Pool).

Step 8.                         Select Configure IPv4 Pool and provide the information to define a pool for KVM IP address assignment including an IP Block.

Note:      The management IP pool subnet should be accessible from the host that is trying to access the KVM session. In the example shown here, the hosts trying to establish a KVM connection would need to be able to route to 10.101.1.0/24 subnet.

Step 9.                         Click Next.

Step 10.                     Unselect Configure IPv6 Pool.

Step 11.                     Click Create to finish configuring the IP address pool.

Step 12.                     Click Create to finish configuring the IMC access policy.

Procedure 2.             Management Configuration - IPMI Over LAN policy

Follow these steps to configure IPMI Over LAN policy.

Step 1.                         Click Select Policy next to IPMI Over LAN and then, in the pane on the right, click Create New.

Step 2.                         Verify the correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-IPMIoverLAN-Policy).

Step 3.                         Turn on Enable IPMI Over LAN.

Step 4.                         Click Create.

Procedure 3.             Management Configuration - Local User policy

Follow these steps to configure local user policy:

Step 1.                         Click Select Policy next to Local User and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-VM-IPMI-Pol).

Step 3.                         Verify that UCS Server (FI-Attached) is selected.

Step 4.                         Verify that Enforce Strong Password is selected.

Step 5.                         Click Add New User and then click + next to the New User.

Step 6.                         Provide the username (for example, fpadmin), choose a role (for example, admin), and provide a password.

Note:      The username and password combination defined here can be used to log into KVMs as well as for IPMI access.

Step 7.                         Click Create to finish configuring the user.

Step 8.                         Click Create to finish configuring the local user policy.

Step 9.                         Click Next to move to Storage Configuration.

Procedure 4.             Management Configuration - Virtual KVM Policy

Follow these steps to configure the KVM policy:

Step 1.                         Click Select Policy next to Virtual KVM and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04 and provide a name for the policy (for example, AA04-VM-KVM).

Step 3.                         Verify that UCS Server (FI-Attached) is selected.

Step 4.                         Turn on Allow Tunneled vKVM and leave the other two options on as well.

Step 5.                         Click Create.

Note:      To fully enable Tunneled KVM, make sure under System > Settings > Security and Privacy>Configure, “Allow Tunneled vKVM Launch” and “Allow Tunneled vKVM Configuration” is turned on.

Step 6.                         Click Next to move to Storage Configuration.

Storage Configuration

Storage configuration covers the local disk configuration options. In this configuration two M.2 SSD devices connected to a M.2 RAID controller were used.

Procedure 1.             Storage Configuration – Storage

Follow these steps to configure Storage Policy to configure a RAID1 on the two M.2 SSD devices for the operating system.

Step 1.                         Click Select Policy next to Storage and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, MStor-Boot), click Next.

Step 3.                         Enable Use JBOD drives for virtual drive creation, enable M.2 RAID Configuration, and select MSTOR-RAID1, MSTOR-RAID2 from the drop-down list, click Create.

Step 4.                         Back in the storage configuration window, click Next.

Network Configuration

Network configuration encompasses both LAN and SAN connectivity policies.

Network Configuration - LAN Connectivity

LAN connectivity policy defines the connections and network communication resources between the server and the LAN. This policy uses pools to assign MAC addresses to servers and to identify the vNICs that the servers use to communicate with the network. For consistent vNIC and vHBA placement, manual vHBA/vNIC placement is utilized.

The RKE2 hosts use two vNICs configured as shown in Table 6.

Table 6.          vNICs for setting up LAN Connectivity Policy

Procedure 1.             Create LAN Connectivity Policy

Step 1.                         Click Select Policy next to LAN Connectivity and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-RKE-LanCon-Pol). Click Next.

Step 3.                         Under vNIC Configuration, select Manual vNICs Placement.

Step 4.                         Click Add vNIC.

Network Configuration - LAN Connectivity - MAC Pool

If the MAC address pool has not been defined yet, you will need to create a new MAC address pool when creating the first vNIC.

Note:      For better segmentation two separate MAC address pools can be used, like AA04-Mac-Pool-A for all Fabric-A vNICs and AA04-Mac-Pool-B for all Fabric-B vNICs.

Table 7.           MAC Address Pools

Note:      Each server requires 2 MAC addresses from the pool. Adjust the size of the pool according to your requirements. “A4” in the MAC address pool above is a unique identifier representing the rack ID. Adding a unique identifier helps with troubleshooting of switching issues.

Procedure 1.             Define the MAC Pool

Step 1.                         Click Select Pool under MAC Address Pool and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the pool from Table 7.

Step 3.                         Click Next.

Step 4.                         Provide the starting MAC address from Table 7 (for example, 00:25:B5:A4:0B:00)

Step 5.                         Provide the size of the MAC address pool from Table 7 (for example, 255).

Step 6.                         Click Create to finish creating the MAC address pool.

Step 7.                         From the Add vNIC window, provide a vNIC Name, Slot ID, Switch ID, and PCI Order information from Table 6.

Step 8.                         For Consistent Device Naming (CDN), from the drop-down list, select vNIC Name.

Step 9.                         Verify that Failover is enabled.

Network Configuration - LAN Connectivity– Ethernet Network Group Policy

Ethernet Network Group policies will be created and reused on applicable vNICs as explained below. Ethernet network group policy defines the VLANs allowed for a particular vNIC therefore multiple network group policies will be defined as listed in Table 8.

Table 8.           Ethernet Group Policy Values

Note:      *Adding an Out-of-Band Management VLAN is optional and depends on your requirements.

Procedure 1.             Define Ethernet Group Policy for a vNIC

Step 1.                         Click Select Policy under Ethernet Network Group Policy and then, in the pane on the right, click Create New.

Step 2.                         Verify the correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy from the Table 8 (for example, AA04-RKE-Access).

Step 3.                         Click Next.

Step 4.                         Enter the Allowed VLANs and Native VLAN from the Table 8.

Step 5.                         Click Create to finish configuring the Ethernet network group policy.

Note:      When ethernet group policies are shared between two vNICs, the ethernet group policy only needs to be defined for the first vNIC. For subsequent vNIC policy mapping, click Select Policy and pick the previously defined ethernet group policy from the list.

Network Configuration - LAN Connectivity– Ethernet Network Control Policy

Ethernet Network Control Policy is used to enable Cisco Discovery Protocol (CDP) and Link Layer Discovery Protocol (LLDP) for the vNICs. A single policy will be created here and reused for all the vNICs.

Procedure 1.             Create the Ethernet Network Control Policy

Step 1.                         Click Select Policy under Ethernet Network Control Policy and in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-VM-NetControl-Policy).

Step 3.                         Click Next.

Step 4.                         Enable Cisco Discovery Protocol and both Enable Transmit and Enable Receive under LLDP.

Step 5.                         Click Create to finish creating Ethernet network control policy.

Network Configuration - LAN Connectivity - Ethernet QoS

Ethernet QoS policy is used to enable jumbo maximum transmission units (MTUs) for the vNICs. A single policy will be created and reused for all the vNICs.

Procedure 1.             Create the Ethernet QoS

Step 1.                         Click Select Policy under Ethernet QoS and in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-QoS-9000).

Step 3.                         Click Next.

Step 4.                         Change the MTU, Bytes value to 9000.

Step 5.                         Click Create to finish setting up the Ethernet QoS policy.

Network Configuration - LAN Connectivity - Ethernet Adapter

The Ethernet adapter policy is used to set the interrupts and the send and receive queues. The values are set according to the best-practices guidance for the operating system in use. Cisco Intersight provides default Ethernet Adapter policy for typical Linux deployments.

You can also configure a tweaked ethernet adapter policy for additional hardware receive queues handled by multiple CPUs in scenarios where there is a lot of storage traffic and multiple flows. In this deployment, an unmodified ethernet adapter policy is created and attached to the interfaces.

Procedure 1.             Create the Ethernet Adapter

Step 1.                         Click Select Policy under Ethernet Adapter and then, in the pane on the right, click Create New.

Step 2.                         Verify correct organization is selected from the drop-down list (for example, RTP4-AA04) and provide a name for the policy (for example, AA04-Linux-Adapter-Policy).

Step 3.                         Click Select Default Configuration under Ethernet Adapter Default Configuration.

 

Step 4.                         From the list, select Linux.

Step 5.                         Click Next.

Step 6.                         For the AA04-Linux-Adapter Policy, click Create and skip the rest of the steps in this “Create Ethernet Adapter Policy” section.

Step 7.                         Click Add to add the vNIC to the LAN connectivity policy.

Step 8.                         Go to Step 7 Add vNIC and repeat vNIC creation for all vNICs.

Step 9.                         Verify all four vNICs were successfully created for appropriate LAN connectivity Policy.

Step 10.                     Click Create to finish creating the LAN Connectivity policy.

Step 11.                     Back in the network configuration window, click Next and click Close.

Derive Server Profiles

Procedure 1.             Derive the number of server profiles for the planned deployment from the configured template

Step 1.                         From the Infrastructure Services > Configure > Templates, click “…” next to the host template name and select Derive Profiles.

Step 2.                         Under the Server Assignment, select Assign Now and pick the number of servers required to create the K3s or RKE2 cluster. You can adjust the number of servers depending on the number of profiles to be deployed.

Note:      For Non-Production systems a single node K3s cluster is okay to use. For production systems it is recommended to use at the minimum a three node RKE2 cluster for high availability.

Figure 4.       Derive Server Profiles for three RKE2 nodes

Step 3.                         Click Next.

Step 4.                         Intersight will fill in “default” information for the selected servers (only two out of four servers shown below):

Step 5.                         Adjust the Prefix name and number (if needed).

Step 6.                         Click Next.

Step 7.                         Verify the information and click Derive to create the Server Profiles.

Step 8.                         Intersight will start configuring the server profiles and will take some time to apply all the policies. Use the Requests tab to see the progress.

Step 9.                         When the Server Profiles are deployed successfully, they will appear under the Server Profiles with the status of OK:

Software Deployment

This chapter contains the following:

·     Prerequisites

·     Deploy and Configure Network Services

·     SUSE Linux Enterprise Installation on all Nodes

·     Install RKE2 Cluster

·     Install Rancher

·     Deployment Option with SUSE SLE Micro and K3s

·     Deploy and Configure NetApp Astra Trident

·     NetApp Astra Trident Deployment

To deploy the required services, like DNS, we used a management node to deploy and configure them. This is to highlight the required or recommended configuration on one example. Please use your existing systems to configure the requirements accordantly.

Note:      Before continuing the SUSE Rancher deployment, review the prerequisites: https://ranchermanager.docs.rancher.com/pages-for-subheaders/installation-requirements

Table 9.          Software versions

Prerequisites

The following are the prerequisites for SUSE Rancher Enterprise Container Manager:

·     Configure DHCP or set static IP addresses on each node.

·     DNS is an absolute MUST. Without proper DNS configuration, the installation will not continue.

·     Ensure network connectivity among nodes, DNS, DHCP (if used), HAProxy, Installer or Management node.

Note:      Using the DHCP server to manage the machines for the cluster long-term is recommended. Ensure that the DHCP server is configured to provide persistent IP addresses and host names to the cluster machines.

Note:      In the deployment, DHCP server is setup for installing Rancher and RKE2 master and worker nodes.

Minimum Required Machines

The smallest RKE2 cluster require the following hosts:

·     3 x hosts to run control plane, etcd, and worker function.

Deploy and Configure Network Services

For detailed information about the installation options, please refer to SUSE Rancher documentation.

A SUSE Rancher on RKE2 installation involves the following high-level steps:

1.    Prepare SUSE Linux and RKE2 installation.

2.    Install and configure SUSE operating system.

3.    Install and configure RKE2 cluster.

4.    Install and configure SUSE Rancher.

5.    Validate the installation.

1.    To provide the deployment of the required services – DNS, Load balancer, DHCP, and https – a single SUSE Linux system was used in this document. Please use your existing services and add the required information or use the documentation from SUSE to make the services high available for production landscapes.

Procedure 1.             Install the Required Software Components

Follow these steps to install the required software components on the system hosting the services – we named it sle-svc.

Step 1.                         Take care that the system is registered with SUSE to get the latest versions, and add the required package modules:

Step 2.                         Install the DNS and DHCP server components:

Step 3.                         Install the http server:

Step 4.                         Install the haproxy software:

Set up DNS

DNS is used for name resolution and reverse name resolution. DNS A/AAAA or CNAME records are used for name resolution and PTR records are used for reverse name resolution. The reverse records are important because SUSE Rancher and RKE2 uses FQDN to create self signed certificates. Additionally, the reverse records are used to generate the certificate signing requests (CSR) for official certificates.

Note:      In this design, we setup the DNS server (named) on the management node by executing zypper in -t pattern dhcp_dns_server to install the required software components

The following DNS records are required for an RKE2 cluster that hosts Rancher ECM. A complete DNS record takes the form: <component>.<cluster_name>.<base_domain>.

Table 10.       Required DNS records

Step 5.                         For high availability, it is required to configure the host entry for the Rancher deployment to the ip address of the load balancer. In this installation the load balancer is running on the management node:

Procedure 2.             Set up Webserver

A webserver is used in the setup to make the agent ISO-Image available via the vMedia policy to the UCS servers. The webserver must be reached by UCS over the OOB-Management and IB-Management network during the installation.

Note:      In this design, we set up the Apache web server (httpd) on the management node.

Note:      If you are setting up webserver and HAProxy in management node for serving installation files, and iso, make sure it is not using a port which conflicts with the HAproxy configuration.

Step 1.                         If the webserver is not already installed. Run the following command in installer server:

Step 2.                         Edit ‘httpd.conf’ file with port number accessible.

Note:      We configured port 8080 as shown below. Add port 8080 to the list of allowed ports in the firewall if firewall service is running.

Step 3.                         Start httpd service:

Procedure 3.             Set up Load Balancer

A Linux based load balancer is used to balance the network traffic to all the RKE2 nodes. The load balancer must be reachable for the access network only. In this deployment we have used HAProxy on the management node. It is recommended to use external load balancer in production environment or implement HA for HAproxy load balancers with keepalived VIP.

Load Balancer must allow non-terminating HTTPS and in one case "websockets" via port 80 (required for CML). HA proxy must have good network connectivity. It load balances all master (control traffic) and application traffic. Port 80 and 443 are required for traffic handled by HAProxy and must not be occupied by any other service running on this node.

Step 1.                         If the HAProxy is not already installed. Run the following command in installer server:

Step 2.                         Edit ‘haproxy.cfg’ file with port number accessible:

Step 3.                         Start haproxy service:

Procedure 4.             Set Up and Configure DHCP

Note:      DHCP is recommended for large scale production deployment to provide persistent IP addresses and host names to all the cluster nodes. Use IP reservation, so that IP should not change during node reboots.

Note:      In this deployment, PXE server is setup on the management node. In this configuration, we specified IP reservation of RKE2 nodes with MAC address of the interfaces. MAC address of the interface can be obtained from Cisco Intersight.

Note:      In this reference design, the DHCP setup is for reference purposes only and is deployed on the management node. It is not recommended for production grade setup to deploy a DHCP server on a standalone Linux system. In many cases, already existing enterprise grade DHCP setup can be utilized.

Note:      For configuring DHCP, specify the subnet and range used for offering the IP address via DHCP. You can also specify the lease time.

Step 1.                         Configure DHCP using the following conf file. This configuration file is for reference purposes only, change according to your environment. Adding more worker nodes for scaling requires an entry in DNS and IP reservation with MAC address in below dhcp.conf file:

Step 2.                         Add all network interfaces to answer dhcp requests into /etc/sysconfig/dhcpd:

Step 3.                         Each time the dhcpd.conf is modified, restart the dhcpd service:

SUSE Linux Enterprise Installation on all Nodes

For detailed information about the installation options, please refer to SUSE Rancher documentation.

A SUSE Rancher on RKE2 installation involves the following high-level steps:

1.    Prepare SUSE Linux and RKE2 installation.

2.    Install and configure SUSE operating system.

3.    Install and configure RKE2 cluster.

4.    Install and configure SUSE Rancher.

5.    Validate the installation.

6.    To show the deployment of the required services – DNS, Load balancer, DHCP, and https – in this document a single SUSE Linux system was used. Please use your existing services and add the required information or use the documentation from SUSE to make the services highly available for production landscapes.

Procedure 1.             Prepare SUSE Linux Enterprise Installation

Step 1.                         Download the SUSE Linux Enterprise 15 SP5 ISO image from the SUSE web site and store it on a place reachable from the OOB_Management network. We used the management node under /srv/www/htdocs:

Procedure 2.             Update the vMedia Policy in Intersight

Step 1.                         Open Intersight and browse to Configure – Policies and select the vMedia policy created for this installation, we used AA04-SLE-15:

Step 2.                         Click the ellipses and click Edit:

Step 3.                         Under Policy Details, click Add Virtual Media:

Step 4.                         Select the protocol and enter the required details to access the ISO image. We used HTTP on the management node. Click Add:

Step 5.                         Click Safe and Deploy:

The vMedia policy gets activated on all UCS Server Profiles created with the Template:

Note:      Wait until the deployment of the vMedia policy is finished on all nodes before continuing to the next step.

Procedure 3.             Install SUSE Linux Enterprise

Step 1.                         Under Profiles or Servers, click the ellipses and open a vKVM.

Step 2.                         Power on or reset the server to boot from the ISO image defined in the vMedia policy.

Step 3.                         Follow the SUSE Linux Enterprise installation.

Note:      This document does not provide a comprehensive installation. Please use the default settings or the settings that fit best to your local installation.

Step 4.                         Select SUSE Linux Enterprise Server 15 SP5 and click Next.

Step 5.                         Under Extension and Module Selection, select Basesystem Module and click Next. Selecting the Containers Module is optional.

Step 6.                         Under System Role, select Text Mode and click Next.

Note:      It is required to configure NTP on all servers used for RKE.

Step 7.                         Configure the filesystems using the guided setup.

Note:      Do not create a separate partition for /home since no user will work on this host and the storage is required for the containers.

Step 8.                         Configure the network adapters as required for the installation. We used DHCP to configure the network, so the configuration is left as it is.

After the OS installation is finished the server will reboot automatically into the installed SLE15 SP5 system.

The IP addresses should be configured correctly using DHCP.

Step 9.                         If not already done, register the systems with SUSE to get access to the online software repositories and updates.

Install RKE2 Cluster

There are a few tasks to complete the RKE2 cluster installation.

Note:      For more information, go to the official Rancher installation site: https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher

Procedure 1.             Create the Initial RKE2 Cluster Configuration

Step 1.                         Create the base directory for RKE2 configuration files on all nodes:

Step 2.                         Create the configuration file for the RKE2 cluster based on your requirements, as documented here: https://docs.rke2.io/install/configuration.

Procedure 2.             Install RKE2 on the first node

The installation process occurs in multiple steps. The first step is to install the RKE2 software on the first node of the cluster. The parameter INSTALL_RKE2_VERSION is optional. Without this option, the latest version will be installed.

Step 1.                         Run the following command:

Step 2.                         Enable and start RKE2 system service:

Step 3.                         Get the created node-token required for the remaining nodes to join the new cluster:

Procedure 3.             Check the RKE2 installation status on the first node

The installation process on the first node will up to 30 minutes.

Step 1.                         Check the list of RKE2 nodes in the cluster. Only the first node should be listed:

Step 2.                         Check the running pods in the cluster. All entries must be either in the Running or Completed state:

Procedure 4.             Install RKE2 on the remaining nodes

Step 1.                         Create the configuration file on every remaining node to join the cluster.

Note:      It is important to add the node-token and the server entry with the ip address from the first node.

Step 2.                         Install RKE2 software on remaining nodes:

Step 3.                         Create the configuration file on every remaining node to join the cluster one after the other:

Procedure 5.             Check the RKE2 cluster installation

Note:      Before you install SUSE Rancher RKE2, download the helm package client (helm) to the management node.

Step 1.                         Run the following command on one of the nodes to list all nodes in the cluster:

Step 2.                         Run the following command on one of the nodes to see the state of all pods:

Procedure 6.             Copy the kubeconfig file to the management node

Step 1.                         Run the following command on the management node:

Step 2.                         Change the server entry in the config file to match the load balancer IP for the installed cluster:

Install Rancher

This section contains the following:

·     Prepare the RKE2 luster for the Rancher installation

·     Install cert-manager

·     Install Rancher on Three Node RKE2 Cluster

Procedure 1.             Prepare the RKE2 Cluster for the Rancher installation

Step 1.                         Create the namespace for Rancher (default is cattle-system):

Step 2.                         Add the Rancher helm repository to the system:

Step 3.                         Check for the latest Rancher versions in the repository:

Procedure 2.             Install Cert-manager

Step 1.                         To use Rancher managed certificates it is required to install the cert-manager on the cluster. Run the following commands:

Step 2.                         Check that the cert-manager installation was successful:

Procedure 3.             Install Rancher on Three Node RKE2 Cluster

Step 1.                         Run the following command to initiate the Rancher installation:

Step 2.                         Monitor the installation progress by running the following commands:

Step 3.                         Open a web browser, like Google Chrome, and navigate to the hostname provided in the rancher installation procedure. We used rancher.rke1.aa04.cspgb4.local as the hostname.

Step 4.                         If no password for user admin was specified at installation time run the following command to get the generated password for the user admin:

Step 5.                         Use the password to log on and accept the EUL as shown below:

The Rancher home screen is shown below:

Deployment Option with SUSE SLE Micro and K3s

If a deployment based on SUSE Linux Enterprise 15 and RKE2 is too big or if a simplified and downsized container optimized stack is required, the deployment with SUSE SLE Micro and K3s is a viable option. This stack can be deployed as a single node stack – i.e. for non-production systems, or as multi-node stack with full high availability.

SUSE Linux Enterprise Micro 5 Installation

Procedure 1.             Prepare SUSE Linux Enterprise Micro installation

Step 1.                         Download the SUSE Linux Enterprise Micro 5.x ISO image from the SUSE web site and store it on a place reachable from the OOB_Management network. We used the management node under /srv/www/htdocs.

Procedure 2.             Update the vMedia Policy in Intersight

Step 1.                         Open Intersight and browse to Configure – Policies and select the vMedia policy created for this installation, we used AA05-SLE-15.

Step 2.                         Click the ellipses and select Edit.

Step 3.                         Under Policy Details, click Add Virtual Media.

Step 4.                         Select the protocol and enter the required details to access the ISO image. We have used HTTP on the management node. Click Add.

Step 5.                         Click Safe and Deploy.

The vMedia policy gets activated on all Server Profiles created with the template:

Note:      Wait until the deployment of the vMedia policy is finished on all nodes before continuing with the next step.

Procedure 3.             Install SUSE Linux Enterprise Micro

Step 1.                         Under Profiles or Servers, click the ellipses and open a vKVM.

Step 2.                         Power on or reset the server to boot from the ISO image defined in the vMedia policy.

Step 3.                         Follow the SUSE Linux Enterprise installation.

Note:      This document does not provide a comprehensive installation. Please use the default settings or the set-tings that fit best to your local installation.

Step 4.                         Configure the network adapters as required for the installation. We used DHCP to configure the network, so the configuration is left as it is.

Note:      It is required to configure NTP on all servers used for RKE.

Step 5.                         After the OS installation is finished the server will reboot.

Step 6.                         After the system is booted you can log on as user root.

Step 7.                         If not already done, register the systems with SUSE to get access to the online software repositories and updates:

SUSE K3s Installation on a Single Node

Procedure 1.             Prepare K3s installation

Step 1.                         Run the following command to install the required apparmor-parser package and reboot the system after the installation:

Step 2.                         Disable the Linux Firewall:

Step 3.                         Create base directory:

Procedure 2.             Install K3s

Step 1.                         Check the web site for the release that is available: https://github.com/k3s-io/k3s/releases/. We chose v1.26.9+k3s1:

Step 2.                         Install K3s:

Step 3.                         Check that the installation is finished by running the following commands:

Step 4.                         Copy the kubeconfig file to the management system:

Step 5.                         Change the server entry in the kubeconfig file to match the FQDN:

Step 6.                         Limit the permissions of the .kube/config file to be accessible from the owner only by running the following command:

Procedure 3.             Install the cert-manager for Rancher managed certificates

Step 1.                         Create the namespace for the cert-manager, default is cert-manager:

Step 2.                         Add the cert-manager crds to the system:

Step 3.                         Add the helm repository for the cert-manager to the system:

Step 4.                         Install the cert-manager:

Step 5.                         Check that the cert-manager installation has finished:

Procedure 4.             Install SUSE Rancher on K3s

Step 1.                         Create the namespace for the Rancher installation, default is cattle-system:

Step 2.                         Add the helm repository for Rancher to the system:

Step 3.                         List the available Rancher versions:

Step 4.                         Install Rancher:

Step 5.                         Monitor the Rancher installation:

Step 6.                         Run the following command to get the initial password for the admin user:

Step 7.                         Open a web browser and navigate to the hostname given at installation time, we used rancher.aa04.cspgb4.local.

Step 8.                         Set a new password for the admin user and accept the EUL.

The Rancher home screen displays with the list of available clusters:

Procedure 5.             Add an existing Kubernetes cluster to Rancher

To add or import existing Kubernetes clusters to Rancher follow these steps:

Step 1.                         Open the Rancher UI and click on Import Existing in the home screen.

Step 2.                         Click Generic.

Step 3.                         Enter a name for the cluster and click Create.

Step 4.                         Copy one of the command options on against the installed Kubernetes cluster:

Step 5.                         Check the installation by executing the following command:

The new cluster is now visible in Rancher:

Deploy and Configure NetApp Astra Trident

Astra Trident is an open-source, fully supported storage orchestrator for containers and Kubernetes distributions. It was designed to help meet the containerized applications’ persistence demands using industry-standard interfaces, such as the Container Storage Interface (CSI). With Astra Trident, microservices and containerized applications can take advantage of enterprise-class storage services provided by NetApp portfolio of storage systems. More information about Trident can be found here: NetApp Trident Documentation. 

Note:      In this solution, we validated NetApp Trident with ontap-nas driver using the NFS protocol. 

NetApp Astra Trident Deployment 

Procedure 1.             Deploy NetApp Astra Trident

Step 1.                         Download Trident software from GitHub and untar the .gz file to obtain the trident-installer folder:

Step 2.                         Create the trident namespace:

Step 3.                         Create the CRD:

Step 4.                         At this point, verify the operator, deployment, and replicasets are created:

Note:      If the Astra Trident deployment fails and does not bring up the pods to Running state, use the tridentctl logs -l all -n trident command for debugging.

Step 5.                         Verify the Server and Client version again:

Note:      Before configuring the backend that Trident needs to use for user apps, go to: https://docs.netapp.com/us-en/trident/trident-reference/objects.html#kubernetes-customresourcedefinition-objects to understand the storage environment parameters and its usage in Trident.

Procedure 2.             Configure the Storage Backend in Trident

Step 1.                         Configure the connection to the SVM on the NetApp storage array created for the RKE2 cluster. For more options regarding storage backend configuration, go to: https://docs.netapp.com/us-en/trident/trident-use/backends.html

Procedure 3.             Backend definition for ONTAP NAS drivers

Step 1.                         Create configuration for the Trident backend configuration:

Step 2.                         Deploy the Trident Backend configuration:

Step 3.                         Backend definition for ONTAP FlexGroup driver:

For information about FlexGroups and workloads that are appropriate for FlexGroups see the NetApp FlexGroup Volume Best Practices and Implementation Guide.

Step 4.                         Activate the backend storage configuration:

Step 5.                         Configure a storage class for ontap-nas and ontap-nas-flexgroup and make ontap-nas as default storage class:

Step 6.                         Create storage class configuration file:

Step 7.                         Create storage class for RKE cluster:

Step 8.                         Create Volume Snapshot class for the RKE cluster:

This completes the NetApp Astra Trident installation and configuration.

GPU Enablement and AI Workloads

This chapter contains the following:

·     Container Creation with NVIDIA Driver, CUDA-Operator, and Toolkit

·     Container Deployment

·     Deploy NetApp DataOps Toolkit

·     Deploy NVIDIA License Server

·     Deploy NetApp DataOps Toolkit

Graphics Processing Units (GPUs) provide unique capabilities for accelerating a broad spectrum of workloads, such as artificial intelligence and machine learning, scientific and engineering simulations, and many more. Developing and deploying these workloads in containers on Kubernetes clusters offers potential for improved portability, scale, and efficiency. However, GPU resources are not automatically visible or accessible to workloads running in containers.

Container Creation with NVIDIA Driver, CUDA-Operator, and Toolkit

To create the container images with the required NVIDIA drivers, the CUDA-Operator, and the CUDA-Toolkit based on SUSE Linux Enterprise Base Container Images (SLE BCI), a system called build node is required. The build node can be a bare-metal server or a virtual machine running SUSE Linux Enterprise – if possible, on the same version as the target RKE2 cluster.

Note:      Please double check the supported kernel versions for the NVIDIA drivers and CUDA software. In this validation we have used the unpatched SLE15 SP5 version due to some issues we saw with a patched kernel.

Procedure 1.             Create Container with NVIDIA driver, CUDA-Operator, and Toolkit

Step 1.                         Enable the Containers, Desktop Applications, and Development Tools modules:

Step 2.                         Install podman. Podman, or Pod Manager Tool) is a daemonless container engine for managing Open Container Initiative containers:

Step 3.                         Install Git. Git is a free and open source, distributed version control system that simplifies access to and management of source code:

Step 4.                         Create a file to set required versions for key components - ~/build-variables.sh.

◦    REGISTRY: URL of the registry where the new container image is to be saved

◦    SLE_VERSION: SUSE Linux Enterprise Base Container Images version

◦    SLE_SP: SUSE Linux Enterprise Base Container Images service pack number

◦    DRIVER_VERSION: NVIDIA GPU Driver version

Find the latest "Data Center Driver for Linux x64" version for your GPU at NVIDIA Driver Downloads.

◦    OPERATOR_VERSION: NVIDIA GPU Operator version

Find the associated NVIDIA GPU Operator version.

◦    CUDA_VERSION: the NVIDIA CUDA version appropriate for the selected NVIDIA GPU Driver

The CUDA version is listed under "Software Versions" when you find your driver version at NVIDIA Driver Downloads.

Step 5.                         Install required NVIDIA software package on each node:

Step 6.                         Source your build-variables.sh file:

Step 7.                         Clone the NVIDIA driver GitLab repository and change to the driver/sle15 directory:

Step 8.                         Locate the file, Dockerfile, and update it to reflect your build requirements:

Step 9.                         Run the podman build command, passing the necessary arguments:

Note:      Watch the build process for errors, warnings, and failures. You can ignore everything that is not stopping the build process. Through the validation of the solution, we phased an issue shown as “Please register your system” or “No source for 5.14.21 found” based on the newest kernel version used. Using an un-patched kernel fixed that issue.

Step 10.                     Verify that the build process finished successfully. You must see a message like this:

Step 11.                     Remove intermediate container images created during the build process:

Step 12.                     Add a tag to the image that will be used when deploying the image to a Kubernetes cluster:

Step 13.                     Push the newly built image to the container registry:

Step 14.                     Verify the image is saved in the registry, and remotely available:

Container Deployment

There are some software components required on the RKE2 nodes with NVIDIA GPUs installed.

Procedure 1.             Deploy Container

Step 1.                         Enable the Containers Module and NVIDIA Compute module on each worker node. The Desktop applications and Development tools module are mandatory to enable the NVIDIA compute module:

Step 2.                         Install required NVIDIA software packages on each node:

Step 3.                         Before starting the container deployment, make sure that no NVIDIA driver is loaded on all of the worker nodes:

Step 4.                         Unload the driver modules if required:

Step 5.                         Add the NVIDIA helm software repository to the management node (build node):

Step 6.                         Make sure your kubectl context points to the correct cluster:

Step 7.                         Deploy the NVIDIA GPU Operator with Helm:

Step 8.                         Verify the deployment:

Step 9.                         Validate the deployment:

Step 10.                     Validate that the NVIDIA GPU Driver is communicating with the GPU:

Response of the previous command:

 

 

 

 

 

 

 

 

 

Procedure 2.             (Optional) Run a Sample “CUDA VectorAdd” GPU Application

NVAIE containers are provided on https://catalog.ngc.nvidia.com/.

Step 1.                         Create a cuda-vectoradd.yaml file:

Step 2.                         Run the pod. (The pod starts, runs the vectorAdd command, and then exits):

Step 3.                         Check the logs from the container:

Output

Step 4.                         Delete the pod:

Deploy NVIDIA License Server

For the NVIDIA AI Enterprise software stack a license server is required to manage the license keys. Please follow these steps to deploy the NVIDIA license server in the environment.

Procedure 1.             Deploy NVIDIA license server

Step 1.                         From https://ui.licensing.nvidia.com/software, download and extract the VMware vSphere NLS License Server (DLS) OVA.

Step 2.                         Follow https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html to install the DLS appliance in the FlexPod-Management cluster.

Step 3.                         Once the DLS instance is installed and powered on, proceed to https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html#registering-dls-administrator-user.

Step 4.                         Proceed to https://docs.nvidia.com/license-system/latest/nvidia-license-system-user-guide/index.html#configuring-service-instance and work through Creating a License Server on the NVIDIA Licensing Portal (making sure to add a feature with a number of licenses to cover the number of vGPUs you plan to deploy), DLS Instance Instructions, Registering an on-Premises DLS Instance with the NVIDIA Licensing Portal, Binding a License Server to a Service Instance, and Installing a License Server on a DLS Instance.

Deploy NetApp DataOps Toolkit

Procedure 1.             Prerequisites

The NetApp DataOps Toolkit for Kubernetes requires that Python 3.8 or above be installed on the local host. Additionally, the toolkit requires that pip for Python3 be installed on the local host. For more details regarding pip, including installation instructions, refer to the pip documentation.

Step 1.                         Check the installed Python version:

Step 2.                         Create a Python 3 virtual environment for the NetApp DataOps Toolkit:

Step 3.                         Use the new Python 3 virtual environment:

Step 4.                         Install the latest version of pip into this Python virtual environment:

Step 5.                         To install the NetApp DataOps Toolkit for Kubernetes, run the following command:

Step 6.                         Confirm that the toolkit installed successfully:

Step 7.                         Now, create a test data scientist workspace using Toolkit:

Step 8.                         Access the workspace, using the URL http://10.104.3.91:31526 and enter the password configured during workspace creation.

Note:      In the later section, for validating AI use cases, Jupyter Notebook will be deployed using NetApp DataOps toolkit.

 

Solution Validation

This chapter contains the following:

·     Generative Inferencing AI Model Deployment and Results

Generative Inferencing AI Model Deployment and Results

Stable Diffusion

Stable Diffusion is an open-source image generation model that allows to generate images using a simple text prompt.

Stable Diffusion is a text-to-image latent diffusion model created by the researchers and engineers from CompVis, Stability AI and LAION. It is trained on 512x512 images from a subset of the LAION-5B database. LAION-5B is the largest, freely accessible multi-modal dataset that currently exists.

Stable Diffusion performs the following image related tasks:

·     Text-to-Image: Create an image from a text prompt. 

·     Image-to-Image: Create an image from an existing image and a text prompt.

·     Depth-Guided Diffusion: Modify an existing image with its depth map and a text prompt.

·     Instruct Pix2Pix: Modify an existing image with a text prompt.

·     Stable UnCLIP Variations: Create different versions of an image with a text prompt.

·     Image Upscaling: Create a high-resolution image from an existing image with a text prompt.

·     Diffusion Inpainting: Modify specific areas of an existing image with an image mask and a text prompt.

Procedure 1.             Create a workspace

Step 1.                         Create a Jupyter notebook using NetApp DataOps toolkit:

Note:      The workspace creates a pod with pytorch image and will use ontap-gold storage class for the persistent volume claim (PVC), 1 GPU will be assigned to this workspace.

Step 2.                         Run the following code on the notebook:

 

Figure 5.       Image generated by Stable Diffusion 1.4

The inferencing was run with one A100 GPU, 10% of tensor core utilization with 3.6 Gigabyte of memory utilization:

Python script for Stable Diffusion XL1.0:

Figure 6.       Image generated by Stable Diffusion XL 1.0

The inferencing was run with one A100 GPU, 48% of tensor core utilization with 14.7 Gigabyte of memory utilization.

Dreamlike Photoreal 2.0

Dreamlike Photoreal 2.0 is a photorealistic model based on Stable Diffusion 1.5, made by dreamlike.art. The same inferencing method was used as Stable Diffusion.

Hugging face: https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0

Figure 7 is generated by the prompt “photo, a church in the middle of a field of crops, bright cinematic lighting, gopro, fisheye lens.”

Figure 7.       Dreamlike photoreal 2.0 image

The inferencing was run with one A100 GPU, 98% of tensor core utilization with 7.7 Gigabyte of memory utilization.

Openjourney

Openjourney is an open-source Stable Diffusion fine-tuned model on Midjourney images. Model used in this validation is prompthero/openjourney.

Figure 8 is generated for the prompt “racing series of different cars with different colors and shapes, mdjrny-v4 style”.

Figure 8.       Openjourney generated image

The inferencing was run with one A100 GPU. 97% of tensor core utilization with 3.6 Gigabyte of memory was consumed.

Hotshot-XL

Hotshot-XL is an AI text-to-GIF model trained to work alongside Stable Diffusion XL. Hotshot-XL was trained to generate 1 second GIFs at 8 FPS.

Hotshot-XL can generate GIFs with any fine-tuned SDXL model. It is possible to make GIFs with any existing or newly fine-tuned SDXL model.

Git repo: https://github.com/hotshotco/Hotshot-XL

Hugging face: https://huggingface.co/hotshotco/Hotshot-XL

Procedure 1.             Create a new Jupyter notebook

Step 1.                         Use the DataOps toolkit and launch the terminal to run the commands:

Step 2.                         Environment Setup:

Step 3.                         Install git-lfs:

Step 4.                         Download the Hotshot-XL Weights:

Step 5.                         Download our fine-tuned SDXL model (or BYOSDXL):

Step 6.                         Text-to-GIF:

Figure 9.       A GIF image is generated by the text prompt

The inferencing was run with one A100 GPU. 99% of tensor core utilization with 12.7 Gigabyte of memory was consumed:

Text Generation Inference

Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation for the most popular open-source LLMs, including Llama, Falcon, StarCoder, BLOOM, GPT-NeoX, and T5.

Figure 10.    TGI Model Representation

A list of supported models can be found here.

Table 11.       Validated models

Step 7.                         Create a PVC for the container.

Step 8.                         Deploy tgi container from the below yaml file:

Step 9.                         Apply the file to create service:

Step 10.                     Run the command from the management host for inferencing:

Results

The TGI container contains a python benchmark script that was run for each of the models utilizing one GPU with results in the following table. Each model was first loaded from the container with:

From another window in the container:

Table 12.       TGI Benchmark Results

NVIDIA NeMo Framework

NVIDIA NeMo™ Framework is a development platform for building custom generative AI models. The framework supports custom models for language (LLMs), multimodal, computer vision (CV), automatic speech recognition (ASR), natural language processing (NLP), and text to speech (TTS). NVIDIA NeMo framework is a scalable and cloud-native generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech). It enables users to efficiently create, customize, and deploy new generative AI models by leveraging existing code and pretrained model checkpoints.

Note:    NVIDIA NeMo inferencing, which uses Triton Inferencing Server, was implemented along with three AI models.

Table 13.       Model information

Procedure 1.             Deployment

To deploy Nemo framework container for hf to nemo conversion, a nemo namespace was created. NGC secret to pull images from NVIDIA registry was created.

Step 1.                         Create a PVC in the same namespace.

Note:      The PVC needs to be configured with access mode ReadWriteMany as it will be shared across multiple containers.

Step 2.                         Deploy Nemo framework container from the below yaml file. This container is used to convert various models from .hf to .nemo.

Step 3.                         Download Llama-2 models from hugging face. We downloaded and converted Llama-2-7b-chat and Llama-2-13b-chat models.

Note:      You need to login to hugging face cli by providing the access token to download Llama-2 models:

Step 4.                         Deploy Nemo Inference container using the deployment file:

Step 5.                         To run NeMo Triton Inferencing server with a model, such as Llama-2-7B, run the following from the /opt/NeMo directory within the container:

The following message displays once the model is loaded:

Step 6.                         Place the example in a python file and run a test python script:

Output

Results

The NeMo Inference container contains a python benchmark script that was run for both models utilizing one GPU with results in the following table. To launch the benchmark script, run the following from the /opt/NeMo directory within the container.

Note:      Delete the trt_llm_model_dir directory between benchmark runs; each run will create a new directory with the provided name in the script.

Step 7.                         For Nemotron-3-8B-QA-4k, run the following:

Step 8.                         To run NeMo Triton Inferencing server with Nemotron-3-8B-QA-4k, run the following command from /opt/nemo directory within the container:

The following message displays once Nemotron-3-8B-QA-4k model is loaded:

Step 9.                         Run the benchmark script:

Note:      For the NeMo models, the model_type is “gptnext.”

Table 14.       NeMo Framework Benchmark Results with 1 GPU: Input Tokens Length: 128 and Output Tokens Length: 20

Table 15.       NeMo Framework Benchmark Results with 1 GPU: Input Tokens Length: 2048 and Output Tokens Length: 300

While running the benchmark for Llama-2-7b-chat with one A100 GPU, 94% of tensor core utilization with 39.1 Gigabyte of memory was consumed:

Storage performance while running benchmark script:

While running the benchmark for Nemotron-3-8B-QA with one A100 GPU, 91% of tensor core utilization with 83.3 Gigabyte of memory was consumed:

Retrieval-Augmented Generation (RAG) on NVIDIA L40S GPU

Retrieval-augmented generation (RAG) is a software architecture that combines the capabilities of large language models (LLMs), renowned for their general knowledge of the world, with information sources specific to a business, such as documents, SQL databases, and internal business applications. RAG enhances the accuracy and relevance of the LLM’s responses. RAG uses vector database technology to store up-to-date information that’s retrieved using semantic searches and added to the context window of the prompt, along with other helpful information, to enable the LLM to formulate the best possible, up-to-date response. This architecture has become famous for many use cases, including its ability to offer detailed, relevant answers by integrating the best of both worlds—knowledge from LLMs and proprietary business data.

Step 1.                         Deploy a Jupyter notebook using NetApp DataOps toolkit:

Step 2.                         Launch the notebook and install the libraries:

Step 3.                         Import the libraries:

Step 4.                         For the embedding, we used sentence-transformers/all-mpnet-base-v2:

Step 5.                         Load Deci/DeciLM-7B-instruct model and create pipeline:

Note:      We created a folder in Jupyter notebook with the name cvd and have uploaded latest CVDs.

Step 6.                         Provide the path of the folder:

Step 7.                         Text splitter with chunk size:

Step 8.                         A database is required to store embeddings and to efficiently search for them. Therefore, for storage and searching purposes, we need vector stores. There are many vector stores integrated with LangChain, we have used Facebook AI Similarity Search (“FAISS”) vector store.

Step 9.                         Performs a similarity check and returns the top K embeddings that are similar to the question’s embeddings:

Step 10.                     Once the data is in the database, the LLM model is prepared, and the pipeline is created, then we need to retrieve the data. A retriever is an interface that returns documents from the query:

Output

Step 11.                     Create a custom template, prompt template and qa chain:

Step 12.                     Run the query:

Output

While running the question and answering with one L40S GPU, 94% of tensor core utilization with 36.6 Gigabyte of memory was consumed:

Resnet32

Deep learning has evolved a lot in recent years and we all are excited to build deeper architecture networks to gain more accuracies for our models. These techniques are widely tried for Image related works like classification, clustering, or synthesis.

Resnet models were proposed in “Deep Residual Learning for Image Recognition.” Restnet32 is pre-trained on the ImageNet dataset which contains 100,000+ images across 200 different classes. Here we have the 5 versions of Resnet models, which contain 18, 34, 50, 101, 152 layers, respectively. In this document, we will look at one of the healthcare use case (Diabetic Retinopathy Detection) of ResNet-32 model using the Tensor flow framework in Python.

Sample training and test dataset: https://www.kaggle.com/code/balajiai/diabetic-retinopathy-detection-using-pytorch/input

Training and validation: https://www.kaggle.com/code/balajiai/diabetic-retinopathy-detection-using-pytorch/notebook

Step 1.                         Deploy a Jupyter notebook with the help of DataOps toolkit:

91% accuracy was achieved on the training set and 80% accuracy on validation set. The model was optimized using 60 Epochs.

Figure 11.    Training and validation loss against epochs

The inferencing was run with one A100 GPU, 48% of tensor core utilization with 24.3 Gigabyte of memory was consumed in each epoch:

Llama 2

Llama 2 is a pretrained and fine-tuned text generation models based on autoregressive, transformer architecture. Llama 2 comes with 3 models with 7 billion, 13 billion and 70 billion parameters. Llama 2 is an opensource model which is at par in performance with ChatGPT. In this validation we have used Llama-2-7b-chat model. PyTorch was used to run the model which was created by NetApp DataOps toolkit.

Procedure 1.             Deployment

Step 1.                         Build a modified PyTorch NGC container using podman and push it to a container registry:

Step 2.                         To prepare to download the Llama 2 models, go to: https://llama.meta.com/llama-downloads and fill out the form. Download instruction will be shared via email. Create a “llama-2” namespace in Rancher for RK2E-BM cluster and create a 400GB PVC using FlexGroup storage class.

Step 3.                         Deploy a TGI container using the deployment file:

Step 4.                         After the pod is running, login to container shell from the Rancher and execute the below commands to download the model.

Result

When running the commands, the Llama-2-7B-Chat model with one GPU loaded in 4.72 seconds and ran in 21.861 seconds.

Conclusion

This solution design is based on Cisco FlexPod for RKE2 running on SUSE Linux Enterprise 15 for production-ready deployment processes with latest best practices, a stable, highly available environment to run enterprise-grade application containers and AL/ML workloads.

FlexPod is the optimal shared infrastructure foundation to deploy a variety of IT workloads. It is built on leading computing, networking, storage, and infrastructure software components. The integration of FlexPod converged infrastructure with SUSE RKE2 and Rancher provides a very good starting point for enterprise IT to make in-roads into DevOps and CI/CD model for application development to address immediate business needs and reducing time to market. Enterprises can accelerate on the path to an enterprise-grade Kubernetes solution with SUSE RKE2 running on Cisco FlexPod infrastructure.

In this solution, we tested various Generative AI use cases and provided the results of the various tests for different models. Running containerized application along with Generative AI workloads on FlexPod can give maximum ROI.

About the Authors

Ulrich Kleidon, Principal Engineer, UCS Solutions, Cisco Systems

Ulrich Kleidon is a Principal Engineer for Cisco's Unified Computing System (Cisco UCS) solutions team and a lead architect for solutions around converged infrastructure stacks, enterprise applications, data protection, software-defined storage, and Hybrid-Cloud. He has over 25 years of experience designing, implementing, and operating solutions in the data center.

Abhinav Singh, Sr. Technical Marketing Engineer, Hybrid Cloud Infra & OEM Solutions, NetApp

Abhinav Singh is a Senior Technical Marketing Engineer for FlexPod solutions team and has more than 14 years of experience in Data Center infrastructure solutions which includes On-prem and Hybrid cloud space. He focuses on the designing, validating, implementing, and supporting converged infrastructure and hybrid cloud infrastructure solutions. Abhinav holds a bachelor's degree in electrical and electronics.

Acknowledgements

 

Appendix

This appendix contains the following:

·     FlexPod Backups

·     Compute

·     NVIDIA

·     Network

·     Storage

·     SUSE Container Platform

·     Interoperability Matrix

FlexPod Backups

Cisco Intersight SaaS Platform

Cisco Intersight SaaS platform maintains your configurations online. No separate backup was created for UCS configuration. If you are using an Intersight Private Virtual Appliance (PVA), ensure that the NetApp SnapCenter Plugin for VMware vSphere is creating periodic backups of this appliance.

Cisco Nexus Backups

The configuration of the Cisco Nexus 9000 switches can be backed up manually at any time with the copy command, but automated backups can be enabled using the NX-OS feature scheduler. 

An example of setting up automated configuration backups of one of the NX-OS switches is shown below:

Note:      Using “vrf management” in the copy command is only needed when Mgmt0 interface is part of VRF management.

Verify the scheduler job has been correctly setup using following command(s):

The documentation for the feature scheduler can be found here: https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/102x/configuration/system-management/cisco-nexus-9000-series-nx-os-system-management-configuration-guide-102x/m-configuring-the-scheduler-10x.html

SUSE Rancher Backup

The configuration of the embedded SUSE Rancher system backup is documented here: https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher and uses a S3 bucket as backup target.

There are alternate options such as NetApp Astra Control Center to protect the SUSE Rancher system and the deployed workloads.

Compute

Cisco Intersight: https://www.intersight.com

Cisco Intersight Managed Mode: https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Intersight_Managed_Mode_Configuration_Guide.html

Cisco Unified Computing System: http://www.cisco.com/en/US/products/ps10265/index.html

Cisco UCS 6536 Fabric Interconnects: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs6536-fabric-interconnect-ds.html

NVIDIA

NVIDIA H100 GPU: https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet

NVIDIA A100 GPU: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-nvidia-us-2188504-web.pdf

NVIDIA L40S GPU: https://resources.nvidia.com/en-us-l40s/l40s-datasheet-28413

NVAIE: https://www.nvidia.com/en-in/data-center/products/ai-enterprise/

NVAIE Product Support Matrix: https://docs.nvidia.com/ai-enterprise/latest/product-support-matrix/index.html#support-matrix__suse-linux-enterprise-server

Network

Cisco Nexus 9000 Series Switches: http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html

Storage

NetApp ONTAP: https://docs.netapp.com/ontap-9/index.jsp

NetApp AFF A series: https://www.netapp.com/data-storage/aff-a-series/ss

NetApp Astra Trident: https://docs.netapp.com/us-en/trident/

NetApp DataOps Toolkit: https://github.com/NetApp/netapp-dataops-toolkit/tree/main/netapp_dataops_k8s

SUSE Container Platform

SUSE Rancher Enterprise Container Management: https://www.suse.com/solutions/enterprise-container-management/#rancher-product

SUSE Linux Enterprise Server OS: https://www.suse.com/products/server/

SUSE Linux Enterprise Micro OS: https://www.suse.com/products/micro/

RKE2: https://docs.rke2.io/

K3S: https://docs.k3s.io/  

Interoperability Matrix

Cisco UCS Hardware Compatibility Matrix: https://ucshcltool.cloudapps.cisco.com/public/  

NetApp Interoperability Matrix Tool: http://support.netapp.com/matrix/  

Feedback

For comments and suggestions about this guide and related guides, join the discussion on Cisco Community at https://cs.co/en-cvds.

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLE-MENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_P1)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)