FlashStack with VMware vSphere 9.1 using Cisco UCS M8 Servers Deployment Guide

 

Published: August 2026

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to: https://www.cisco.com/go/designzone

Executive Summary

The FlashStack solution is a validated, converged infrastructure solution developed jointly by Cisco and Everpure. The solution offers a predesigned data center architecture that incorporates computing, storage, and network design best practices to reduce IT risk by validating the architecture and helping ensure compatibility among the components. The FlashStack solution is successful because of its ability to evolve and incorporate both technology and product innovations in the areas of management, compute, storage, and networking. Some of the key advantages of FlashStack Converged Infrastructure with VMware vSphere Foundation (VVF):

●     Cloud-Ready and programmable Infrastructure: Integration with Cisco Intersight and Everpure Pure1 enables FlashStack to offer simplified cloud-based management, programmable infrastructure, and continuous feature delivery.

●     Consistent Performance and Cost Savings: FlashStack delivers higher and consistent performance across all the layers of the stack (compute, network, storage and Hypervisor) by consistently adapting the innovations across all the layers of the solution. Powered by the latest generation AMD CPUs, Cisco UCS compute provides dense compute packed with high core count and provides consistent performance for latency critical applications while Everpure FlashArray delivers consistent sub-millisecond IO latency for enterprise applications leveraging its 100% NVMe enterprise flash storage.

●     End-to-End 64Gbps Fibre Channel and 100Gbps Ethernet network: Using 6^th Generation Cisco UCS 6664 Fabric Interconnects (FIs) and 5^th Generation Cisco UCS VIC cards, the solution deliver native 64Gbps Fibre Channel and 100Gbps Ethernet connectivity from the servers through the network to the storage.

●     Confidential Computing with AMD CPUs: This solution utilizes AMD Secure Encrypted Virtualization (SEV) and Secure Nested Paging (SEV-SNP) within VMware vSphere to protect data-in-use via hardware-isolated Trusted Execution Environments (TEEs). By cryptographically isolating and encrypting VM memory and CPU runtime states, it eliminates the hypervisor and host administrators from the guest's trust boundary.

●     Modernize IT landscape VMware vSphere Foundation (VVF):  VMware vSphere Foundation (VVF) combines compute, storage, networking and offers a fully integrated and unified virtualization platform helps you achieve workloads modernization. By maximizing data center capacity utilization, improving lifecycle management, running modern applications on a unified platform, enhancing workload performance, and ensuring compliance, VVF empowers organizations to leverage the full potential of private cloud environments. It also acts as the base steppingstone for building a fully integrated private cloud and scaling into the VMware Cloud Foundation.

This document explains the deployment details of VMware vSphere Foundation (VVF) 9.1 on FlashStack Converged Infrastructure incorporating the Cisco Unified Computing System compute nodes, Cisco Nexus  and MDS Switches, and Everpure FlashArray. This guide covers two deployment paths for VVF 9.1: an automated method using the VCF installer, which handles full cluster provisioning and VCF management service configuration, and a manual method for deploying a standalone VVF 9.1 cluster (without management services). It covers configuration steps for various features and technologies including and not limited to Memory Tiering, configuration and allocation of vGPUs to the AI/ML workloads VMs, Confidential Computing and configuration steps for various storage access protocols including iSCSI, FC, NVMe over TCP, NVMe over FC and NFS.

The solution is delivered as Infrastructure as Code (IaC) to eliminate error-prone manual tasks, allowing quicker and more consistent solution deployments. See FlashStack Automation with Ansible for detailed instructions to automate FlashStack components. If you’re interested in understanding the FlashStack design and deployment details, including the configuration of various elements of design and associated best practices, see the Cisco Validated Designs for FlashStack here: https://www.cisco.com/site/in/en/solutions/computing/converged-infrastructure/pure-storage/resources.html

Solution Overview

This chapter contains the following:

●     Audience

●     Purpose of this document

●     New in this release

FlashStack with VMware vSphere Foundation represents a cohesive and flexible validated converged infrastructure solution that combines compute, network, and storage resources into a single, integrated architecture. Designed as a collaborative effort between Cisco, Everpure, and VMware by Broadcom, this converged infrastructure platform is engineered to deliver high levels of efficiency, scalability, and performance, suitable for a multitude of data center workloads. By standardizing on a validated design, organizations can accelerate deployment, reduce operational complexities, and confidently scale their IT operations to meet evolving business demands.

The FlashStack architecture leverages the Cisco Unified Computing System (Cisco UCS) servers, Cisco Nexus networking, Everpure’s innovative storage systems, and VMware vSphere Foundation (VVF) providing a robust foundation for virtualized environments.

Audience

The intended audience for this document includes, but is not limited to IT architects, sales engineers, field consultants, professional services, Cloud Native teams, IT managers, IT engineers, partners, and customers who are interested in taking advantage of an infrastructure built to deliver IT efficiency and enable IT innovation.

Purpose of this document

This document provides deployment guidance for setting up the FlashStack solution with VMware vSphere Foundation (VVF). This document introduces various design elements and explains various considerations and best practices for a successful deployment of VVF 9.1 cluster using Cisco UCS and Everpure Storage.

New in this release

Some of the highlights of FlashStack with VMware by Broadcom are:

●     Support for VMware vSphere Foundation 9.1: VVF 9.1 focuses on modernizing infrastructure economics by significantly reducing total cost of ownership (TCO) and operational downtime through advanced automation and hardware optimizations. VVF 9.1 comes with key features and enhancements such as Native NVMe Memory Tiering, Topology-aware scheduling, vSphere Kubernetes Service (VKS), software NVMe mirroring, Zero-Touch Provisioning and so on, making it more robust, scalable and agile platform for hosting traditional and modern applications.

●     Securing VM workloads with FlashStack, vSphere and AMD SEV-SNP: The FlashStack solution delivers an advanced, multi-layered security architecture that safeguards virtualized environments across two critical vectors. For data-at-rest protection, VMware vSphere leverages native security features—including virtual Trusted Platform Modules (vTPM) and the Native Key Provider (NKP)—to enable selective, hypervisor-level virtual machine encryption. For data-in-use protection, AMD SEV-SNP facilitates the deployment of Confidential VMs. By utilizing a hardware-based Trusted Execution Environment (TEE), this technology isolates active memory, shielding sensitive workloads from potential vulnerabilities in the hypervisor, host operating system, and cloud management layers.

●     Seamless Solution Integration: All the components of the solution are deeply integrated to enable seamless vSphere cluster provisioning, cluster expansion and lifecycle management of the entire solution. Together, Cisco Intersight integration vCenter and Everpure’s FlashArray integration for vSphere Web Client plugin provides operational agility for VMware users and increased application performance and availability for virtual machines. Integration of Intersight with vCenter is another standout capability. The integration unifies server hardware management directly within the vSphere Client by leveraging the Hardware Support Manager (HSM) alongside vSphere Lifecycle Manager (vLCM) to seamlessly update base OS images, drivers, and server firmware. Concurrently, it functions as a native VMware Proactive HA provider, continuously monitoring critical hardware components for impending faults. Upon detecting an anomaly, the plugin automatically alerts vCenter to place the degraded host into Quarantine or Maintenance Mode, triggering live vMotions to protect running virtual machines from unexpected downtime. With the enhanced Everpure FlashArray plug-in, VMware administrators can automate storage provisioning, configuration, and most of the day-1 or day-2 operations from the same vCenter console.

●     Scalability and Consistent performance: The modular architecture of FlashStack enables independent scaling of compute and storage resources, allowing organizations to meet the ongoing business demands of modern IT applications. Cisco UCS’s modular and stateless computing combined with Everpure FlashArray’s modular NVMe architecture delivers a high-performance solution that is ideally suited for mission-critical workloads with varying I/O requirements. In addition, this architecture utilizes NVMe over Fabric (NVMe-oF) protocol over TCP (NVMe/TCP) and Fibre Channel (NVMe/FC) extending the high-performance benefits of NVMe architecture of.

●     Operational efficiency and Consistent infrastructure configuration: The solution provides unified management through Cisco Intersight, vCenter, VCF Operations and Pure1. This integrated approach enables centralized monitoring, orchestration, management, and automation across compute, storage, and virtualization layers, simplifying operations and enhancing visibility for IT administrators. In addition to the compute-specific hardware and software innovations, integration of the Cisco Intersight cloud platform with Everpure FlashArray and Cisco Nexus delivers monitoring, orchestration, and workload optimization capabilities for different layers of the FlashStack solution.

●     Everpure FlashArray//XL170 R5: utilizes ultra-high-speed 200Gbps Ethernet and 64G FC (Fibre Channel) connectivity to deliver massive throughput and ultra-low latency for demanding, Tier-1 environments. These enterprise connectivity options are specifically designed for High-Performance workloads like databases, Next-Generation NVMe over Fabric deployments (NVMe-over-TCP, NVMe-over-FC, NVMe-over-RoCE and so on), large scale virtualization and workload consolidation.

Technology Overview

This chapter contains the following:

●     FlashStack Converged Infrastructure Components

●     VMware vSphere Foundation

FlashStack Converged Infrastructure Components

All FlashStack components are integrated, allowing you to deploy the solution quickly and economically while eliminating many of the risks associated with researching, designing, building, and deploying similar solutions from the foundation. One of the main benefits of FlashStack is its ability to maintain consistency at scale. Figure 1 illustrates the series of hardware components used for building the generic FlashStack architectures. Each of the component families; Cisco Unified Computing System (Cisco UCS), Cisco Nexus, Cisco MDS, Everpure FlashArray and FlashBlade systems, offers platform and resource options to scale-up or scale-out the infrastructure while supporting the same features and functions. For more specific details on the FlashStack architecture, go to: FlashStack.

VMware vSphere Foundation

VMware vSphere Foundation (VVF) is an enterprise-grade workload platform for modern infrastructure delivering virtualization benefits, simplified management, cost efficiency, scalability and serves as core foundation to VMware Cloud Foundation (VCF). It is designed to meet the needs of organizations of all sizes, providing a flexible and scalable solution for your IT infrastructure. It bundles core VMware technologies to deliver a powerful set of capabilities:

●     Compute Virtualization: It provides a full compute virtualization stack to create and manage virtual machines, allocating resources dynamically to meet workload demands.

●     Basic Network Virtualization: VVF provides network connectivity through vSphere Standard Switch (VSS) and vSphere Distributed Switch (VDS). NSX-based software-defined networking — including overlay networking, distributed firewall, and logical routing — is not included in the VVF entitlement and requires VMware Cloud Foundation (VCF).

●     Intelligent operations management: VCF Operations provides deep observability into data center performance, health, and capacity analytics. It shifts your strategy from reactive to proactive management by correlating metrics across the full infrastructure stack. The platform streamlines root-cause analysis with advanced diagnostic troubleshooting and surfaces clear resource-reclamation paths. Ultimately, this allows engineering teams to maximize workload density and continuously optimize cluster efficiency.

VCF installer a unified virtual appliance introduced in VMware Cloud Foundation (VCF) 9.0 to plan, configure, and deploy the entire VCF and vSphere Foundation stack including VVF. It replaces the older Cloud Foundation Builder tool and eliminates the need for complex Excel-based parameter workbooks.

VVF can be deployed in two methods:

●     Automated Deployment with VCF Installer: The VCF Installer automatically deploys and configures all the core VVF components, including vCenter, VCF Operations, and License Manager. It also deploys and configures key VCF management services, such as Fleet Lifecycle Management, Log Management, and Telemetry, and registers these services with the VVF platform. Once the installation is complete, all components and management services are fully integrated and ready to use for centralized management and lifecycle operations of the VVF cluster. To enable this level of automation, the VCF Installer requires a standardized and predefined infrastructure configuration, including networking and storage constructs. As a result, the automated deployment workflow provides limited flexibility for manually customizing networking, cluster configuration, or specific storage options during the deployment. For example, when using an external storage array, installation via VCF installer supports only a defined set of initial storage protocols, such as NFS and Fibre Channel (FC), as part of the automated deployment process. However, when the VVF is deployed and operational, additional storage (iSCSI, NFS, NVMeoF) can be configured. For more details on supported components of VVF cluster and VCF management services, see Components in VCF and VVF.

●     Manual Deployment: All the supported components of VVF cluster needs to be installed and configured individually. If required, supported VCF management services (VCF Operations and License Server) can be individually deployed, configured and registered with the VVF cluster. All the types of vSphere supported storage protocols can be configured. For more details on manual deployment without VCF management services, go here.

This document describes the deployment using both methods.

Cisco UCS is an integrated data center platform that combines compute, networking, and storage into a single, centrally managed system, simplifying management, reducing complexity, and improving efficiency for diverse workloads like virtualization, cloud, and AI. Key components include servers (blades, rack, modular) and Fabric Interconnects (FIs) for unified connectivity.

FlashStack with VMware vSphere Foundation Integrates Cisco UCS servers (Cisco UCS C-Series, Cisco UCS B-Series, and Cisco UCS X-Series) as compute nodes with VMware ESXi and external Everpure FlashArray, creating a flexible, disaggregated infrastructure managed by Cisco Intersight.

●     Cisco UCS compute solutions are based on the architectural concept of stateless computing. Stateless computing with Cisco UCS and Intersight uses server profiles to abstract the identity and configuration of servers, allowing for flexible and dynamic server management. This approach enables consistent compute configuration across the datacenter and allows you to repurpose the existing servers or replace the faulty servers with new servers without needing to reconfigure LAN/SAN switches or manually input server identities. Booting ESXi hosts from FlashArray further amplifies this benefit and greatly reduces the server provisioning times by multifold.

●     New workloads can be deployed seamlessly by cloning and assigning existing profiles, which ensures exact configuration consistency across the data center.

●     Updating firmware, BIOS or any server identity or setting requires updating the Service Profile, which automatically pushes changes to the associated hardware.

Everpure FlashArray provides storage capabilities that supports persistent application and user data, data resiliency, data availability, data security, and data efficiency.

●     The Everpure Platform delivers a unified, cloud-like storage-as-a-service foundation that consolidates block, file, and object with simple, centralized management, cutting complexity and accelerating outcomes.

●     Built on Evergreen architecture for non‑disruptive upgrades (even with in-place upgrades) and zero‑downtime operations, it future-proofs the estate while driving cost, space, and energy efficiency so teams can focus on innovation instead of migrations.

●     It’s always-on global compression and deduplication deliver industry-leading data efficiency, often requiring significantly less hardware than alternatives. The platform’s modular NVMe architecture enables true disaggregated simplicity, separating compute and storage for sub-millisecond latency across all workloads. 

●     With integration into VMware vCenter through the Everpure vSphere Web Client Plugin, administrators can provision and manage FlashArray-backed storage, apply protection policies, monitor datastores, and perform supported snapshot-based VM recovery workflows from a single interface. This simplifies Day 1 and Day 2 operations. FlashArray has demonstrated six-nines availability across the Pure1 installed base, including maintenance, failures, and generational upgrades, while Evergreen enables non-disruptive in-place upgrades. Supported FlashArray configurations now scale with DirectFlash Modules up to 300TB per module, providing higher capacity density with fewer devices.

Cisco Cloud Control

Cisco Cloud Control is a unified, AI-native management platform that converges identity, governance, and topology across every IT domain (compute, networking, observability, and security), enabling governed, agentic execution at scale. The key components include AI Assistant, AI Canvas, AgenticOps Framework, Cisco Intersight integration and so on. AI Assistant is a natural language interface that provides context-aware guidance across domains and escalates complex issues into AI Canvas. AI Canvas is a multiplayer, generative workspace where human operators and AI agents collaborate to investigate and resolve cross-domain issues.

The screenshot below demonstrates the strength of Cisco Cloud Control AI Canvas in delivering cross-domain operational intelligence by transforming raw infrastructure data and configuration states into actionable troubleshooting guidance. The AI Canvas analyzed the Pro’s and Con’s of available methods for claiming Nexus switches into Intersight, identified the root cause of the issue and finally recommended re-claiming the Nexus switches using “Claim Target with Assist” through an Intersight Assist appliance.

Cisco Intersight

Cisco Intersight is a lifecycle management platform for your infrastructure, regardless of where it resides. In your enterprise data center, at the edge, in remote and branch offices, at retail and industrial sites—all these locations present unique management challenges and have typically required separate tools. Cisco Intersight Software as a Service (SaaS) unifies and simplifies your experience of Cisco UCS.

In addition to the Cisco Intersight SaaS platform, air-gapped Cisco Intersight Private Virtual Appliance (PVA) and Cisco Intersight Connected Virtual Appliance (CVA) are also supported for managing hardware infrastructure and support the deployment of vSphere cluster on FlashStack. If an air-gapped Cisco Intersight Private Virtual Appliance is used, updates and downloads must be managed manually. The firmware bundles for all the supported hardware platforms such as Cisco UCS C-Series, Cisco UCS X-Series must be uploaded to the PVA software repository.

Everpure Pure1

Everpure Pure1 is a cloud-based, AI-driven SaaS platform designed to simplify and optimize data storage management for Everpure arrays. It offers features such as proactive monitoring, predictive analytics, self-service upgrades, and automated tasks.

●     Provides a centralized intuitive interface for monitoring and managing all your Everpure FlashArrays, FlashBlades, Portworx integrating capabilities for capacity management, security monitoring, data protection, and troubleshooting—all in one place.

●     Provides proactive recommendations before the storage array faces an issue. The Self-Service Upgrades enable eligible FlashArray and Everpure Cloud appliances to be upgraded on the customer’s schedule, with automated health checks and guided workflows for supported configurations.

●     Pure1’s appliance Genealogy feature tracks your hardware evolution from installation through upgrades and sends reminders about upcoming renewals, helping you avoid lapses in support coverage.

●     Offers robust support for identifying bottlenecks across virtual disks, datastores, hosts, and physical arrays, whether the issues are in the storage or virtualization layer.

●     VM Analytics: Pure1 helps you narrow down the troubleshooting steps in your virtualized environment. VM Analytics provides you with a visual representation of the IO path from the VM all the way through to the FlashArray. Other tools and features guide you through identifying where an issue might be occurring to help eliminate potential candidates for a problem. VM Analytics doesn’t only help when there’s a problem. The visualization allows you to identify which volumes and arrays particular applications are running on. This brings the whole environment into a more manageable domain.

Everpure Integration with Cisco Intersight

Cisco Intersight supports management and monitoring of non-Cisco infrastructure such as Everpure FlashArray. These devices are integrated into Cisco Intersight using a virtual appliance called “Intersight Assist.” To deploy Intersight Assist Virtual Appliance and integrate the Everpure FlashArray and vCenter, see Installing and configuring Intersight Assist.

Everpure is the first non-Cisco storage infrastructure integrated into Cisco Intersight. One can add or remove the required storage widgets which provide high level information and insights for the storage arrays being managed by the Cisco Intersight. You can view the complete inventory of the FlashArray including Hosts, host Groups, volume, controllers, Drives and ports and so on.

Intersight allows infrastructure administrators to automate storage and storage-related virtualization tasks through workflows using the Intersight Workflow Designer. Cisco Intersight provides an orchestration engine to build workflows to automate some of the typical datacenter administration tasks. A workflow stitches together the components of various devices in a particular sequence to accomplish a particular task. For example, following shows a pre-defined workflow designed to deploy a vSphere cluster backed by FlashArray by stitching multiple sub tasks such as server profile association to ucs server, creating vSphere cluster, get datastore details and installing hypervisor on each node and so on. For more information about Everpure integration with Intersight, see Cisco Intersight FlashArray Connector Overview.

Solution Design

This chapter contains the following:

●     FlashStack Design Considerations

●     Physical Topology

●     FlashStack Cabling

●     VMware vSphere ESXi Configuration

●     ESXi Networking

●     Everpure FlashArray - Storage Design

FlashStack Design Considerations

FlashStack with Cisco UCS and Cisco Intersight meets the following design requirements:

●     Resilient design across all the layers of infrastructure with no single point of failure

●     Scalable design with the flexibility to add compute capacity, storage, or network bandwidth as needed

●     Modular design that can be replicated to expand and grow as the needs of the business grow

●     Flexible design that can support different models of various components with ease

●     Simplified design with the ability to integrate and automate with external automation tools

●     AI-Ready design to support required NVIDIA GPUs for running AI/ML based workloads

●     Cloud-enabled design which can be configured, managed, and orchestrated from the cloud using GUI or APIs

●     Unified full-stack visibility for real-time monitoring, faster troubleshooting, and improved digital resilience by correlating metrics, logs, and traces across infrastructure and applications

To deliver a solution that meets all these design requirements, various solution components are connected and configured as explained in the following sections.

Physical Topology

FlashStack with 6^th Generation UCS fabric technology, Cisco UCS X-Series and Cisco UCS C-Series supports both IP-based and Fibre Channel (FC) based storage access designs. For the IP-based designs, Cisco Nexus switches provide the necessary Ethernet fabrics for all types of traffic, including management and storage access. iSCSI, NFS and NVMe-TCP protocols have been validated for IP-based architecture. In an FC-based design, Cisco MDS switches provide the necessary Fibre Channel fabric for storage access, while Nexus switches provide Ethernet fabrics for the remaining traffic. Fibre Channel, NVMe over Fibre Channel (NVMe-FC) protocols have been validated for FC-based design. Everpure FlashArray can be connected to both Ethernet and Fibre Channel switches and provides required shared storage to the vSphere cluster. ESXi hosts are configured to boot from FlashArray using iSCSI and FC protocols.

IP-based Storage Access

The physical topology for IP-based FlashStack design is shown in Figure 2. This reference architecture is used to validate iSCSI, NFS and NVMe-TCP storage protocols.

To validate the IP-based storage access in a FlashStack configuration, the components are set up as follows:

●     A pair of Cisco Nexus 93600CD-GX switches are configured and used to provide the switching fabric.

●     A pair of 6^th Generation Cisco UCS 6664 Fabric Interconnects (FIs) provide the chassis connectivity. At least two 100 Gigabit Ethernet ports from each FI, configured as a Port-Channel, are connected one to each Nexus 93600CD-GX switch.

●     Cisco UCS X9508 Chassis contains a couple of Cisco UCS X215C M8 servers and one Cisco UCS X580p PCIe Node with two NVIDIA RTX Pro 6000 GPUs. Other X-Series and C-Series servers of different sizes with and without GPUs are also supported. The Cisco UCS X9508 Chassis connected to Fabric Interconnects using Cisco UCS 9108-100G Intelligent Fabric Modules (IFMs).

●     A couple of Cisco UCS C245 M8 Rack Server with one NVIDIA L40S GPU. Other configurations of servers with and without GPUs are also supported

●     The Everpure FlashArray//XL170 R5 connects to the Cisco Nexus 93600CD-GX switches using four 200-GE ports.

●     On each Cisco UCS compute node, VMware vSphere ESXi 9.1 is installed on SAN boot volume created on FlashArray and connected using FC/iSCSI protocol.

Fibre Channel-based Topology

The physical topology for FC-based FlashStack design is shown in Figure 3. This reference architecture is used to validate FC ad NVME-FC storage protocols.

To validate the FC-based storage access in a FlashStack configuration, the components are set up as follows:

●     A pair of Cisco MDS 9124V switches are used to provide 64G Fibre Channel storage switching fabric.

●     A pair of Cisco Nexus 93600CD-GX switches are configured to provide ethernet switching fabric.

●     A pair of 6^th Generation Cisco UCS 6664 Fabric Interconnects (FIs) provide the chassis connectivity. At least two 64 Gigabit FC ports from each FI configured as a Port-Channel are connected to the corresponding fabric’s MDS 9124V. 16G and 32G FC connectivity is also supported. At least two 100 Gigabit Ethernet ports from each FI, configured as a Virtual Port-Channel, are connected one to each Nexus 93600CD-GX switch.

●     Cisco UCS X9508 Chassis contains a couple of Cisco UCS X215C M8 Servers and one Cisco UCS X580p PCIe Node with two NVIDIA RTX Pro 6000 GPUs. Other configurations of servers with and without GPUs are also supported. The Cisco UCS X9508 Chassis connected to Fabric Interconnects using Cisco UCS 9108-100G Intelligent Fabric Modules (IFMs).

●     Couple of Cisco UCS C245 M8 Rack Server with one NVIDIA L40S GPU. Other configurations of servers with and without GPUs are also supported.

●     The Everpure FlashArray//XL170 R5 connects to Cisco MDS 9124V switches using four 64G FC ports from each controller. On each controller, two FC ports are used for traditional Fibre Channel connectivity while other two ports are configured for NVMe over Fabre Channel Fabric (NVMe-FC).

●     On each Cisco UCS compute node, VMware vSphere ESXi 9.1 is installed on SAN boot volume created on FlashArray and connected using FC protocol.

FlashStack Cabling

The information in this section is provided as a reference for cabling the physical equipment in a FlashStack environment.

Compute Infrastructure Design

The compute infrastructure in FlashStack solution consists of the following:

●     Cisco UCS X-Series Chassis with Cisco UCSX-9108 Intelligent Fabric Modules and Cisco UCS X215c M8 blades

●     Cisco UCS C245 M8 C-Series Rack servers with Cisco UCS VIC 15237

●     Cisco UCS Fabric Interconnects 6664

Compute System Connectivity

The Cisco UCS X9508 Chassis is equipped with Cisco UCS X9108-100G intelligent fabric modules (IFMs). The Cisco UCS X9508 Chassis connects to each Cisco UCS 6664 FI using at least two 100GE ports as shown in Figure 4. You can start with a minimum of one connection and scale up to eight connections for additional network bandwidth.

Cisco UCS C245 M8 C-Series servers are equipped with Cisco UCS 5^th Generation VIC 15237 dual port 40/100Gbps mLOM network card. Each Cisco UCS C-Series server is connected to Cisco UCS 6664 FIs using two 100GE ports as shown in Figure 5. Cisco UCS C225 M8 server also can be connected in the same fashion.

Compute UCS Fabric Interconnect Connectivity

For both deployment types (FC and IP based), Cisco UCS Fabric Interconnects are connected to upstream Cisco Nexus switches configured in virtual Port Channel (vPC) mode. This setup provides link redundancy, load balancing and high availability for network connectivity between UCS domain and the upstream switches. Each FI is connected to both Cisco Nexus 93600CD-GX switches using 100G connections; additional links can easily be added to the port channel to increase the bandwidth as needed. The 400G ports (from 29 to 36) can be utilized for the peer links for creating the Virtual Port Channel between the two Nexus Switches. Between the FI and MDS Fibre Channel switches, at least two FC ports (from 13 to 20) from each FI must be connected to the MDS switch. Figure 6 shows the Fabric Interconnect connectivity to the Upstream Nexus and MDS switches.

Everpure FlashArray//XL170 R5 Ethernet Connectivity

The Everpure FlashArray//XL170 R5 uses enterprise-grade expansion cards for networking. These are PCIe Gen4 cards operating at 100/200 GbE speed. In this architecture, each FlashArray//XL170 R5 controller is equipped with dual port 200Gbps network card and connected to a pair of Cisco Nexus 93600CD-GX switches for redundancy. The last eight ports (29 to 36) on Nexus 93600CD-GX switch are  QSFP-DD ports and support various speeds such as 10/25/40/50/100/200/400G. QSFP-200G-SR4 transceivers are used for connecting 200Gbps ports on FlashArray//XL170 R5 controllers to 400Gbps QSFP-DD ports on Nexus 93600CD-GX switches as shown in Figure 7.

Everpure FlashArray//XL170R5 Fibre Channel Connectivity

Everpure FlashArray controllers are connected to Cisco MDS 9124V switches using 64Gbps ports as shown in Figure 8.

Figure 9 details the cable connections used in the validation lab for the FlashStack topologies.

On each Fabric Interconnect, E1/3 and E1/4 ports are configured as Server ports for discovering and managing the Cisco UCS X9508 chassis through Cisco UCS 9108 100G IFMs. on each FI, five 100G ports (from 7 to 11) are as Server Ports for connecting and managing Cisco UCS C-Series servers. E1/63 and E1/64 ports on each FI are configured as Ethernet Uplink ports and connected to the pair of Cisco Nexus 93600CD-GX switches that are configured with a vpc domain. On each Fabric Interconnect, port 25 and 26 are connected to MDS switch on ports 1 and 2. Everpure FlashArray//Xl170 connected to the pair of Nexus 93600CD-GX switches over 200G ports. Similarly, each FlashArray controller is connected to pair of MDS switches using 2x 64Gbps FC ports. As mentioned earlier, Additional 1Gbps network (not shown in Figure 9) is required for Out of Band management connectivity.

VMware vSphere ESXi Configuration

In IP-based FlashStack design, ESXi hosts are configured to boot from SAN using iSCSI protocol. Required vNICs are created in the Server profile and a separate boot order policy is used for booting the servers from SAN. In FC-Based design, ESXi nodes are configured to boot from SAN using Fibre Channel protocol. Required vHBAs created in the Server profile and a separate boot order policy is used for booting the servers from SAN. Cisco UCS X580p PCIe Nodes provides PCIe expansion for Cisco UCS X-Series compute nodes. NVIDIA RTXPro 6000GPUs are installed in Cisco UCS X580p nodes and exposed to the Cisco UCS X215c M8 nodes using Intersight PCI Connectivity policy. For the Cisco UCS C-Series servers, GPUs can be installed directly in the PCIe slots.

From a networking perspective, in IP-based  deployment, various storage protocols such as iSCSI, NVMe-TCP and NFS are supported. The ESXi hosts are configured with required vNICs to support the storage protocols. Similarly, in FC-based deployments, ESXi hosts are configured with required vHBAs to support traditional FC and NVMe-FC storage protocols. In addition to these vNICs or vHBAs, additional vNICs are created to support other traffic such as host management, VM management and vMotion. Different VLANS and VSANS are used to segregate these traffics and are described in the following tables.

VLAN and VSAN Configuration

Table 1 lists the VLAN and VSAN IDs configured for setting up the FlashStack environment.

Table 1.        VLANs used in this solution

 

Table 2 lists the infrastructure services running on either virtual machines or bar mental servers required for deployment as outlined in the document. All these services are hosted on pre-existing ESXi infrastructure.

Table 2.        Infrastructure services

Software Revisions

The FlashStack Solution with vSphere 9.1 is built using the following components.

Table 3 lists the required software revisions for various components of the solution.

Table 3.        Software Revisions

ESXi Networking

This solution validates support for a range of storage protocols, including iSCSI, NFS, FC, NVMe-TCP, and NVMe-FC. To facilitate this, ESXi nodes are provisioned with the necessary vNICs and vHBAs, defined via Intersight templates and policies. These components are organized within LAN and SAN connectivity policies to ensure precise enumeration on the server. These policies are then integrated into Server Profile Templates to generate the final Server Profiles, which are subsequently applied to UCS blade and rack servers. The following figures illustrate the vNIC configurations for ESXi hosts across both IP and FC-based architectures.

The below list provide more details on the vNICs and vHBAs used for the solution;

●     00-IBMGMT-A and 01-IBMGMT-B: This pair of vNICs are used for ESXi host management traffic; With in the ESXi hots, a standard vSwitch (vSwitch0) will be created using these two vNICs in Active-Active Failover Order. These vNICs are enumerated as vmnic0 and vmnic1 within the ESXi host.

●     02-vDS-A and 03-vDS-B: : These two vNICs are used for various traffics such as VM Management traffic, vMotion and NFS storage access. With in the ESXi host, a vSphere Distributed switch (vDS0) and multiple distributed Port Groups will be configured for each type of traffic. Based on the traffic type, each port group is configured with either Active-Active or Active-Standby fail over order. Please see ESXi host networking configuration section for more specific details. These vNICs are enumerated as vmnic2 and vmnic3 with in the ESXi host.

●     04-iSCSI-A and 05-iSCSI-B:  This pair of vNICs are used for block storage traffic over iSCSI protocol. With in the ESXi hots, standard vSwitches (iScsiBootvSwitch-A and iScsiBootvSwitch-B) will be created for each Fabric using the corresponding vNIC. These vNICs are also used for booting the ESXi host from SAN boot volumes using the iSCSI protocol. These vNICs are enumerated as vmnic4 and vmnic5 within the ESXi host.

●     06-NVMe-TCP-A and 07-NVMe-TC-B: This pair of vNICs are used for block storage traffic using NVMe over TCP protocol. With in the ESXi hots, standard vSwitches (NVMe-TCP-A and NVMe-TCP-B) will be created for each Fabric using the corresponding vNIC. These vNICs are enumerated as vmnic6 and vmnic7 with in the ESXi host.

●     04-vHBA-A and 05-vHBA-B: This pair of vHBAs are used for block storage traffic over Fibre Channel (FC) protocol. MDS switches need to be configured with required zoning for the ESXi HBAs to discover the correct FlashArray target volumes; These vHBAs are also used for booting the ESXi host from SAN boot volumes using the FC protocol. These vHBAs are enumerated as vmhba0 and vmhba1 within the ESXi host.

●     06-vHBA-NVMe-A and 07-vHBA-NVMe-B: This pair of vHBAs are used for block storage traffic using NVMe over FC protocol. MDS switches are needs to be configured with required zoning for the ESXi HBAs to discover the correct FlashArray target volumes. These vHBAs are enumerated as vmhba2 and vmhba3 with in the ESXi host.

Figure 12 illustrates the logical networking diagram of esxi host for both IP-based and FC-Based architectures.

Everpure FlashArray - Storage Design

Once the required VMKernel or vmhbas are provisioned and configured within the ESXi hosts, the following items must be configured for FlashArray storage connectivity.

●     Gather IQN/WWPN/NQN: Once the server profiles are derived, you must gather the following logical constructs.:

◦     Gather ISCSI Qualified Names (IQN) from each server profile configured to access to the storage using iSCSI protocol; Click the Server profile, under General tab, click the Identifiers tab and capture the IQN.

◦     Gather NVMe Qualified Name (NQN) from the ESXi host that are  going to access FlashArray using NVMe over Fabric protocols such as NVMe over TCP or NVMe over Fiber Channel.

◦     Gather Worldwide Port Name (WWPN) from each server profile configured to access to the storage using FC or NVMe over FC protocols; Click the Server profile, under General tab, click the vNIC/vHBAs tab and capture the WWPN for both the Fabrics (A and B).

●     Volumes

◦     ESXi boot LUNs: These LUNs enable ESXi host boot from SAN functionality using Fibre Channel or iSCSI protocols.

◦     vSphere Datastores: vSphere uses the datastore(s) to store the virtual machines. These volumes can be exposed to the ESXi hosts using traditional protocols like iSCSI and Fibre Channel. Latest high-speed protocols such as NVMe over FC and NVMe over TCP can also be used for provisioning high-performance datastore in vSphere environments. Additionally, FlashArray fully supports NFS datastores, providing VM-aware unified block and file services natively. This allows you to easily provision and mount NFS file systems directly to your ESXi hosts without the traditional complexity of carving out pools or flexible volumes beneath the file layer.

●     Hosts and Host Groups

◦     All FlashArray ESXi hosts are defined using the FC WWNs (scsi-fc based initiators) or IQNs (iSCSI initiator) or NQN (NVMe initiators).

◦     Add every active initiator for a given ESXi host.

◦     All ESXi hosts in a VMware cluster are part of the host group.

◦     Host groups are used to mount VM infrastructure application storage datastores in the VMware environment so that all ESXi hosts that are part of Host Group will be able to access storage volumes.

The following figures illustrate the simplified logical connectivity between the ESXi hosts and FlashArray//XL170 storage array for both the architectures.

Network Switch Configuration

This chapter contains the following:

●     Prerequisites

●     Cisco Nexus Switch Manual Configuration

Prerequisites

The following are the prerequisites for network switch configurations:

●     Physical cabling should be completed by following the diagram and table references in section FlashStack Cabling. The following procedures assume that all the FlashStack components are connected to a dedicated Out-Of-Band/IPMI network for KVM session access.

●     The procedures in this chapter describe how to configure the Cisco Nexus 93600CD-GX switches for use in a FlashStack environment. This procedure assumes Fabric Interconnects firmware and Cisco Nexus switches NXOS is upgraded to the supported versions.

●     The procedure includes the setup of NTP distribution on both the mgmt0 port and the in-band management VLAN. The interface-vlan feature and ntp commands are used to set this up.

●     This document assumes that initial day-0 switch configuration is already done using switch console ports and ready to use the switches using their management IPs.

●     This document assumes that initial day-0 Everpure FlashArray configuration is already done using console ports and ready to use the FlashArray Management Console using their virtual management IP.

●     It is recommended to list and note all VLANs, and IP addresses for vSphere cluster provisioning. Table 4 lists the supporting components or modules used for deploying the solution. These supporting components deployed outside the FlashStack environment.

Table 4.        Supporting Components used for vSphere Cluster manual Deployment

DHCP Configuration

For this validation, DHCP services are used for assigning the management addresses automatically to the ESXi hosts. Create a DHCP scope for management VLANs with appropriate subnet.

Table 5 lists the information for fully qualified domain names, in-band and out-of-band management IPs used during the validation.

Table 5.        DNS FQDN Names and IP Addresses

Cisco Nexus Switch Manual Configuration

This section contains the procedure to configure the two Nexus switches manually.

Procedure 1.    Enable Features on Cisco Nexus A and Nexus B

Step 1.          Run the following commands on both the switches.

Step 2.          Enable the switch features as described below:

Procedure 2.    Set Global Configurations on Cisco Nexus A and B

Step 1.          Run the following commands on both the switches.

Step 2.          Run the following commands to set the global configurations:

Procedure 3.    Create VLANs on Cisco Nexus Switches A and B

Step 1.          From the global configuration mode, run the following:

Procedure 4.    Define Port Channel on Cisco Nexus A and B

Step 1.          From the global configuration mode, run the following:

Procedure 5.    Configure Virtual Port Channel (VPC) Domain on Cisco Nexus Switches A and B

Step 1.          Run the following commands to set the global configurations:

Procedure 6.    Configure Individual Interfaces on Nexus Switch A and B

Step 1.          From the global configuration mode, run the following to configure the individual ports:

Procedure 7.    Update the port channels

Step 1.          From the global configuration mode, run the following:

Procedure 8.    Claim the Nexus Switches into Intersight

Step 1.          Log into Cisco Intersight with your login credentials. From the drop-down list select System.

Step 2.          Go to System > Target then click Claim a New Target. Under Categories, select Network, click Cisco Nexus Switch and then click Start.

Step 3.          Click Claim Target with Cisco Assist and select the Cisco Assist name which is already deployed and configured. Provide the Cisco Nexus Switch management IP address, username and password details and click Claim.

Step 4.          Repeat steps 1 through 3 to claim the remaining Nexus Switch B.

Step 5.          After the switches are successfully claimed, Go to Operate > Networking to get complete inventory of the switches.

Procedure 9.    Claim Network Switches using the Claim Target method without using Intersight Assist

Step 1.          From the global configuration mode, run the following to claim the Nexus switches into Cisco Intersight:

Step 2.          Claim the Nexus Switches into Intersight; Go to System > Targets . Click Claim a New Target. Select Network and then the Cisco Nexus Switch. Click Start. Claim the switch using the information collected from the switch itself.

Everpure FlashArray Configuration

This chapter contains the following:

●     iSCSI and NVMe-TCP Configuration

●     FC and NVMe-FC Configuration

●     NFS Configuration

iSCSI and NVME-TCP Configuration

In this solution, Everpure FlashArray//XL170 R5 provides storage for all the workloads running on the vSphere Compute Clusters. This chapter describes the high-level steps to configure FlashArray//X170 network interfaces required for storage connectivity using iSCSI and NVMe over TCP. While these procedures specifically detail configuration steps for FlashArray //XL170, the configuration steps are substantially similar for other supported models, including the FlashArray //X and //C series.

As detailed in the previous chapter, each storage controller of FlashArray is connected to the pair of Nexus switches using 2x 200GpE ports offering aggregated network bandwidth of 800Gbps from the two controllers.

Everpure FlashArray network settings were configured with three subnets across three VLANs. Storage Interfaces CT0.Eth0 and CT1.Eth0 were configured to access management for the storage on VLAN 1030. Storage Interfaces (CT0.Eth10, CT0.Eth11, CT1.Eth10, and CT1.Eth11) were configured to run iSCSI or NVMe-TCP Storage network traffic on the VLAN 3010 and VLAN 3020. Later, the same interfaces are used for NFS traffic using VLAN 3060.

Table 6 lists the IP addresses used for the interfaces for iSCSI or NVMe-TCP protocols.

Table 6.        FlashArray//XL170 Interface Configuration for iSCSI and NVMe-TCP protocols

Procedure 1.    Configure Storage Interfaces for iSCSI and NVMe-TCP

Step 1.          SSH in to the FlashArrray and run the following commands based on choice of protocol:

FC and NVMe-FC Configuration

In this architecture, each storage controller of FlashArray is connected to the pair of Cisco MDS switches using 4x 64Gbps ports offering aggregated fabric bandwidth of 512Gbps from the two controllers. The WWNs of each fibre channel port on the storage controllers are preconfigured. It is required to gather WWNs of each fibre channel port which will be later used for creating Intersight boot policy and creation of Zones and Device aliases in MDS switches.

Table 7 lists the WWNs of fibre channel ports of each controller gathered from the FlashArray controllers.

Table 7.        WWNs of Fibre Channel Ports

Procedure 1.    Configure Storage Interfaces for FC and NVMe-FC

Step 1.          SSH in to the FlashArrray and run the following commands based on choice of protocol.

NFS Configuration

While NFS file services can be implemented in many methods on FlashArray, this section covers two simplified  and highly available NFS implementations only. For more details, go to: https://support.everpuredata.com/r/user-guides-for-vmware-solutions/creating-a-new-file-server-using-the-file-server-wizard

Network Configuration with Single VLAN

In this method, a single Subnet with single VLAN is created and all the storage interfaces will be members of the same subnet. One or more Virtual Interface (VIF) can be created and associated with the NFS File service. Follow the below steps for configuring FlashArray networking for NFS file services with single VLAN.

Procedure 1.    Configure NFS networking on FlashArray with Single VLAN

Step 1.          Log into the FlashArray Management Portal using the management IP address of the Flasharray.

Step 2.          Ensure the IP Addresses from ethernet interfaces are removed and set service type to iscsi for the storage interfaces.

Step 3.          Create a subnet; go to Settings > Network > Configuration and click +. Provide a Name and click Enabled, provide Prefix, VLAN and MTU. Click Create.

Step 4.          Add interfaces on the newly created subnet; click +. From the drop-down list, select ct0.eth10.3060 and click Save. Create one interface for each FlashArray port (CT0.eth10, CT0.eth11, CT1.eth10, CT1.eth11).

Step 5.          Create Virtual Interface (VIF); click Storage > Servers > Configuration Assistant. Provide a name to the VIF, from the drop-down list select the Subnet, provide an IP address and click Next.

Step 6.          Select management for DNS config and click Next. Select None for Directory Services, review the summary and click Create. When the VIF is created successfully, click Done.

Network Configuration with LACP using Single VLAN

In this method, LACP Bond is configured on two ethernet ports on each storage controller. And then create a Subnet and add interfaces on each port-channel created above. Finally, one or more Virtual Interface (VIF) can be created and associated with the NFS File service. Follow the below steps for configuring FlashArray networking for NFS file services with LACP  using single VLAN.

Procedure 1.    Configure NFS networking on FlashArray with LACP using Single VLAN

Step 1.          Log into the FlashArray management portal using the management IP address of the FlashArray.

Step 2.          Ensure the IP addresses from Ethernet interfaces are removed and set service type to iscsi for the storage interfaces.

Step 3.          Create a LACP on storage Interfaces; go to Settings > Network > Configuration > Interfaces. Click the ellipses and select Create LACP Bond. Provide a name to the LACP Bond and select the two interfaces of first controller and then click Add.

Step 4.          Repeat steps 1 - 3 to create LACP Bond on the second controller ports.

Step 5.          Create a Subnet; go to Settings > Network > Configuration and click +. Provide a Name and click Enabled, provide a Prefix, VLAN and MTU. Click Create.

Step 6.          Create an interface for each LACP Bond created in the previous step. On the newly created subnet, click +, select lacp1.3060 from the drop-down list and click Save. Create a second interface for the other LACP Bond.

Step 7.          Create Virtual Interface (VIF) by following Procedure 1. Configuring NFS networking on FlashArray with Single VLAN.

The following screenshot shows the details of the LACP bonds and virtual interfaces (VIF)s configured for this deployment:

Procedure 2.    Claim Everpure FlashArray into Intersight

Step 1.          Log into Cisco Intersight using your login credentials. From the drop-down list select System.

Step 2.          Under Admin, select Target and click Claim a New Target. Under Categories, select Storage, click Everpure FlashArray and then click Start.

Step 3.          Select the Cisco Assist name which is already deployed and configured. Provide the Everpure FlashArray management IP address, username, and password details and click Claim.

Step 4.          When the storage is successfully claimed, from the drop-down list, select Infrastructure Services. Under Operate, click Storage. You will see the newly claimed Everpure FlashArray; browse through it to view the inventory details.

Configure Intersight for Cisco UCS

The chapter contains the following:

●     Fabric Interconnect Domain Profile and Policies

●     Pools and Policies for Templates

●     vNIC Templates, vNICs, and LAN Connectivity Policy

●     vHBA Templates, vHBAs, and SAN Connectivity Policy

●     Compute Configuration Policies

●     Management Configuration Policies

●     Server Profile Templates and Server Profiles

●     PCIe Connectivity Policy

The procedures in this chapter describe how to configure a Cisco UCS domain for use in a base FlashStack environment. A Cisco UCS domain is defined as a pair for Cisco UCS FIs and all the Cisco UCS X-Series and Cisco UCS C-Series servers connected to it. These can be managed using two methods: UCSM and IMM. The procedures detailed below are for Cisco UCS Fabric Interconnects running in Intersight managed mode (IMM).

Fabric Interconnect Domain Profile and Polices

This section contains the procedures to claim the FIs to Intersight account, create fabric interconnect domain profiles.

Procedure 1.    Claim Fabric Interconnect into Intersight

Step 1.          Log into your Intersight account with your credentials. Go to System > Targets and click Claim a New Target option.

Step 2.          Select Cisco UCS Domain (Intersight Managed) option and click Start.

Step 3.          Retrieve the Device ID and Claim Code for the Fabric Interconnect from its web console and enter the details and click Claim. The FIs will be claimed into your Default Resource Group of your Intersight account. The Custom Resource Groups (RGs) and Organizations can be created later and add the FIs to the custom Resource Groups.

Procedure 2.    Upgrade UCS Domain and Server Firmware

Step 1.          Upgrade the firmware of the UCS Domain and servers to the latest version in Cisco Intersight ahead of the vSphere cluster deployment is recommended. See the Cisco UCS Compatibility for supported firmware for each server type. For this validation, all the Cisco UCS FIs are upgraded to 6.0(2.260067), and Cisco UCS X-Series and Cisco UCS C-Series servers are upgraded to the latest firmware 6.0(2.260143).

Procedure 3.    Create Organization and Resource Groups

It is recommended to create dedicated Organizations and custom Resource Groups for managing vSphere Cluster nodes. This approach simplifies management and enhances the security of server access. Follow these steps to create new Organization and Resource Group and add FI to the newly created Resource Group.

Step 1.          Log into the Intersight portal, select System > Resource Groups > click Create Resource group. Provide the name as vvf-rg and set resources as Custom. Select the 6664 FI and click the Pen symbol. Select all the Cisco UCS X-Series and Cisco UCS C-Series servers that are going to be part of the vSphere cluster. Along with the FIs, select all the remaining hardware components like Nexus switches, MDS switches, FlashArray and Intersight Assist and so on that are going to be part of your vSphere Cluster as shown below. The following screenshot shows the servers along with other hardware components like Switches, Everpure and Intersight Assist. After selecting the required components, click Create.

Step 2.          Go to System > Organizations and click Create Organization. Provide a name as vvf and click Next. Select the vvf-rg created in the above step and click Next. Review the summary and click Create.

Procedure 4.    Fabric Interconnect Domain Profile and Policies

This section contains the procedures to create fabric interconnect domain profiles and policies.

Step 1.          Log into the Intersight portal, select Configure > Profiles then select UCS Domain Profiles > Create Domain Profile.

Step 2.          Set the Organization to default and provide a name to the FI domain profile (AA06FI6664-UCSDomain/) and click Next. Click Assign Later to assign this domain profile to a FI later. Click Next.

Step 3.          Click Next to go to VLAN & VSAN Configuration.

Step 4.          Under VLAN & VSAN Configuration > VLAN Configuration, click Select Policy then click Create New.

Step 5.          On the Create VLAN page, enter a name (AA06-FI6664-VLANs) and click Next. To add a VLAN, click Add VLANs.

Step 6.          For the Prefix, enter the VLAN name as OOB-Mgmt-VLAN. For the VLAN ID, enter the VLAN ID 1060. Leave Auto Allow on Uplinks enabled and Enable VLAN Sharing disabled.

Step 7.          Under Multicast Policy, click Select Policy and select Create New to create a Multicast policy.

Step 8.          On the Create Multicast Policy page, enter the name (AA06-FI-MultiCast) of the policy and click Next to go to Policy Details. Leave the Snooping State and Source IP Proxy state checked/enabled and click Create. Select the newly created Multicast policy.

Step 9.          Repeat steps 6 through 8 to add all the required VLANs to the VLAN policy.

Step 10.       After adding all the VLANs, click Set Native VLAN ID and enter the native VLANs (for example 2) and click Create. The VLANs used for this solution are shown below:

Step 11.       Select the newly created VLAN policy for both Fabric Interconnects A and B.

Step 12.       Click Select Policy to create a VSAN policy for Fabric-A. Provide a name to the VSAN policy (AA06FI6664-VSAN-A-103). Click Next. Enable the Uplink Trunking option. Click Add VSAN. Provide name (MDSUplink103), select Uplink for VSAN Scope, set VSAN and FCoE VLAN IDs to 103. Click Save. Click Create to complete  the creation of the VSAN policy for Fabric-A.

Step 13.       Repeat step 12 to create another VSAN policy (AA06FI6664-B-VSAN-104) for Fabric-B with VSAN and FCoE IDs as 104. The final VLAN and VSAN configurations is shown below:

Step 14.       Click Next to go to the Port configuration.

Step 15.       Create a new Ports Configuration Policy for each Fabric Interconnect. Provide a name (AA06FI6664-A-PortConf) and select the UCS-FI-6664 from Fabric Interconnect model drop-down list. Click Next and go to Unified Ports.

Step 16.       For the Unified Port option, ensure the ports 25 to 28 have been selected and configured as unified ports by adjusting the slide. Click Next and go to Port Roles configuration.

Step 17.       Select the appropriate ports, where the X-series blade chassis and C-series servers are connected, Click Configure to set them as server ports by setting Role to Server.

Step 18.       Click the Port Channels tab and click Create Port Channel. Select Port 63 and 64 for FI6664, set Role to Ethernet Uplink Port Channel, set 100 for Port Channel ID and admin speed to 100Gbps and FEC to Cl91. Under Link Control, create a new link control policy with settings described in the following table.

Step 19.       Click Create Port Channel to create Port Channel for MDS connectivity. Select Ports 25 and 26 and set Role to FC Uplink Port Channel. Set Admin speed to 64Gbps and enter VSAN ID to 103. Click Save to complete Port Policy Configuration.

Step 20.       Repeat steps 15 through 19 to create another port configuration policy (AA06FI6664-B-PortConf) for Fabric Interconnect B. Use 200 for Ethernet Uplink Port Channel ID and 104 for FC Port Channel and VSAN ID.

Step 21.       For Link Control policy of Ethernet Uplink configuration, create a new link control policy with the options in Table 8. Once created, select the policy.

Table 8.        Link Control Policy

Step 22.       When two Port configurations are selected for FI-A and B, click Next to go to the UCS Domain Configuration page. The following tables list the management and network related polices created and used for this validation. Create NTP, Network Connectivity, SNMP and QoS policies as described below and complete the UCS Domain Profile creation for 6664 Fabric Interconnects used in this validation.

Table 9.        NTP policy

Table 10.      Network Connectivity Policy

Table 11.      SNMP Policy

Table 12.      QoS Policy

Step 23.       When the  of UCS Domain profile is completed using the policies explained above, edit the policy and assign it to the Fabric Interconnects 6664. Intersight will go through the discovery process and discover all the Cisco UCS C-Series and Cisco UCS X-Series compute nodes attached to the Fabric Interconnects.

Pools and Policies for Templates

In the Cisco Intersight platform, a server profile enables resource management by simplifying policy alignment and server configuration. The server profiles are derived from a server profile template. A Server profile template and its associated policies can be created using the server profile template wizard. After creating the server profile template, you can derive multiple consistent server profiles from the template.

The server profile templates captured in this deployment guide are intended to use for Cisco UCS X-Series and Cisco UCS C-Series AMD M8 compute nodes with 5th Generation VICs and can be modified to support other Cisco UCS blades and rack mount servers.

Pools

The following pools need to be created with in the vvf organization before proceeding with server profile template creation. Ensure you set the Organization name to vvf for each of the pool and policy you create.

MAC Pools

Table 13 lists the two MAC pools for the vNICs that will be configured in the templates.

Table 13.      MAC Pool Names and Address Ranges

UUID Pool

Table 14 lists the settings for the UUID pools.

Table 14.      UUID Pool Names and Settings

Out of Band Management IP Pool

Create the ipv-4 OOB management IP pool (AA06-VVF-OOBIPPool) with the settings listed in Table 15.

Table 15.      IPV4 Pool for Out of Band Management

IQN Pool

Create the IQN pool (AA06-VVF-IQNPool) for iSCSI storage protocol with the settings listed in Table 16.

Table 16.      IQN Pool

IP Pools for iSCSI via Fabric-A & B

Create the ipv-4 OOB management IP pool (AA06-VVF-iSCSI-A-IPPool, AA06-VVF-iSCSI-B-IPPool) with the settings listed in Table 17.

Table 17.      IP Pool for iSCSI

WWNN Pool

Create the World Wide Node Name pool (AA06-VVF-WWNN) for FC storage protocol with the settings listed in Table 18.

Table 18.      WWNN Pool

WWPN Pools

Create the World Wide Node Name pool (AA06-VVF-WWPN-A, AA06-VVF-WWPN-B) for FC storage protocol with the settings listed in Table 19.

Table 19.      WWPN Pools

vNIC Templates, vNICs, and LAN Connectivity Policy

The following vNIC templates are used for deriving the vNICs for ESXi nodes for host management, VM management, VM Live migration  and storage traffics.

Table 20.      vNIC Templates used for FlashStack networking

Ethernet Adapter Policy for Storage Traffic

The ethernet adapter policy is used to set the interrupts, send and receive queues, and queue ring size. The values are set according to the best-practices guidance for the operating system in use. Cisco Intersight provides a default Linux Ethernet Adapter policy for typical Linux deployments.

Optionally, you can configure a tweaked ethernet adapter policy for additional hardware receive queues handled by multiple CPUs in scenarios where there is a lot of traffic and multiple flows. In this deployment, a modified ethernet adapter policy, AA06-EthAdapter-16RXQs-5G, is created and attached to storage vNICs templates AA06-VVF-iSCSI-A, AA06-VVF-iSCSI-B, AA06-VVF-NVMe-TCP-A, AA06-VVF-NVMe-TCP-B. Other vNICs  templates are configured to use the default VMware-v2 Ethernet Adapter policy. Table 21 lists the settings that are changed from defaults in the Adapter policy used for the iSCSI traffic. The remaining settings are left at defaults

Table 21.      Ethernet Adapter Policy used for Storage traffic

iSCSI Boot Policies

An iSCSI boot policy in Cisco Intersight defines how a server initializes and boots its operating system from a remote iSCSI storage target. In IP-Based architecture, the ESXi hosts are configured to boot from FlashArray iscsi target. Create required iSCSI boot policies using the information provided in the following tables for the two vNIC templates AA06-VVF-iSCSI-A and AA06-VVF-iSCSI-B.

Table 22.      iSCSI Boot Policies

The following screenshot settings used for the policy AA06-VVF-iSCSIBoot-A:

Using the templates listed in Table 21, a LAN connectivity policy is created for IP-based ESXi hosts.

For FC-based FlashStack deployment, the vNICs (04-iSCSI-A, 05-iSCSI-B,06-NVMe-TCP-A, 07-NVMe-TCP-B) that carry storage traffic over ethernet are not required to be configured. Hence, the LAN connectivity policy for FC-Based FlashStack design, would have only four vNICs (derived from templates: AA06-VVF-IBMgmt-A, AA06-VVF-IBMgmt-B, AA06-vDS-A and AA06-vDS-B) to carry management and other traffics. The below screenshot shows the LAN connectivity policy (AA06-VVF-LANConn_FCBoot) used for FC-based FlashStack deployment. For the FC-Based ESXi hosts, the following LAN connectivity policy has been created using the first four vNIC templates detailed in Table 21.

vHBA Templates, vHBAs, and SAN Connectivity Policy

For FC-Based FlashStack design, Fibre Channel and NVMe over FC protocols are used for accessing the Storage through Cisco MDS Fibre Channel switches. Compute nodes are configured with required virtual vHBAs (scsi-fc and nvme-fc) for accessing the FlashArray over Fibre Channel network. The ESXi nodes in FC-based architecture are configured to boot from FlashArray volumes over FC.

Table 23 provides vHBA templates used for deriving the required vHBAs for the FC-based FlashStack design.

Table 23.      vHBA templates used for FC-Based FlashStack design

Using these vHBA templates, the following vHBAs are derived by creating the SAN Connectivity Policy (AA06-VVF-SANConn).

Compute Configuration Policies

This section discusses some of the important compute polices used for creating the Server profile templates.

Boot Order Policies for iSCSI and FC boot

As discussed, in the IP-Based architecture, compute nodes are configured to boot from FlashArray iSCSI volumes using iSCSI protocol while in FC-Based architecture, ESXi hosts are configured to boot from FC volumes using traditional FC (scsi-fc) protocol. Each of these architectures uses different Boot policies as shown below.

Create a Boot policy for FC targets by adding “SAN Boot” boot device. For each SAN Boot device, select the appropriate vHBA interface name and FlashArray Target WWPN combination based on how FlashArray FC ports are connected to the MDS switches.

Table 24.      Boot Order policy for SAN boot using iSCSI protocol

Create a Boot policy for iSCSI targets by adding “iSCSI Boot” boot device. For each iSCSI Boot device, select the appropriate iSCSI interface name and FlashArray iSCSI Target IP combination based on how FlashArray iSCSI ports are connected to the Nexus switches.

Table 25.      Boot Order policy for iSCSI Boot using iSCSI protocol

Virtual Media (vMedia) Policy

Virtual Media policy is used to mount the ESXi 9.1 Cisco custom ISO to the server using CIMC Mapped DVD policy. Download the ESXi 9.1 Cisco Custom ISO from Broadcom website and share it using a https file server. Enter URL of the ISO under File Location in the vMedia policy.

BIOS Policy

This section contains the procedures to create a BIOS policy for AMD M8 compute nodes.

Procedure 1.    BIOS policy for AMD M8

Step 1.          Log into Intersight and go to Configure > Polices and click Create Policy. Click UCS Server and select BIOS and click Start.

Step 2.          Enter a name ( AA06-VVF-ESXi-AMDBIOS )to BIOS policy and click Use Cisco Predefined Configuration and then set Server Generation to M8, CPU Type to AMD, Workload Type to Virtualization and click Next.

Step 3.          In addition to the default BIOS settings for AMD M8 virtualization, set the following BIOS tokens when you would like to leverage AMD SEV-SPN features for confidential Compute VMs on vSphere environments.

Table 26.      BIOS token for AMD SEV-SPN

Step 4.          Ensure the following BIOS tokens are enabled for detecting NVIDIA GPUs in OS/Hypervisor.

Table 27.      BIOS token for NVIDIA GPUs

Firmware Policy

The following image shows the firmware versions used for each type of servers in this validation:

Management Configuration Policies

The following policies will be added to the management configuration:

●     IMC Access to define the pool of IP addresses for compute node KVM access

●     IPMI and Local User to provide local administrator to access KVM

●     Virtual KVM to allow the Tunneled KVM

Cisco IMC Access Policy

This section details the procedure to create a CIMC access policy.

Procedure 1.    Create a CIMC Access Policy

Step 1.          Since certain features are not yet enabled for Out-of-Band Configuration (accessed using the Fabric Interconnect mgmt0 ports), you need to access the OOB-MGMT VLAN (1060) through the Fabric Interconnect Uplinks and mapping it as the In-Band Configuration VLAN.

Procedure 2.    IPMI and Local User Policy

The IPMI Over LAN Policy can be used to allow both IPMI and Redfish connectivity to Cisco UCS Servers. vCenter uses these two policies to power manage (power off, restart, and so on) the bare metal servers.

Step 1.          Create IPMI over LAN policy and Local User policies as shown below.

Virtual KVM Policy

The following image shows the virtual KVM policy used in the solution:

Server Profile Templates and Server Profiles

Create the Server Profile templates using pools, polices, vNIC and vHBA templates created in the previous sections. One separate Server Profile template is created for each type of architecture.

Table 28 lists the policies and pools used to create the Server Profile template AA06-VVF-iSCSIBoot-ESX for IP-Based architecture and AA06-VVF-FCBoot-ESX for FC-based FlashStack architecture.

Table 28.      Policies and Pools used for IP and FC-Based FlashStack Designs

The following images show the server profile templates created for the IP and FC based designs:

Once the Server Profile Templates are created, the server profiles can be derived from each template. The following screenshot shows a total of seven server profiles derived (three for FC-Based design and five for IP-based design).

When the Server profiles are created, associate these server profiles to the UCS compute nodes. After attaching the server profiles successfully, gather following information.

To gather the following information, click the Server Profile, go to General > Identifiers > vNICs/vHBAs tab. These details are required for booting for SAN volume and zoning configuration on MDS switches:

●     From each profile gather the MAC addresses of first two vNIC interfaces (00-IBMgmt-A and 01-IBMgmt-B)

●     From each FC-Based server profile gather the WWPN addresses of all the vHBAs (00-vHBA-A, 01-vHBA-B, 02-vHBA-NVMe-A, and 03-vHBA-NVMe-B)

●     From each IP-Based server profile gather the IQN name of the host initiator

Table 29.      Identifiers of Server Profiles

PCIe Connectivity Policy

The Cisco Intersight PCIe Connectivity Policy defines how PCIe devices and expansion nodes are configured and mapped to specific CPUs within a server. This applies specifically to Cisco UCS X-Series servers paired with the Cisco UCS X580p PCIe Nodes and X9516 X-Fabric Modules. The following PCIe policy was created and used to map one NVIDIA RTP Pro 6000 GPU from the X580p PCI node:

Cisco MDS MDS9124V Switch Manual Configuration

This chapter contains the following:

●     Configure Cisco MDS MDS9124V Switches

●     Create Boot and Storage Volumes for VVF Deployment

This chapter provides the procedures to configure MDS switches with the required zoning configuration for the compute nodes to be able to access the FlashArray volumes over FC and NVMe-FC protocols.

Configure Cisco MDS MDS9124V Switches

This section provides the procedures to manually configure the Cisco MDS MDS9124V switches.

Procedure 1.    Enable features on Cisco MDS A and Cisco MDS B

Step 1.          SSH into both MDS switches as admin and enable the switch features as described below:

Procedure 2.    Configure Individual Ports on each Switch

Step 1.          Log into MDS Switch-A and B as admin using ssh.

Step 2.          Run the following commands to configure the individual ports and port channels, and so on:

Procedure 3.    Create VSAN & Device Aliases on each Switch

Step 1.          Log into MDS Switch-A and B as admin using ssh.

Step 2.          Run the following commands to configure the VSAN and device aliases:

Procedure 4.    Create Zones and Zonesets

Step 1.          Log into MDS Switch-A and B as admin using ssh.

Step 2.          Run the following commands to configure the individual zones and zoneset and so on:

Create Boot and Storage Volumes for VVF Deployment

When the required vHBAs configuration and zone configuration is completed on the hosts and switches, boot volumes can be created and mapped to the corresponding hosts in the FlashArray. The procedures in this section detail how to create Hosts, Boot volumes and attach Boot volumes to the Hosts with in the FlashArray//XL170.

Procedure 1.    Create Hosts with WWPN

Step 1.          Log into Everpure FlashArray with admin credentials.

Step 2.          Click the Storage tab > Hosts > click the + symbol to create a host(s). Click Create Multiple to create multiple hosts at a time. Provide a name (vvf-esxhost-FC-#), set start Number to 1, count to 3 and Number of Digits to 2 , select ESXI for Personality and click Create. Ignore Add HostGroup option when it is prompted.

Step 3.          Click the newly created host  > click the + symbol under Host Ports and select Configure WWNs.

Step 4.          Click the + symbol and provide WWPN of the host from 00-vHBA-A and 01-vHBA-B by referring to Table 29. Repeat this steps for all the hosts.

Step 5.          Click Storage > Volumes tab, click the + to create boot volumes. Click Create Multiple to create multiple boot volumes at a time. Provide a name (VVF-ESXi-FC-#), set size to 500GB, set start Number to 1, count to 3 and Number of Digits to 2 and click Create.

Step 6.          Click each volume, click the + under Connect Hosts box, click Connect. Select the corresponding host name from the available host list, set LUN ID as 1 and click Connect.

Step 7.          Power on each host and connect to one of the KVM consoles of a host. If it is configured correctly, the host will be able to detect the FC boot volume with four paths as shown below:

When you are deploying the VMware vSphere Cluster with VCF installer, a datastore is required for storing all the management VMs ( vCenter, Operations, License manager VM, and so on). Complete the following steps to create a volume on Everpure storage and associate it with the FC based Host group.

Step 8.          Log into Everpure FlashArray with admin credentials.

Step 9.          Click Storage > Volumes tab, click the + to create volumes. Provide a name (VVF-FC-Primary), set size to 5TB and click Create.

Step 10.       Click Hosts > Host Group, click + to create a new Host Group. Provid the name VVF-FC-Clus and add all FC hosts defined in the previous hosts.

Step 11.       Click the newly created Host Group > Connected Volumes, click the ellipses and select Connect. Select VVF-FC-Primary volume and click Connect. With this, the volume will be connected to all the ESXi hosts.

Procedure 2.    Create Hosts with IQN Targets

Step 1.          Log into Everpure FlashArray with admin credentials.

Step 2.          Click the Storage tab > Hosts > click the + symbol to create a host(s). Click Create Multiple to create multiple hosts at a time. Provide a name (vvf-esxhost-iSCSI-#), set start Number to 1, count to 5 and Number of Digits to 2 and click Create. Ignore Add HostGroup option when it is prompted.

Step 3.          Click the newly created host  > click the + symbol under Host Ports and select Configure IQNs.

Step 4.          Enter IQN of the host from is IP-based compute nodes by referring to Table 29. Repeat this step for all the hosts.

Step 5.          Click Storage > Volumes tab, click the + to create boot volumes. Click Create Multiple to create multiple boot volumes at a time. Provide a name (vvf-esxhost-iSCSI-#), set size to 500GB, set start Number to 1, count to 5 and Number of Digits to 2 and click Create.

Step 6.          Click each volume, click the + under Connect Hosts box, click Connect. Select the corresponding host name from the available host list and click Connect.

Step 7.          Power on each host and connect to one of the KVM consoles of a host. If it is configured correctly, the host will be able to detect iSCSI boot volume with two paths as shown below:

As explained in the previous section, when you are deploying VVF with VCF installer, a datastore is required for storing all the management VMs. If you want to leverage the NFS volume, first configure the NFS service on the Everpure FlashArray and mount it to manually as explained here.

VVF Deployment using VCF Installer

This section contains the following:

●     Prerequisites

●     Deploy VVF using VCF Installer

Prerequisites

The following list provides the necessary prerequisites for the VVF deployment with VCF installer.

●     Download the latest ESXi ISO, VCF Installer from support.broadcom.com portal with your credentials.

●     DNS and DHCP (optional) Configuration.

●     Deploy ESXi on all the compute nodes and prepare the nodes

●     Install VCF installer on an external ESXi host.

●     Log into VCF installer portal and download the required binaries from Broadcom.

Procedure 1.    Download ESXi and VCF installer

Step 1.          Log into the Broadcom website with your credential and click My Downloads.

Step 2.          Search for vsphere and click VMware vSphere Foundation and then click 9.1.0.0 release.

Step 3.          From the Primary Downloads list, download latest VCF installer.

Step 4.          Click the Custom ISOs tab, download the ESXi 9.1.0.0 ISO image. This image includes all the latest enic and fnic drives and Addons.

Procedure 2.    DNS and DHCP Configuration (optional)

DHCP and DNS infrastructure services need to be preconfigured as required by VVF deployment by using VCF installer.

For Manual deployment, you can also configure DNS and DHCP for automatic IP and fully qualifies names assignment to the ESXi hosts.

The following tables details the required DNS entries for VVF deployment using VCF installer. An additional entry for the VCF installer is required as list in Table 30.

Table 30.      DNS entries for VVF deployment

 

Procedure 3.    Install ESXi on UCS Servers and Configuration

Cisco Intersight enables you to install vMedia-based operating systems managed servers in your data center. This capability allows you to perform an unattended OS installation on one or more Cisco UCS  blade/rack servers through a simple process. Operating system installation requires Cisco Intersight Advantage license.

This procedure focus on how to use the built-in keyboard, video, mouse (KVM) console and virtual media features in Cisco Intersight to map remote installation media to individual servers.

Step 1.          Log into Intersight and go to Operate > Servers, select a server and launch KVM console. Power Cycle the server and ensure the server has detected the SAN boot volume and start loading the ESXi hypervisor.

Step 2.          After the ESXi installer is finished loading (from the last step), press Enter to continue with the installation.

Step 3.          Read and accept the end-user license agreement (EULA). Press F11 to accept and continue.

Step 4.          Select the Everpure boot LUN that was previously set up as the installation disk for ESXi, and press Enter to continue with the installation.

Step 5.          Select the appropriate keyboard layout and press Enter.

Step 6.          Enter and confirm the root password and press Enter.

Step 7.          The installer issues a warning that the selected disk will be repartitioned. Press F11 to continue with the installation.

Step 8.          After the installation is complete, disconnect the ISO from KVM session and press Enter to reboot the server.

Step 9.          Repeat steps 1 – 8 for all ESXi hosts.

Step 10.       After the server has finished rebooting, in the UCS KVM console, press F2 to customize VMware ESXi.

Step 11.       Log in as root, enter the password set during installation, and press Enter to log in.

Step 12.       Select the Configure Management Network option and press Enter.

Step 13.       Select VLAN and provide VLAN number 1061 (IB-Mgmt VLAN) and press Enter.

Step 14.       Select IPv4 Configuration and then select Set Static IPV4 address option and provide the IP Address, Subnet, and Gateway details. Press Enter.

Step 15.       Select DNS Configuration and press Enter. Provide DNS Server details and Hostname and press Enter. Select Custom DNS Suffixes and press Enter. Ensure that no suffixes are listed and press Enter.

Step 16.       Select IPv6 Configuration and press Enter. Disable the IPv6 address and press Enter again. Press y to reboot the host. When the host is back online, Static IP must be seen on the ESXi host console.

Step 17.       When ESXi host is back online, log into the Web Client UI using ESXi IP address.

Step 18.       Go to Action Menu, click Services > Enable Secure Shell (SSH).

Step 19.       SSH to ESXi host using root user and password.

Step 20.       Validate the ESXi host name and Fully Qualified Domain Name (FQDN) by running the command esxcli system hostname get.

Step 21.       If the FQDN is not configured correctly, you can edit it by running the command esxcli system hostname set –fqdn = NEW_FQDN.

Step 22.       Regenerate the self-signed certificate by running the following command and then reboot the ESXi host:

Step 23.       Log back into the ESXi host using Web Client and Disable Secure Shell.

Step 24.       Right-click Storage and select New Datastore. Select VMFS or NFS based on your Primary Storage. Click Next.

Step 25.       Provide a name (aa06fs-vvfmgmt-ds01) to the datastore and select the Everpure Volume created for Primary storage purpose. Click Next and complete datastore creation.

Step 26.       Repeat this procedure for all remaining hosts.

Procedure 4.    Install VCF installer

An ESXi host is required for installing the VCF installer VM using the OVA file downloaded in the previous steps.

Step 1.          To deploy the VCF installer on a ESXi host see Deploy VCF Installer. Ensure to meet the strong password requirements for root and local users.

Step 2.          When completed, log into the VCF installer using its FQDN name as shown below:

Procedure 5.    Download Binaries for VVF Deployment

Step 1.          After logging into the VCF installer, click DEPOT SETTINGS AND BINARY MANAGEMENT. Under Online Depot, click Configure.

Step 2.          Use the Service ID and generate the Activation code using the VCF console as explained here.

Step 3.          Paste the Activation code and click AUTHENTICATE.

Step 4.          Once authenticated successfully, select VMware vSphere Foundation from the Product drop-down list and 9.1.0 for version list. Select all binaries and click Download. Wait until all the binaries are successfully downloaded into the VCF installer VM.

Step 5.          Click Return Home to return to the launch page of the VCF installer. Under the Deploy section, select VMWare vSphere Foundation to launch the VVF deployment wizards.

Deploy VVF using VCF Installer

This section provides detailed steps to deploy a VVF cluster using the VCF installer Deployment wizard started in the previous step.

Procedure 1.    Install VMWare vSphere Foundation Platform using VCF Installer

Step 1.          In a web browser, log into the VCF Installer appliance here: https://installer_appliance_FQDN.

Step 2.          Enter the admin@local user and password that you provided when you deployed the appliance and then click Log In.

Step 3.          Click Deployment Wizard > VMware vSphere Foundation and then select Deploy new VMware vSphere Foundation. Click Next.

Step 4.          Do not select any existing components. Click Next.

Step 5.          For the Network Options, choose the Default option where in a dedicated VLAN is used for VM management and VCF management VMs and services.

Step 6.          For Storage option, select  VMFS on Fibre Channel (FC) option. Click Next.

Step 7.          On the Review prerequisites page,  click PRE-FILL GENERATED FQDNS IN WIZARD and for the suffix enter aa06fs and for domain name enter flashstack.local. The complete FQDNs names are prepopulated as defined in your DNS list (refer DNS and DHCP configuration section). Click VALIDATE ALL to validate the DNS entries. Click Save and then click Next.

Step 8.          On the General Information tab, set version to 9.1.0.0, provide your DNS and NTP server details and set DNS suffix (flashstack.local). Click Next.

Step 9.          On the Hosts tab, provide a FQDN of all the nodes and click CONFIRM ALL FINGER PRINTS. Click Next.

Step 10.       On the Networks tab, provide gateway IPs and VLAN (1061- IB-MGMT and 1062 - VM-MGMT1062) for ESX Management Network and VM Management Network, respectively. Provide a pools IP (at least 10) for Services IP Pool. Provide VLAN and MTU for vMotion traffic and provide list of  IPs for vmks that are going to be created on each ESXi node for vMotion traffic. Click Next.

Step 11.       On the VCF management tab, ensure the green tick mark is displayed on all the services. Click Next.

Step 12.       On the vCenter tab provide, change the vCenter, datacenter and cluster names optionally and click Next.

Step 13.       On storage tab, provide a datastore name (aa06fs-vvfmgmt-ds01) that was mounted on esxi nodes. Click Next.

Step 14.       On the Distributed Switch, click Add DISTRIBUTED SWITCH to add distributed switches and Distributed PortGroups. Initially, the following Distributed switches (vDS) and Distributed PortGroups can be created. Once the cluster is deployed, other vDS can be created based on the requirement for other traffics. Table 31 and the following screenshot lists the settings used for vDS used for this validation. Click Next when the two switches are configured.

Table 31.      Distributed Switch Configuration

Step 15.       Review the summary and then click Deploy. The deployment will take two to three hours.

Step 16.       Once the cluster is deployed successfully, click REVIEW PASSWORDS and then click the CSV icon to download copy of all component’s passwords in a CSV format.

Step 17.       Launch the Operations manager with credentials and review the entire cluster. Click GO TO LICENSES & REGISTRATION to license your VVF components.

For the most common issues of VVF deployment, see Troubleshooting VMware Cloud Foundation or vSphere Foundation Deployments.

Manual Deployment and Configuration of VVF

This chapter contains the following:

●     Prerequisites

●     Install and Configure ESXi 9.1

●     Install and Configure vCenter 9.1

●     Update Drivers with vSphere Lifecycle Manager

●     ESXi Network Configuration

●     ESXi Storage Configuration

●     Manual Installation of VCF Management Services

Deploying VMware vSphere Foundation 9.1 as a standalone infrastructure is fully supported, operating independent of automated VCF management services and the VCF installer. The following functionalities are available without VCF management services:

●     Compute, storage, networking, and administrative operations by using native vSphere management interfaces.

●     Running, patching, and upgrading vSphere Foundation by using standard vSphere lifecycle mechanisms.

However, specific VCF management services such as VCF operations and License server, can be individually installed and register them with your VVF platform and then use them for managing your VVF platform. For more details on deploying VCF operations manually, see Manual deployment of VCF Operations in a vSphere Foundation 9.0, 9.1 environment.

Prerequisites

Optionally, for manual deployment of VVF, DHCP and DNS infrastructure services can be optionally preconfigured which help automatic IP and fully qualified names assignment to the ESXi hosts.

DHCP and DNS Configuration

For manual deployment, for the automatic assignment of IP addresses and fully qualified names to the ESXi hosts, a network administrator must reserve IP addresses that you would like to assign to the ESXI hosts and not get assigned to the other hosts or services. IP reservations have been created in DHCP server using the MAC addresses of 00-IBMgmt-A vNIC from each server profile. You can find the MAC addresses of server profiles gathered in Table 29. DNS entries for each of ESXi host and vCenter server which we are going to install have been created as shown below:

Install and Configure ESXi 9.1

This section provides the procedure to install VMware ESXi 9.1 in a FlashStack environment. On successful completion of these steps, multiple ESXi hosts will be provisioned and ready to be added to VMware vCenter.

Several methods exist for installing ESXi in a VMware environment. These procedures focus on how to use the built-in keyboard, video, mouse (KVM) console and virtual media features in Cisco Intersight to map remote installation media to individual servers.

Cisco Intersight vKVM enables the administrators to begin the installation of the operating system (OS) through a vMedia connection to the Cisco Custom ISO.

Procedure 1.    Install ESXi 9.1

Step 1.          Log into Intersight and go to Operate > Servers, select a server and launch the KVM console. Power Cycle the server and ensure the server has detected the SAN boot volume and start loading the ESXi hypervisor.

Step 2.          After the ESXi installer is finished loading (from the last step), press Enter to continue with the installation.

Step 3.          Read and accept the end-user license agreement (EULA). Press F11 to accept and continue.

Step 4.          Select the Everpure boot LUN that was previously set up as the installation disk for ESXi, and press Enter to continue with the installation.

Step 5.          Select the appropriate keyboard layout and press Enter.

Step 6.          Enter and confirm the root password and press Enter.

Step 7.          The installer issues a warning that the selected disk will be repartitioned. Press F11 to continue with the installation.

Step 8.          After the installation is complete, press Enter to reboot the server.

Step 9.          Repeat this procedure for all ESXi hosts.

Procedure 2.    Update the Management Network

Step 1.          After the server has finished rebooting, in the UCS KVM console, press F2 to customize VMware ESXi.

Step 2.          Log in as root, enter the password set during installation, and press Enter to log in.

Step 3.          Use the down arrow key to select Troubleshooting Options and press Enter.

Step 4.          Select Enable ESXi Shell and press Enter and select Enable SSH and press Enter. Press Esc to exit the Troubleshooting Options menu.

Step 5.          Select the Configure Management Network option and press Enter.

Step 6.          Select Network Adapters and press Enter. Ensure the vmnic numbers align with the numbers under the Hardware Label (for example, vmnic0 and 00-IBMgmt-A). Select the vmnic1, press the space bar and press Enter.

Step 7.          Select IPv4 Configuration and press Enter. Enter the required details and complete the IPv4 address configuration. Select IPv6 Configuration and press Enter. Disable the IPv6 addressing and press Enter. Press Y to reboot the host.

The ESXi host should reboot successfully with the IPv4 and hostname set as per the DHCP IP and name reservation.

Step 8.          Repeat this procedure for all ESXi hosts.

Reset VMware ESXi Host VMKernel Port MAC Address

By default, the MAC address of the management VMkernel port vmk0 is the same as the MAC address of the Ethernet port it is placed on. If the ESXi host’s boot LUN is remapped to a different server Service Profile with different MAC addresses, a MAC address conflict will exist because vmk0 will retain the assigned MAC address unless the ESXi System Configuration is reset.

Procedure 1.    Reset VMKernel vmk0 MAC addresses

Step 1.          From the ESXi console menu main screen, select Macros > Static Macros > Ctrl + Alt + F > Ctrl + Alt + F1 to access the VMware console command line interface.

Step 2.          Log in as root.

Step 3.          Type esxcfg-vmknic –l to get a detailed listing of interface vmk0. vmk0 should be a part of the Management Network port group. Note the IP address and netmask of vmk0.

Step 4.          To remove vmk0, type esxcfg-vmknic –d Management Network.

Step 5.          To add vmk0 with a random MAC address, type esxcfg-vmknic –a –i <vmk0-ip> -n <vmk0-netmask> Management Network.

Step 6.          Verify vmk0 has been re-added with a random MAC address by typing esxcfg-vmknic –l.

Step 7.          Tag vmk0 as the management interface by typing esxcli network ip interface tag add -i vmk0 -t Management.

Step 8.          When vmk0 was added, if a message displays vmk1 was marked as the management interface, type esxcli network ip interface tag remove -i vmk1 -t Management.

Step 9.          Press Ctrl-D to log out of the ESXi console.

Step 10.       Select Macros > Static Macros > Ctrl + Alt + F > Ctrl + Alt + F2 to return to the VMware ESXi menu.

Step 11.       Ensure you update the DHCP Reservation for the ESXI hosts with new MAC address that was picked up by the ESXi

Install and Configure vCenter 9.1

To install VMware vCenter 9.1, see Install vCenter 9.1. The following screenshot shows the vCenter landing page when vCenter accessed using vCenter Fully Qualified Name (https://vcsa9.flashstack.local):

Configure VMware vCenter Server

Procedure 1.    Configure VMware vCenter Server

Step 1.          Log into a vCenter using https://<vcenter-ip-address>:5480 and enter the root credential that were provided during the vCenter installation.

Step 2.          Click Time, click Edit, and select the appropriate timezone for your vCenter server.

Step 3.          Log into a vCenter using https://vcenter-FQDN  and enter administrator credential that has been provided during the vCenter installation.

Step 4.          Right-click the vCenter server and select New Datacenter. Enter a name to the datacenter and click OK.

Step 5.          Right-click the newly created Data Center and click New Cluster. Enter a name for the cluster, Enable DRS and HA feature and click Extract an image from a new  host option and click Next.

Step 6.          Enter the first esxihost server’s fully qualified name along with root credentials and click Next. Click Extract Image. Confirm the image shows it matches with what you have installed. Click Next, click Review and then click Finish.

Step 7.          Add the remaining hosts to the cluster by right-clicking the newly created cluster and select Add Hosts. Enter the root credentials for adding all ESXi hosts to the cluster. On the Import Image page, select Keep existing image on cluster.

Step 8.          When all hosts are added to the cluster, clear any alerts and alarms associated with the hosts.

Procedure 2.    Avoid boot failures when moving server profiles with uefi secure enabled

Typically, hosts in FlashStack Datacenter are configured for boot from SAN. Cisco UCS supports stateless compute where a server profile can be moved from one blade or compute node to another seamlessly. When a server profile is moved from one blade to another blade server with the following conditions, the ESXi host runs into PSOD and ESXi will fail to boot:

●     TPM present in the node (Cisco UCS M5/M6/M7/M8 family servers)

●     Host installed with ESXi 7.0 U2 or above

●     Boot mode is UEFI Secure

●     Error message: Unable to restore system configuration. A security violation was detected. https://via.vmw.com/security-violation.

Step 1.          SSH into each ESXi host.

Step 2.          Gather the Recovery Key of each ESXi host using the following command and preserve them in a safe location:

Step 3.          After associating the Server Profile to the new compute-node or blade, stop the ESXi boot sequence by pressing Shift + O when the ESXi boot screen appears.

Step 4.          Add the recovery key using following boot option: encryptionRecoveryKey=recovery_key. Use File > Paste Clipboard Text and Send to paste in the recovery key. Press Enter to continue the boot process.

Step 5.          To persist the change, enter the following command at the VMware ESXi ssh command prompt:

Step 6.          Exit the ESXi nodes from maintenance mode and clear any alerts and alarm message on each ESXi host

Procedure 3.    NTP Configuration on ESXi Nodes

To update the NTP server list on each ESXi nodes, follow this procedure.

Step 1.          Log  into vCenter and select an ESXi node. Go to Configure > System > Time Configuration.

Step 2.          Click Add Service and select Network Time Protocol. Enter one or more NTP server list and click OK.

Step 3.          Repeat this procedure on each ESXi server.

Procedure 4.    Intersight Plugin for vCenter

Integration of Intersight with vCenter is a standout capability of this solution. The integration unifies server hardware management directly within the vSphere Client by leveraging the Hardware Support Manager (HSM) alongside vSphere Lifecycle Manager (vLCM) to seamlessly update base OS images, drivers, and server firmware. Concurrently, it functions as a native VMware Proactive HA provider, continuously monitoring critical hardware components for impending faults. Upon detecting an anomaly, the plugin automatically alerts vCenter to place the degraded host into Quarantine or Maintenance Mode, triggering live vMotions to protect running virtual machines from unexpected downtime. This integrated approach simplifies infrastructure lifecycle management while ensuring maximum application uptime.

Step 1.          Log into the Intersight and go to Systems > Targets and click Claim a Target.

Step 2.          Select Hypervisor and select the VMware vCenter icon. Click Start.

Step 3.          Select the Cisco Assist and enter the vCenter server details as shown below:

Step 4.          To install the Intersight plugin within vCenter, select Enable Hardware Support Manager Plugin and Enable Cisco Intersight Plugin for VMware vSphere Client options. Click Claim.

Step 5.          Logout from vCenter and log back in. The Intersight plugin will be listed under vCenter inventory as shown below:

Step 6.          Server inventory can be viewed from the Intersight plugin with in the vCenter server as shown below:

Complete the following steps to configure proactive HA in vCenter to monitor server hardware components such as CPU, memory, and storage and take proactive actions to prevent the server failures:

Step 7.          Click Menu > Inventory > select the Cluster you want to where you want to enable proactive HA.

Step 8.          Click Configure > vSphere Availability > Edit to edit proactive HA settings.

Step 9.          Enable Proactive HA and select the Intersight provider and then click Status to enable feature.

Step 10.       Click OK to save the settings. Click Edit again and click Failures & Responses tab.

Step 11.       For Automation, select Automated to perform the remediation action automatically and select Remediation to Quarantine mode as shown below. Click OK to save settings.

Update Drivers with vSphere Lifecycle Manager

vSphere Lifecycle Manager (vLCM) is a tool in vCenter Server that automates software and firmware updates for ESXi hosts using a single image model. It replaces the older vSphere Update Manager (VUM), enforcing cluster-wide consistency for hypervisors, drivers, and hardware firmware.

Key Features and Operation:

●     Declarative Images: Define a single desired-state image (ESXi version, vendor add-ons, and firmware) for an entire cluster.

●     Full-Stack Updates: Manages ESXi patches, drivers, and hardware firmware updates simultaneously.

●     Compliance Checks: Automatically scans clusters for configuration drift and highlights non-compliant hosts.

●     Automated Remediation: Works with vSphere DRS to safely place hosts into maintenance mode, evacuate virtual machines, update, and reboot

Leveraging vLCM and Intersight plugin for vCenter, VIC drivers, UCS tools and Server firmware can be updated from vCenter server. However, since we have used Cisco Customer image for vSphere ESXi 9.1 is used and it includes all the latest drivers as shown below. Hence there is no need to update the drivers for now. Refer NVIDIA vGPU Configuration section for detailed steps for updating the vCenter Cluster image with NVIDIA vGPU drivers and for installing the drivers in the ESXi hosts. Same steps can be followed even for updating vNIC drivers and firmware.

ESXi Network Configuration

This section provides information about the ESXi host network configuration used for this FlashStack system. The ESXi hosts in IP-Based design should discover eight vmnic network adapters. The first two (vmnic0 and vmnic1) adapters are used for creating a standard switch (vSwitch0) which is used for management traffic. Vminic2 and 3 are used for creation of Distributed vSwitch which is used for VM Management, NFS and vMotion traffics. Vmnic4 and 5 are used for creating two standard switches for iSCSI storage traffic via Fabric A and B, respectively. Vmnic6 and 7 are used for creating a Distributed switch with two different Distributed Groups each used NVMe over TCP traffic via Fabric A and B, respectively. The following table provide more details in all the standard vswitch and Distributed vSwitch details along with their corresponding PortGroups and vmnic used.

Table 32.      ESXi Host Network Configuration for IP-Based design

The ESXi hosts in FC-Based design should discover four vmnic network adapters. The first two (vmnic0 and vmnic1) adapters are used for creating a standard switch (vSwitch0) which is used for management traffic. Vminic2 and 3 are used for creation of Distributed vSwitch which is used for VM Management and vMotion traffics. Table 33 lists more details about the standard vSwitch and Distributed vSwitch details along with their corresponding PortGroups and vmnic used.

Table 33.      ESXi Host Network Configuration for FC-Based Design

Procedure 1.    Add secondary iSCSI vmnic and verify the MTU on iSCSI vmkernel adapters

Step 1.          Log into vCenter and select the ESXi host and go to Configure > Virtual Switches. Click iScsiBootvSwitch and click Edit.

Step 2.          From the Properties tab, set the MTU to 9000 and click the Teaming and Failover option. Under Active Adapters ensure vmnic4 is set. Click OK.

Step 3.          Expand the vSwitch and select the vmk1 and click the ellipses and select Edit Settings.

Step 4.          Set the MTU to 9000 and click OK.

Step 5.          Select the ESXi host and go to Configure > Virtual Switches. Click Add Networking.

Step 6.          Select VMKernel Network Adapter and click Next.

Step 7.          Select New Standard Switch and set the MTU to 9000. Click Next.

Step 8.          Using the UP and Down controllers, move vmnic5 under Active Adapters. Click Next.

Step 9.          Under Port Properties tab, for the Network label enter iSCSIBootPg-B and set the MTU to 9000 by selecting Custom value. Click Next.

Step 10.       Click Use static ipv4 settings option and enter the IP address and the Subnet. Click Next. You can get IP Addresses from the Server Profiles as listed in the Table 29.

Step 11.       Review the summary and click Finish.

Step 12.       SSH into the ESXi host and ensure the FlashArray iSCSI IP addresses are reachable with MTU and without defragmenting the packet using the following command:

The following screenshot shows the iSCSI VMKernel adapters after being configured:

Procedure 2.    Create and Configure Distributed Virtual Switch VM Management, NFS, and vMotion Traffic

Step 1.          Log  into vCenter and go to Inventory and click the Networking icon.

Step 2.          Right-click the Data Center and select Distributed Switch. Select New Distributed Switch.

Step 3.          Enter a Name for the vDS (vDS0) and click Next. Leave the Switch version as Default (9.1.0) and click Next.

Step 4.          Enter 2 for Number of Uplinks and enter a Name to Default Port Group (VMMgmt-1062). Click Next.

Step 5.          Review the settings and click Finish.

Step 6.          Right-click the newly created vDS, go Settings > Edit Settings.

Step 7.          Click the Advanced tab and set the MTU to 9000 and set the Protocol Type to Link Layer Discovery Protocol. Click OK.

Step 8.          Right-click vDS0 and go to Distributed Port Group and select New Distributed Port Group.

Step 9.          Enter the name as vMotion and Set VLAN Type to VLAN and for the VLAN ID enter 3000. Click Customize default policies Configuration checkbox and click Next to go to the Teaming and Failover tab.

Step 10.        Select Uplink 2 and move it under Active uplinks and move Uplink2 to Standby uplinks by using the Move UP and Down controllers.

Step 11.       Review the settings and click Finish.

Step 12.       Repeat steps 1 - 11 to create another Distributed Port Group for NFS. Set the name to NFS-VLAN3060, set the VLAN to 3060 and ensure Uplink 1 and Uplink2 are under Active uplinks.

Step 13.       Log into vCenter and go to Inventory and click the Networking icon. Right-click the newly created Distributed Switch (vDS0). Click Add and Manage Hosts.

Step 14.       Click Add Hosts then click Next. Select the required ESXi hosts that are configured with the required vNICs. Click Next.

Step 15.       Under the Manage Physical Adapters screen, set vmnic2 to uplink 1 and vmnic3 to uplink 2. Click Next.

Step 16.       Skip Manage VMKernel adapters and Migrate VMs Networking. Review settings and click Finish.

Step 17.       Select the ESXi host and go to Configure > Virtual Switches. Click Add Networking.

Step 18.       Select VMKernel Network Adapter and click Next. Select vMotion DGP under target Device and click Next.

Step 19.       Under the Port Properties tab, set the MTU to 9000 by selecting Custom and set the TCP/IP stack to vMotion. Click Next.

Step 20.       Enter an IP address and Subnet mask then click Next. Review the settings and click Finish. After configuring the VMkernels for vMotion on each ESXi hosts, verify the ESXi hosts can communicate with each other over the VMKernel adapter with MTU as shown below:

Step 21.       To Create VMKernel for NFS traffic, go to vDS0 and select Add Networking. Select NFS-VLAN3060 DGP under Target Device and click Next.

Step 22.       Under Port Properties tab, Set MTU as 9000 by selecting Custom and Click Next.

Step 23.        Enter IP address and Subnet mask for NFS adapter and click Next. Review the settings and Click on Finish.

Procedure 3.    Create and Configure Distributed Virtual Switch for NVMe over TCP Storage Traffic

For NVMe over TCP storage traffic, two separate distributed port groups one for each Fabric are created each with one vmnic only.

Step 1.          Right-click the Data Center and select Distributed Switch and then select New Distributed Switch.

Step 2.          Right-click vDS1 and go to Distributed Port Group and select New Distributed Port Group.

Step 3.          Enter the name NVMe-TCP-A and do not change VLAN Type. Click the Customize default policies Configuration checkbox and click Next to go to Teaming and Failover tab.

Step 4.           Select Uplink 1 and move it under Active uplinks. Do not use Uplink2 for NVMe-TPC-A port group as shown below.

Step 5.          Review the settings and click Finish.

Step 6.          Create another Port Group NVMe-TCP-B and configure vmnic2 as Active uplinks. Do not use Uplink 1 for NVMe-TPC-B port group.

Step 7.          Log into vCenter and go to Inventory and click the Networking icon. Right-click the newly created Distributed Switch (vDS1). Click Add and Manage Hosts.

Step 8.          Click Add Hosts and click Next. Select the required ESXi hosts that are configured with the required vNICs. Click Next.

Step 9.          Under the Manage Physical Adapters screen, set vmnic6 to uplink 1 and vmnic7 to uplink 2. Click Next.

Step 10.       Skip Manage VMKernel adapters and Migrate VMs Networking. Review settings and click Finish.

Step 11.       Select the ESXi host and go to Configure > Virtual Switches. Click Add Networking.

Step 12.       Select VMKernel Network Adapter and click Next. Select NVMe-TCP-A under target device as shown below and click Next.

Step 13.       Under the Port Properties tab, set MTU to 9000 and enable NVMe over TCP Service. Click Next.

 

Step 14.       Enter the IP address and Subnet mask and click Next.

Step 15.       Review the settings and click Finish.

Step 16.       Repeat steps 1 - 16 to add VMKernel for the NVMe-TCP-B Port Group.

Step 17.       Repeat steps 12 to 16 to create two VMkernels on all the required ESXi hosts. The following screenshot shows the VMkernels configured for TCP over NVMe storage traffic:

Step 18.       Make sure the FlashArray IP addresses are reachable from each ESXi host with MTU by running the following command:

ESXi Storage Configuration

This section provides storage configuration steps for the various storage protocols validated in this architecture.

Procedure 1.    Update ESXi Host iSCSI Storage Adapters with all Target IP addresses

Step 1.          Log into the vCenter server, expand the cluster and select the ESXi host. Go to Config and select Storage Adapters and select iSCSI Software Adapter (vmhba64).

Step 2.          Click Dynamic Discovery and click Add. Enter FlashArray CT0.eth10 IP address and click OK.

Step 3.          Repeat steps 1 and 2 to add the remaining three FlashArray iSCSI IP addresses.

Step 4.          Click the Static Discovery tab and confirm all Target IP addresses are added along with Target IQN as shown below:

Step 5.          When all target IPs are added, log into FlashArray and check the connections; go to Health > Connections as shown below. The ESXi hosts must be connected to the FlashArray with Redundant paths.

Procedure 2.    NVMe over TCP Configuration

Step 1.          Under Inventory select an ESXi host running NVMe-TCP. In the center pane, go to Configure > Storage > Storage Adapters.

Step 2.          Click ADD SOFTWARE-ADAPTER > Add NVMe over TCP adapter. From the drop-down list select vmnic6/nenic and click OK. A new vmhba should appear under Storage Adapters.

Step 3.          Repeat steps 1 and 2 to map the vmnic7 to another storage adapter.

Step 4.          Select the first VMware NVMe over TCP Storage Adapter added (for example, vmhba65). In the middle of the window, select the Controllers tab. Click ADD CONTROLLER.

Step 5.          Enter the IP address of the FlashArray’s Controller 0.eth10 interface that is configured with nvme-tcp service and then click DISCOVER CONTROLLERS. Select the two controllers in the NVMe-TCP-A subnet and click OK. The two controllers should now appear under the Controllers tab after clicking Refresh.

Step 6.          Repeat steps 1 - 5 for the second VMware NVMe over TCP Storage Adapter added (for example, vmhba66). In the middle of the window, select the Controllers tab. Click ADD CONTROLLER. This time, enter the IP address of FlashArray’s Controller 0.eth11 interface which is configured with nvme-tcp service and then click DISCOVER CONTROLLERS.

Step 7.          When a NVMe-Device (volume) is created and connected ESXi host, the NVMe Device must be connected to the ESXi hosts using 4 paths.

Step 8.          SSH to the ESXi host and execute the following to configure NVMe Devices to use High-Performance Plugin (HPP).

Step 9.          Gather the above NQN ID for each ESXi hosts and create a Host and Host Groups in the FlashArray. Or use the Everpure Plugin to provision the Datastores from vCenter itself.

Step 10.       Repeat this procedure for each ESXi host running NVMe over TCP.

Procedure 3.    NVMe over FC Configuration

For NVMe over FC, no special configuration is required. Once the zoning is configured properly in the MDS switches (using the Host nvme initiators and FlashArray nvme Targets), the nvme disks will be automatically discovered.

Step 1.          SSH to the ESXi host and run the following command to configure NVMe Devices to use High-Performance Plugin (HPP):

Step 2.          Gather the FC-NVMe initiator NQN IDs for each ESXi hosts and create a Host and Host Groups in the FlashArray. Or use the Everpure Plugin to provision the Datastores from vCenter itself.

Step 3.          Under Inventory select an ESXi host running FC-NVMe. In the center pane, go to Configure > Storage > Storage Devices. The NVMe Fibre Channel Disk should be listed under Storage Devices.

Step 4.          Select the NVMe Fibre Channel Disk, then select Paths underneath. Verify 2 paths have a status of Active (I/O) and 2 paths have a status of Active.

Step 5.          Repeat this procedure on each ESXi host that run NVMe over FC.

Manual Installation of VCF Management Services

Deploying and running a vSphere Foundation 9.1 without VCF management services is a supported deployment model. If you need VCF management services, you can deploy them individually and use them to manage your VVF cluster as explained here.

Table 34 lists the references to deploy and registering VCF operations and License Server with your standalone VVF cluster.

Table 34.      Manual deployment of VCF management services

Install Everpure Remote Plugin for vSphere Client and Provision Datastore

The Everpure vSphere Client Plugin integrates directly into the VMware vSphere HTML5 interface. It let administrators provision datastores (VMFS, NFS, vVols), monitor per-VM performance, manage snapshots, and handle replication arrays straight from the vCenter dashboard without switching management tools.

With vSphere 8.0, VMware no longer supports local plugins to be installed in vCenter. Remote versions of Everpure's vSphere plugin must be used. For this validation, the Remote Plugin version 5.6.0 is downloaded and deployed on a ESXi host hosted outside the FlashStack environment. For the installation instruction, see Configure the VMware Appliance and Register the vSphere Plugin. Once installed, vCenter needs to be registered for plugin to be installed and shown up in vCenter.

Procedure 1.    Register vCenter with the Everpure Plugin

Step 1.          Launch the Console of the Everpure plugin VM and register the vCenter with remote plugin as shown below:

Step 2.          Log into the vCenter and click the three line menu icon located left top corner and select the Pure Storage plugin as shown below.

Step 3.          Click ADD ARRAY OR FLEET and provide the FlashArray details. You can also import the registered Everpure arrays directly from Pure1; click Import Arrays from Pure1. Click Submit when the details are provided.

Step 4.          Log out and log into the vCenter with administrator account.

Procedure 2.    Provision Datastores and Volumes using Everpure Plugin

When the plugin is installed, the datastores can be provisioned directly from vCenter without having to touch the components in the solution stack. When the datastore is provisioned using the plugin, the corresponding volume will be created in the FlashArray, mapped to the ESXi hosts using host and host groups constructs and finally datastore is mounted to the ESXi hosts.

Step 1.          Log into vCenter, right-click the ESXi cluster, go to Pure Storage plugin and select Create Datastore.

Step 2.          Select VMFS/vVols Datastore when you would like to provision datastore using iSCSI/FC /NVMe over FC. Select NFS if you would like to mount NFS file share to the ESXi cluster. For this validation, NFS is selected. Click Next.

Step 3.          Select the ESXi Cluster under Compute Resource. Click Next.

Step 4.          Select the FlashArrayXL//170 under Storage and click Next.

Step 5.          From the File System tab, enter a name (nfs-datastore) for the Datastore and size (4TB). Click Next.

Step 6.          Under Option, select NFS v4.1 and set Number of TCP Connections to 6. Click Next.

Step 7.          Select the VIF that was created in the FlashArray for Network Interface. Click Next.

Step 8.          From Vmnic binding, check the boxes Enable vmknic binding and Use the same vmknic for all hosts. Select the VMKernel adapter that was added for NFS storage traffic. Click Next.

Step 9.          Set None for Pod and click Next. From Policy, ensure no-root-squash is policy is selected and click Next.

Step 10.       Review the settings and click Finish.

The File system is mounted successfully to the ESXi Cluster and appears in the Datastore tab.

The Corresponding volume, file system, and export policies are automatically created in the FlashArray as shown below:

Configure NVIDIA Virtual GPU Manager in VMware vSphere ESXi

DirectPath I/O ( Dedicated Passthrough) option maps the entire physical GPU directly to a single virtual machine (VM) with zero overhead, making it ideal for massive LLM training or heavy HPC workloads. NVIDIA Virtual GPU Manager (vGPU) allows the GPUs to be shared across the multiple VMs in the vSphere environment. It uses an ESXi-level host driver (VIB) to carve a physical GPU into multiple isolated virtual pieces which can be assigned VMs. It supports vMotion, DRS automation, and HA clustering while preserving near-native computing speeds.

This section provides the procedures to configure vGPUs with NVIDIA Licensing Service (NLV) in a vSphere environment. NVIDIA Enterprise Account is required to setup either on-prem Delegated Licensing Server (DLS) or Cloud based Licensing server (CLS). This section assumes that you have required NVIDIA account and you have installed and configured either DLS or CLS services by following this NVIDIA quick setup link: https://docs.nvidia.com/vgpu/20.0/grid-software-quick-start-guide/index.html

For the supported list of NVIDIA GPUs with vSphere 9.1 environment, see the NVIDIA and Broadcom compatibility list here: https://docs.nvidia.com/vgpu/latest/product-support-matrix/vmware-vsphere.html
https://compatibilityguide.broadcom.com/search?program=sptg&persona=live&column=partner&order=asc

In this validation, the NVIDIA vGPU license is imported from CLS service and then attached to the Guests VMs to have a valid licensed GPU on the Guest VMs. The following screenshot shows the CLS and DSL service instances are created using the Nvidia Enterprise account:

Procedure 1.    Install NVIDIA GPU Host Drivers using vLCM

Step 1.          Log into the NVIDIA License System using your NVIDIA Enterprise account and click Software Downloads.

Step 2.          Filter the list by platform name and platform version as shown below. Download the latest vGPU drivers (20.1) for VMware vSphere 9.0. Once downloaded unzip the file.

Step 3.          You can install the vGPU driver in all ESXi hosts using vSphere Lifecycle manager (vLC) or you can copy these files to each ESXi host manually and install vGPU drivers. This procedure details installing the drivers using vLC.

Step 4.          Log into the vCenter and click the three bar menu icon and select Lifecycle Manager. Click Action and Import Updates. Import the following two files into vLC:

You can view the imported files under vLC as shown below:

Step 5.          Edit the Cluster base image and include the previous two components in the base cluster image. Go to Inventory > ESXi Cluster > Updates. Click Edit and add the vGPU drivers under the Component section to the cluster image as shown below. Click Save.

Step 6.          When the vGPU driver components are added to the cluster image, all the ESXi hosts become incompatible. Select the required hosts that have GPUs installed and click REMEDIATE to install the drivers. vLC installs the drivers one by one and update all the hosts.

Step 7.          SSH into the ESXi host that have GPUs and run the following commands to ensure required vGPU drivers are installed and loaded. Make use the GPUs are visible using nvidia-smi tool. You can view the GPUs on the ESXi hosts by navigating to ESXi Host > Configure > Hardware > Graphics

Step 8.          The vGPU drivers automatically creates different GPU profiles which you can assign to the virtual machines. Create a virtual machine and click Add PCI Device and select the required profiles.

Procedure 2.    Install NVIDIA GPU Guest Drivers and Configure License in Guest VMs

Step 1.          The Client Configuration Token (.tok) from the CLS server. To generate the .tok file, click Licensing Server in the NLS portal and click Actions and click Generate client config token as shown below:

Step 2.          Set the expiration date for the vGPU License and click Download Client Configuration Token.

Step 3.          Copy the *.deb (guest gpu driver) and *.tok file to the guest VM.

Step 4.          Install the GPU driver and copy the .tok file to /etc/nvidia/ClientConfigToken/ directory in Linux hosts and in Windows Vms the location is C:\Program Files\NVIDIA Corporation\vGPU Licensing\ClientConfigToken. For Windows VM, install the guest driver using the *.exe file with simple clicks.

Step 5.          When the license is copied and nvidia-gridd service is restarted, you can see the license is picked up and displays an expiration date. Nvidia-smi tool now reports the virtual gpu details. The following screenshots show Ubuntu and Window guest VMs configured with GPUs running two different ESX hosts configured different GPU models:

The Guest vm license details are reflected in the CLS as shown below:

Step 6.          For live migration of VMs that are configured vGPUs from one ESXi node to other, enable vgpu.hotmigrate.enabled setting by going to vCenter > Configure > Advanced Settings > filter the settings with the phrase vgpu and check the box Enabled as shown below. Live migration is possible if the target host has the same GPU model and profile.

Memory Tiering with NVMe Disks

Memory Tiering over NVMe (Memory Tiering) allows you to extend memory capacity of an ESX host by using NVMe devices that are installed locally on the ESX host as tiered memory. Memory Tiering optimizes performance by intelligently allocating VM memory to either NVMe devices or faster dynamic random-access memory (DRAM) in the ESX host. With the Memory Tiering feature, workload capacity can be increased with a minimal impact on performance. This feature allows us to consolidate more workload VMs on the ESX host, thus better utilization of available CPU resources. For more details on memory tiering, go to: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-1/vsphere-resource-management/memory-tiering-over-nvme.html

With vSphere 9.1, Memory tiering can be performed at the ESXi cluster level when all the esxi host has similar hardware (NVMe disks must present on all the hosts) configuration. However, if the hosts have different hardware configuration, memory tiering can be configured at each host level. This section details the steps for configuring memory tiering at ESXi host level using ESXCLI.

Procedure 1.    Configure Memory Tiering on ESX host

Step 1.          SSH into the ESXi host where Memory Tiering needs to be configured.

Step 2.          Put the node into maintenance mode. List the eligible NVMe disks for Memory tiering by running the following commands. Create the Memory Tiering using the two nvme disks. The two nvme disks are mirrored with each other using ESXi software mirroring.

 

Step 3.          Confirm the memory tiering by running the following command:

Step 4.          When the memory tiering is configured, you can verify the vCenter by selecting the individual nodes.

 

Step 5.          As shown in the above screenshot, the ESXi host has around 384GB memory, and the memory is extended by 100% resulting in doubling the available memory size for allocation to the Virtual machines. You can create a VM and allocate memory higher than 384GB (or collectively from all the VMs) as shown below:

Secure VM Workloads with vSphere Native Security Features and AMD SEV-SNP

This chapter contains the following:

●     Virtual Trusted Platform and Native Key Provider for Data-at-Rest Encryption

This chapter describes two important VM encryption capabilities being offered by the FlashStack solution using vSphere 9.1 and AMD CPUs.

Virtual Trusted Platform (vTPM) and Native Key Provider (NKP) for Data-at-Rest Encryption

Virtual Trusted Platform Module (vTPM) and Native Key Provider (NKP) work together in VMware vSphere to deliver software-based data-at-rest encryption. NKP manages the underlying cryptographic keys natively within the hypervisor cluster, while vTPM provides a secure cryptographic processor implementation for individual virtual machines (VMs) to protect sensitive data like BitLocker keys, credentials, and secure boot measurements.

Built natively into vCenter, NKP manages cryptographic key completely local to the vCenter environment thus eliminating the infrastructure complexity of external Key management Systems (KMS). It also mandates the password protected backup of the keys before the keys become operational.

vTPM is a TPM 2.0 implementation and it can be added as device (as Trusted Platform Module device) to the VM that enables the workload VMs to store their keys and passwords in NKP and when it is added , it automatically encrypts the configuration files of the VM.

Procedure 1.    VM encryption using vTPM and NKP

Step 1.          Log into the vCenter and go to Configure > Security >  Key Provider and click Add. Provide a name for NKP and click ADD KEY PROVIDER.

Step 2.          Click Backup to backup the key Provider. Protect the backup with a password and complete the backup of Key Provider.

Step 3.          Edit the VM and add TPM module as shown below. When the vTPM is added, VM configuration files will be encrypted automatically. To encrypt the individual hard disks, expand each Hard disk and set VM Storage Policy to VM encryption Policy by selecting the option from the drop-down list.

Secure VM Workloads with AMD Secure Encrypted Virtualization-Secure Nested Paging

AMD Secure Encrypted Virtualization-Secure Nested Paging (SEV-SNP) builds on AMD SEV and AMD SEV-ES allowing the creation of confidential VMs, which are protected from higher-privileged software on the host, such as the hypervisor. SEV-SNP protects against memory integrity attacks by requiring that if a VM can read an encrypted location in memory, the value it reads must be equal to the value that it last wrote to that memory location. For more information on SEV-SNP, see Strengthening VM isolation with Integrity  Protection and more.

For more details on the prerequisites and limitations associated with AMD SEV-SNP, go to: https://techdocs.broadcom.com/us/en/vmware-cis/acc/advanced-cyber-compliance/9-1/confidential-computing/securing-virtual-machines-with-amd-secure-encrypted-virtualization-secure-nested-paging.html

Procedure 1.    Secure VM Workloads

To enable the confidential computing on the workload VM, ensure the required BIOS tokens, as explained in Table 27, are used in the BIOS policy and applied to the ESXI host.

Step 1.          To configure the confidential computing on a workload VM, power off the VM, right-click VM and select Edit Setting and then go to VM options and expand Confidential Compute option.

Step 2.          Select SEV-SNP from the Confidential VM mode drop-down list as shown below:

About the authors

Gopu Narasimha Reddy, Technical Marketing Engineer, Cisco Systems, Inc.

Gopu is a Technical Marketing engineer with the UCS Solutions team at Cisco. He is currently focused on validating and developing Cisco UCS infrastructure solutions for enterprise workloads with different operating environments including Windows, Nutanix, VMware, Linux, and OpenShift. Gopu is also involved in publishing database benchmarks on Cisco UCS servers. His areas of interest include building and validating reference architectures, and development of sizing tools in addition to assisting customers in database deployments.

Gaurav Nigam, Senior Solutions Architecture, Everpure, Inc.

Gaurav is a Senior Solutions Architect at Everpure, serving on the Solutions team. He brings extensive expertise in architecting, developing, and optimizing enterprise solutions across storage, converged and hyperconverged infrastructure, networking, cloud technologies. His work focuses on establishing best practices, driving automation , and developing high-quality technical content that enables customers and teams to achieve successful outcomes. Throughout his career, Gaurav has worked with leading technology organizations, including EMC², VMware, and Broadcom. He also holds several industry-recognized certifications, including the Cisco Certified Internetwork Expert (CCIE) Data Center and VMware Certified Cloud Foundation Architect certification.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

●     Chris O’Brien, Senior Director, Technical Marketing, Cisco Systems, Inc.

●     John George, Technical Marketing, Cisco Systems, Inc.

●     Craig Waters, Solutions Director, Everpure, Inc.

Appendix

This appendix contains the following:

●     Appendix A – References used in this guide

Appendix A – References used in this guide

Compute

Cisco Intersight: https://www.intersight.com

Cisco Intersight Managed Mode: https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Intersight_Managed_Mode_Configuration_Guide.html

Cisco Unified Computing System: http://www.cisco.com/en/US/products/ps10265/index.html

Cisco UCS M8 AMD Servers: https://www.cisco.com/c/m/en_us/solutions/computing/ucs-amd.html#~models

Cisco UCS 6600 Fabric Interconnect Data Sheet: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6600-series-fabric-interconnect-ds.html

Network

Cisco Nexus 9300-GX Series Switches: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/nexus-9300-gx-series-switches-ds.html

Pure Storage

FlashStack: https://flashstack.com

Everpure FlashArray//X: https://www.everpuredata.com/content/dam/pdf/en/datasheets/ds-flasharray-x.pdf

Everpure FlashArray//XL: https://www.everpuredata.com/content/dam/pdf/en/datasheets/ds-flasharray-xl.pdf

Broadcom VMware vSphere

VVF: https://www.vmware.com/products/cloud-infrastructure/vsphere-foundation

Interoperability Matrix

Cisco UCS Hardware Compatibility Matrix: https://ucshcltool.cloudapps.cisco.com/public/ 

Everpure FlashStack Compatibility Matrix

This interoperability list will require a support login from Pure: https://support.everpuredata.com/r/product-information/flashstack-compatibility-matrix

Purity FA interoperability Matrix : https://support.everpuredata.com/r/flasharray-release/purityfa-interoperability-matrix

Broadcom VMware

Broadcom VMWare Compatibility Matrix: https://compatibilityguide.broadcom.com/

Broadcom Systems/Server compatibility matric for VVF and VCF https://compatibilityguide.broadcom.com/search?program=server&persona=live&column=partnerName&order=asc

CVD Program

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS X-Series, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trade-marks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. (LDW_P2)

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)