AI Infrastructure with Cisco Nexus 9000 Switches

 

Introduction. 3

Hardware. 4

Cisco UCS C885A M8 Rack Server. 4

Cisco Nexus 93108TC-FX3 switch. 5

Cisco Nexus 9332D-GX2B switch. 5

Cisco Nexus 9364E-SG2 switch. 5

Cisco UCS C225 M8 Rack Server. 6

Networking topologies. 6

Topology for Cisco UCS C885A M8 Rack Servers with NVIDIA HGX.. 7

Cluster fabric sizing tables. 11

Optics and cables. 12

Storage architecture. 12

Software. 13

Cisco NX-OS and Nexus Dashboard. 13

NVIDIA AI Enterprise. 14

Compute server stack. 14

NVIDIA Spectrum-X.. 14

Security. 14

Testing and certification. 15

Summary. 15

Appendix A – Compute server specifications. 15

Appendix B – Control-node server specifications. 16

References. 16

Document history. 17

Featuring Cisco UCS^® C885A M8 Rack Servers with NVIDIA HGX^™ H200 and NVIDIA Spectrum^™-X

Introduction

Cisco^® Enterprise Reference Architecture (ERA) is based on Cisco Nexus^® 9000 Series Switches for networking AI clusters managed by the on-premises Cisco Nexus Dashboard platform. It adheres to the NVIDIA Enterprise Reference Architecture for NVIDIA HGX H200 with NVIDIA Spectrum-X networking.

Cisco Nexus 9000 Series Switches, powered by Cisco Silicon One^® and Cisco Cloud Scale architectures, provide high-speed, deterministic, low-latency, and power-efficient connectivity for AI and high-performance computing (HPC) workloads. With the availability of multiple form-factors, optics, and rich software features of the Cisco NX-OS operating system, Nexus 9000 switches provide a consistent experience for frontend, storage, backend, and out-of-band (OOB) management networks (see Figure 1).

Cisco Nexus Dashboard is the operations and automation platform for managing the Nexus 9000 switch–based fabrics. It complements the data plane features of the Nexus 9000 switches by simplifying their configuration using built-in templates. It detects network health issues, such as congestion, bit errors, and traffic bursts, in real time and automatically flags them as anomalies. These issues can be resolved faster using integrations with commonly used tools, such as ServiceNow and Ansible, allowing the networks of an AI cluster to be aligned with the existing workflows of an organization.

Cisco Nexus 9000 Series Switches for networking the AI clusters, managed by Nexus Dashboard Platform

Hardware

Cisco UCS C885A M8 Rack Server

The Cisco UCS C885A M8 Rack Server is an 8RU, dense GPU server (node) that delivers massive, scalable performance for AI workloads such as Large Language Model (LLM) training, fine-tuning, inferencing, and Retrieval Augmented Generation (RAG). It is based on the NVIDIA HGX reference architecture in 2-8-10-400 (C-G-N-B) configuration where the C-G-N-B naming convention is defined as:

●     C: number of CPUs in the node

●     G: number of GPUs in the node

●     N: number of network adapters (NICs), categorized into:

◦    North-south: communication between nodes and external systems through the frontend network.

◦    East-west: communication within the cluster through the backend network.

●     B: average network bandwidth per GPU in gigabits per second (Gb/s)

The 8x NVIDIA H200 SXM GPUs within the server are interconnected using high-speed NVLink interconnects. GPU connectivity to other physical servers is performed using 8x NVIDIA BlueField-3 B3140H SuperNICs for East-West (E-W) traffic. For North-South (N-S) traffic, 1x NVIDIA BlueField-3 B3220 DPU NIC (in 2x200G mode) is available. Each server contains 2x AMD EPYC CPUs, up to 3 TB of DDR DRAM, 30 TB of NVMe local storage, and hot swappable fan trays and power supplies. Detailed specifications of the server are captured in Appendix A.

Cisco UCS C885A M8 Rack Server with NVIDIA HGX

This ERA applies to any other server based on NVIDIA HGX in the 2-8-10-400 (C-G-N-B) configuration, although Cisco UCS C885A M8 Rack Server has been used for illustration.

Cisco Nexus 93108TC-FX3 switch

The Cisco Nexus 93108TC-FX3 switch (see Figure 3) provides 48 100-Mbps or 1/10-Gbps 10GBASE-T ports and six 1/10/25/40/100-Gbps QSFP28 ports in 1RU form-factor. This switch can be used in a management network.

Cisco Nexus 93108TC-FX3 switch

Cisco Nexus 9332D-GX2B switch

The Cisco Nexus 9332D-GX2B switch (see Figure 4) provides 32 400G QSFP-DD ports with 10/25/50/100/200-Gbps breakout support in 1RU form-factor. This switch can be used in a leaf or spine role.

Cisco Nexus 9332D-GX2B switch

Cisco Nexus 9364E-SG2 switch

The Cisco Nexus 9364E-SG2 switch (see Figure 5) provides 64 800G ports or 128 400G ports in 2RU form factor, available in both QSFP-DD and OSFP port types. This switch can be used in a leaf or spine role.

Cisco Nexus 9364E-SG2 switch

Cisco UCS C225 M8 Rack Server

The Cisco C225 M8 Rack Server is a 1RU general-purpose server that can be used in many roles, such as application server, support server, control nodes for Kubernetes (K8s) and Slurm, or even for a storage platform. For example, these servers are also used to run the VAST storage solution as described in the “Storage architecture” section.

Cisco UCS C225 M8 Rack Server

Networking topologies

Cisco Nexus 9000 Series Switches can be used for various types of data-center fabric designs. However, only the designs aligning closely to the NVIDIA ERA are considered part of this reference architecture.

Like the NVIDIA ERA, this Cisco ERA allows scale up to 128 NVIDIA eight-GPU HGX nodes for a total of 1024 H200 GPUs using a modular design of four-node Scalable Unit (SU), scalable up to 32 SUs to accommodate 128 nodes. Each SU is a discrete computation entity tied to the port availability size of the network devices. SUs can be replicated to adjust the scale of the environment with more ease.

As noted below, minor modifications are made to accommodate specific aspects of the Cisco design:

●     Cisco UCS C885A M8 Rack Servers x86 management ports uses a speed of 10G instead of 1G.

●     The VAST storage solution requires a minimum of 8x400G links to the storage network, although this requirement may change if a solution from a different NVIDIA-certified storage partner is used.

●     The BMC ports of NVIDIA BlueField-3 SuperNICs are not connected, and they will be managed from an x86 host. However, the BMC ports of NVIDIA BlueField-3 DPUs are connected.

Topology for Cisco UCS C885A M8 Rack Servers with NVIDIA HGX

Figure 7 shows the cluster topology for up to 12 Cisco UCS C885A M8 Rack Servers with NVIDIA HGX, grouped in three SUs. At this small scale, a pair of Cisco Nexus 9364E-SG2 switches provide enough port density to converge frontend, backend, storage, and management network. This convergence increases affordability and simplicity while maintaining logical separation using VLANs.

 

Enterprise RA for 12 Cisco UCS C885A M8 Rack Servers with NVIDIA HGX (96 GPUs)

Table 1 shows the Bill of Materials (BOM) for a 12-node Cisco UCS C885A M8 Rack Server cluster with NVIDIA HGX.

Table 1.        BOM for a 12-node Cisco UCS C885A Rack Server cluster with NVIDIA HGX (96 GPUs)

 

Nexus 9364E-SG2-O switches use OSFP-800G-DR8 twin-port transceivers (see Figure 8) with 2x400G MPO-12 connectors for 800G to 400G connections.

Cisco OSFP-800G-DR8 transceiver module

Each connection independently supports 400G without the need for breakout cables (see Figure 9).

Cisco OSFP-800G-DR8 plughole view

Figure 10 shows the cluster topology for up to 16 Cisco UCS C885A M8 Rack Servers with NVIDIA HGX. The E-W network is rail-aligned with Rails 1 to 4 falling on the left E-W spine and Rails 5 to 8 on the right E-W spine.

 

Enterprise RA for 16 Cisco UCS C885A M8 Rack Servers with NVIDIA HGX (128 GPUs)

Table 2 shows the BOM for a 16-node Cisco UCS C885A M8 Rack Server cluster with NVIDIA HGX.

Table 2.        BOM for a 16-node Cisco UCS C885A M8 Rack Server cluster with NVIDIA HGX (128 GPUs)

For a cluster size greater than 16, the east-west compute network will expand into a spine-leaf fabric. For the largest cluster sizes, the north-south network will also be spine-leaf, as shown in Figure 11 of the 128 Cisco UCS C885A M8 Rack Server cluster. The E-W network is rail-aligned with each rail 1 to 8 falling on each E-W leaf 1 to 8.

 

Enterprise RA for 128 Cisco UCS C885A M8 Rack Servers with NVIDIA HGX (1024 GPUs)

Cluster fabric sizing tables

Sizing with Cisco UCS C885A M8 Rack Server

Tables 3 and 4 show the quantity of different units required for different cluster sizes using Cisco UCS C885A M8 Rack Server with NVIDIA HGX, eight E-W B3140H NVIDIA BlueField-3 SuperNICs, and two N-S B3240 NVIDIA BlueField-3 DPU NICs.

Table 3.        East-west compute fabric table for Cisco UCS C885A M8 Rack Server with NVIDIA HGX –switch, transceivers, and cable counts

Table 4.        North-south fabric table for Cisco UCS C885A M8 Rack Server with NVIDIA HGX - switch, transceivers, and cable counts

Optics and cables

In the first phase, the optics and cables listed in Table 5 are used in various devices in the system.

Table 5.        Supported list of optics and cables in various devices

Storage architecture

Cisco has partnered with VAST Data to onboard their storage software on Cisco UCS C225 M8 Rack Servers in EBOX architecture. VAST Data supports a “Distributed And Shared Everything” (DASE) architecture that allows for horizontally scaling storage capacity and read/write performance by incrementally adding servers. To support all stages of an AI data pipeline, all protocol servers, such as NFS, S3, and SMB, are enabled.

Figure 12 shows the overall network connectivity of storage servers and the BOM for a single EBOX with two storage leafs. For Data Path, each server uses two NVIDIA BlueField-3 B3220L 2x200G NICs: NIC0 is used for internal network within the servers, allowing any server to access storage drives from any other server, and NIC1 is used for the external network, supporting client traffic such as NFS, S3, and SMB. Note: The internal network traffic is switched locally at the leaf (it never goes to the spine) because every server connects to every leaf. For client-facing external traffic, per EBOX, the minimum requirement over the spine is 11x 200G or 6x 400G. The 1G BMC and 10G x86 management ports are connected to a management leaf switch.

Block diagram and BOM of storage subsystem

As the cluster size increases, the number of storage leaf switches and EBOX will linearly increase, as captured in the cluster-sizing tables.

Besides VAST Data, other NVIDIA-certified storage partners can also be used in this reference architecture.

Software

Cisco NX-OS and Nexus Dashboard

As mentioned earlier, Cisco NX-OS is the operating system running on Nexus 9000 Series Switches, whereas Nexus Dashboard is the operations and automation platform for managing the fabrics.

Cisco NX-OS and Nexus Dashboard will not:

●     Configure compute or storage in any way

●     Manage the BMC or host CPU software lifecycle of servers

●     Manage the kernel and distribution on the NVIDIA BlueField-3 NICs.

Server configuration and management functions must be managed through some other means (Cisco Intersight^® is an option). The customer will be solely responsible for deploying and using these tools. In addition to being the appropriate scope for a network controller, this separation of concerns aligns with the dominant operational paradigm that segments network operations from compute and storage.

NVIDIA AI Enterprise

This reference architecture includes NVIDIA AI Enterprise, deployed and supported on NVIDIA-certified Cisco UCS C885A M8 Rack Servers with NVIDIA HGX NVIDIA AI Enterprise is a cloud-native software platform that streamlines development and deployment of production-grade AI solutions, including AI agents, generative AI, computer vision, speech AI, and more. Enterprise-grade security, support, and API stability ensure a smooth transition from prototype to production.

NVIDIA NIM^™ microservices provide a complete inference stack for production deployment of open-source community models, custom models, and NVIDIA AI Foundation models. Their scalable, optimized inference engine and ease of use accelerate models, improve TCO, and make production deployment faster.

Compute server stack

The entire cluster solution has been verified with compute nodes running Ubuntu Linux 22.04 LTS and NVIDIA Cloud Native Stack (CNS) version 12.3, which includes the compatible drivers, GPU and network operators within Kubernetes (K8s) environment. Slurm version 24.11.1 has been verified as a workload orchestration engine. Containers under NVIDIA NGC^™ catalog can be launched both with Kubernetes and with Slurm.

NVIDIA Spectrum-X

The NVIDIA Spectrum-X Networking technology significantly improves the performance and efficiency of Ethernet-based GPU and storage networks. Its benefits are available with Cisco Nexus 9000 switches running NX-OS 10.6(1)F onwards when connected to NVIDIA BlueField-3 SuperNICs. Fine Grain Load Balancing (FGLB) license is needed in addition to DCN Advantage License to enable NVIDIA Spectrum-X on Cisco Nexus 9000 switches.

Compute server software stack

Customers can run their choice of OS distribution and software versions as per NVIDIA AI Enterprise, drivers, and the CNS compatibility matrix published by NVIDIA.

Security

Cisco network security and observability services can be optionally integrated in different hardware (switches, hosts, NICs) and software components of the cluster.

Testing and certification

The overall solution has been thoroughly tested considering all aspects of management plane, control plane, and data plane combining together compute, storage, and networking. A number of benchmark test suites such as HPC Benchmark, IB PerfTest, NCCL Test, MLCommons Training, and Inference benchmarks have also been run to evaluate performance and assist with tuning. Different elements and entities of the NVIDIA AI Enterprise ecosystem have been brought up and tested to evaluate a number of enterprise-centric customer use cases around fine-tuning, inferencing, and RAG. Results from running NVIDIA-Certified Systems Test Suite version 3.5 for both single-node and multi-node with networking have passed for Cisco UCS C885A M8 Rack Servers.

Summary

Cisco Nexus 9000 Series Switches and the Nexus Dashboard platform provide scalable, easy-to-manage, and high-performance networking for AI Infrastructure powered by NVIDIA-accelerated computing .

Appendix A – Compute server specifications

Table 6 provides the specification of the Part# UCSC-885A-M8-H12 for Cisco UCS C885A M8 Rack Server used as GPU node in this ERA.

Table 6.        Cisco UCS C885A M8 8RU rack server

Appendix B – Control-node server specifications

The versatile Cisco UCS C225 M8 1RU rack server can be used as a support server (also called a control-node server) for Slurm and Kubernetes (K8s), etc. Table 7 shows the minimum specifications of the server.

Table 7.        Cisco UCS C225 M8 1RU rack server

Deployments looking for 2-socket CPUs can use the Cisco UCS C245 M8 2RU rack server variant along with B3220 DPU NICs.

References

●     NVIDIA AI Enterprise Software Reference Architecture:
https://docs.nvidia.com/ai-enterprise/reference-architecture/latest/index.html

●     NVIDIA HGX:https://www.nvidia.com/en-us/data-center/hgx

●     NVIDIA Spectrum-X networking: https://www.nvidia.com/en-us/networking/spectrumx/

●     Cisco Nexus 9000 Series Switches:
https://www.cisco.com/site/us/en/products/networking/cloud-networking-switches/nexus-9000-switches/index.html

●     Cisco Nexus Dashboard:
https://www.cisco.com/site/us/en/products/networking/cloud-networking/nexus-platform/index.html

●     Cisco NX-OS Licensing Options Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/licensing-options/cisco-nexus-licensing-options-guide.html

Document history