Cisco Nexus Hyperfabric Enterprise RA

Contents

Introduction. 3

Hardware. 4

GB300 NVL72 Rack. 4

Cisco HF6100-64ED.. 5

Cisco N9164E-NS4-O.. 5

Cisco HF6100-32D.. 5

Cisco HF6100-60L4D.. 5

Cisco UCS C225 M8 Rack Server. 6

Cisco Optics and cables. 6

Networking topologies. 6

Overview.. 6

Compute (Node East-West) Network. 7

Converged (Node North-South) Network. 7

Out-of-Band Management Network. 8

Topology with 2 compute racks (144 GPUs) 8

Topology with 4 compute racks (288 GPUs) 9

Topology with 8 compute racks (576 GPUs) 10

Topology with 16 compute racks (1152 GPUs) 11

Cluster BOM.. 12

High-Performance Storage. 15

Software. 16

Network Controller. 16

Compute Controller. 16

Storage Controller. 16

NVIDIA AI Enterprise. 16

NVIDIA Spectrum-X.. 16

Security. 17

Observability. 17

Testing and certification. 18

Summary. 18

Appendix A – Control-node server specifications. 19

Introduction

Cisco Nexus^® Hyperfabric Full Stack AI Infrastructure is an on-premises AI cluster that is managed by a cloud-hosted controller. It empowers and simplifies your AI initiatives and accelerates AI deployments with a comprehensive, integrated, cloud-managed solution. This reference architecture (RA) adheres to the NVIDIA Enterprise Reference Architecture (Enterprise RA) for NVIDIA GB300 NVL72 with up to 1152 GPU scale. Figure 1 shows the key components of the solution. The hardware components used in the cluster are described in the next section.

Key components of Cisco Nexus Hyperfabric Full Stack AI Infrastructure

Hardware

GB300 NVL72 Rack

This RA uses NVIDIA-Certified^™ GB300 NVL72 racks where each rack consists of 18 compute trays and 9 NVL5 switch trays. Each compute tray is in 2-4-5-800 (C-G-N-B) configuration where C-G-N-B naming convention is defined as:

●     C: Number of CPUs in the node.

●     G: Number of GPUs in the node.

●     N: Number of network adapters (NICs), categorized into:

◦    North/South: Communication between nodes and external systems.

◦    East/West: Communication within the cluster.

●     B: Average network bandwidth per GPU in Gigabits per second (GbE).

Within each compute tray, there are 4x NVIDIA B300 GPUs paired with 2x Grace CPUs – one Grace CPU and two B300 GPUs together form a GB300 SuperChip. Each NVL5 switch tray consists of 2 NVSwitch ASICs per tray connecting to every GPU within the rack using NVL5 links. The 72 GPUs via the 18 NVSwitch ASICs together form a single coherent memory domain using the scale up network within the rack. Across the racks, GPU connectivity from every compute tray to other servers is via the use of 4x integrated NVIDIA ConnectX^®-8 SuperNICs for East-West scale out traffic and via 1x NVIDIA BlueField^®-3 B3240 DPU for North-South traffic. Supermicro SRS-GB300-NVL72 is an NVIDIA-Certified^™ GB300 NVL72 rack supported in this RA.

Supermicro SRS-GB300-NVL72 Rack

Cisco HF6100-64ED

The Cisco HF6100-64ED is a Cisco Silicon One^™ Ethernet switch ASIC based 2RU switch supporting 64 800G OSFP modules allowing 64 800GE or 128 400GE ports. It will be used in both leaf and spine roles.

Cisco HF6100-64ED switch

Cisco N9164E-NS4-O

The Cisco N9164E-NS4-O is a NVIDIA Spectrum^™-4 Ethernet switch ASIC based 2RU switch supporting 64 800G OSFP modules allowing 64 800GE or 128 400GE ports. This switch can be used in both leaf and spine role in East-West compute network as an alternative to HF6100-64ED switch.

Cisco N9164E-NS4-O switch

Cisco HF6100-32D

The Cisco HF6100-32D is a 1RU Silicon One NPU–based high-density 400G-port–capable switch supporting 32 ports of QSFPDD with breakout support. This switch can be used in the leaf or spine role depending on the requirements of the cluster.

Cisco HF6100-32D switch

Cisco HF6100-60L4D

The Cisco HF6100-60L4D is a 1RU Silicon One NPU–based high-density switch supporting 60 SFP56 ports capable of 1/10/25/50GE speeds, plus 4 ports of 400 QSFPDD with breakout support. This switch is used in many different roles, such as management network, connectivity to applications and support servers, etc.

Cisco HF6100-60L4D switch

Cisco UCS C225 M8 Rack Server

The Cisco UCS C225 M8 Rack Server is a 1RU general-purpose server that can be used in many roles, such as application server, support server, control nodes for Kubernetes (K8s) and Slurm. In this RA, these servers are also used to run the VAST storage solution as described in the “High-Performance Storage” section.

Cisco UCS C225 M8 Rack Server

Cisco Optics and cables

The following Cisco optics and cables shown in Table 1 are being used on the listed devices.

Table 1.        Supported list of optics and cables on different devices

Networking topologies

Overview

Overall, the networking topology is split into three separate fabrics:

●     Compute (Node East-West) Network

●     Converged (Node North-South) Network

●     Out-of-Band (OOB) Management Network

Compute (Node East-West) Network

This network is meant for communication between the GPUs via ConnectX-8 SuperNICs configured in 2 x 400G mode. Both single and dual plane fabrics are supported though dual plane fabric is recommended to avoid a single point of failure. In case of single plane, only 1 400G port per ConnectX-8 SuperNIC is used. However, in case of dual planes, 1 400G port per ConnectX-8 SuperNIC is connected to each of the two planes. Each GB300 NVL72 rack has 18 compute trays and each compute tray has 4 GPUs paired with 4 ConnectX-8 SuperNICs on a one-to-one basis representing 4 Rails 1, 2, 3, 4. This requires connecting 72 400G ports to plane 1 and 72 400G ports to plane 2 per rack split over 4 rail groups. Table 2, 3 shows the quantity of different units required for building a single and dual plane compute network with compute rack count ranging from 2 to 16.

Table 2.        Single plane East-West compute fabric table – switch, transceivers, and cable counts

 

Table 3.        Dual plane East-West compute fabric table – switch, transceivers, and cable counts

 

Converged (Node North-South) Network

The converged North-South network is used for communication with compute, storage, in-band management, support, and end-customer connections. It connects each compute rack using 36 400G ports, over 18 compute trays, to two separate switches providing redundancy and high storage throughput. Each rack has 2 top-of-rack (ToR) switches each with 2 100G uplinks for redundancy and connect to the converged network. Enough storage facing ports are pre-allocated ensuring 12.5 gbps of network bandwidth per GPU. A total of 48 200G ports (12 800G ports) are also reserved for 12 support servers to allow running Slurm and Kubernetes control nodes, NVIDIA Base Command Manager^™ head nodes, NVIDIA Mission Control^™, NVIDIA NVLink Management Software Manager, and additional control monitoring applications.

Table 4 shows the quantity of different units required for building a converged network considering different cluster sizes.

Table 4.        Converged North-South fabric table - switch, transceivers, and cable counts

 

Out-of-Band Management Network

The OOB Management network is primarily used for node management connecting to the ToR switches inside the compute racks allowing access to the Base Management Controller (BMC) and host management ports of the different entities inside the compute rack and storage servers. The 100G uplinks of ToR switches inside the compute rack and 400G uplinks of OOB management leaf switches are connected to converged North-South spine switches.

Additionally, the 1G management ports of the networking switches need to be connected separately to allow for their configuration and monitoring.

Topology with 2 compute racks (144 GPUs)

Figure 8 shows the overall cluster topology interconnecting 2 compute racks with a total of 36 compute trays and 144 GPUs. Each plane of the compute network uses two spine and four leaf HF6100-64ED or N9164E-NS4-O switches. The four-leaf switches represent Rail 1, 2, 3, 4. The 4 800G NVIDIA ConnectX-8 SuperNICs on each compute tray are each plugged with OSFPR-800G-DR8 modules allowing 2 400G ports per NIC where the two ports connect to two different planes. With 2 compute racks consisting of 36 compute trays, there are a total of 144 800G or 288 400G ports, connecting 144 400G ports to compute network plane1 and 144 400G ports to compute network plane2. Within a plane, 36 400G ports connect to every leaf switch.

The converged North-South network uses a pair of HF6100-64ED switches. The 2 400G NVIDIA Bluefield^®-3 DPU ports per compute tray of every rack connect to two different switches for redundancy. With 2 compute racks consisting of 36 compute trays, there are a total of 72 400G DPU ports evenly split between switch#1 and switch#2.

A total of 12 Cisco EBox nodes are used as high-performance storage connected to 2 HF6100-32D storage leaf switches for a total of 48 200G downlinks and 8 400G uplinks evenly split between the two leaf switches.

The 2 compute rack’s ToR switches use a total of 8 100G uplinks connected to the converged network. Each Cisco EBox also requires 1 10G host management port and 1 1G server BMC port connected to a HF6100-60L4D OOB management leaf switch.

Topology with 2 compute racks (144 GPUs)

Topology with 4 compute racks (288 GPUs)

Figure 9 shows the overall cluster topology interconnecting 4 compute racks with a total of 72 compute trays and 288 GPUs. Each plane of the compute network uses four spine and eight leaf HF6100-64ED or N9164E-NS4-O switches. The eight-leaf switches are divided into Rail 1, 2, 3, 4 with two leaf switches per rail-group. The 4 800G NVIDIA ConnectX-8 SuperNICs on each compute tray are each plugged with OSFPR-800G-DR8 modules allowing 2 400G ports per NIC where the two ports connect to two different planes. With 4 compute racks consisting of 72 compute trays, there are a total of 288 800G or 576 400G ports, connecting 288 400G ports to compute network plane1 and 288 400G ports to compute network plane2. Within a plane, 36 400G ports connect to every leaf switch.

The converged North-South network uses two spine and four leaf HF6100-64ED switches. The 2 400G NVIDIA Bluefield^®-3 DPU ports per compute tray of every rack connect to two different switches for redundancy. With 4 compute racks consisting of 72 compute trays, there are a total of 144 400G DPU ports evenly split between four leaf switches.

A total of 12 Cisco EBox nodes are used as high-performance storage connected to 2 HF6100-32D storage leaf switches for a total of 48 200G downlinks and 12 400G uplinks evenly split between the two leaf switches.

The 4 compute rack’s ToR switches use a total of 16 100G uplinks connected to the converged network. Each Cisco EBox also requires 1 10G host management port and 1 1G server BMC port connected to a HF6100-60L4D OOB management leaf switch.

Topology with 4 compute racks (288 GPUs)

Topology with 8 compute racks (576 GPUs)

Figure 10 shows the overall cluster topology interconnecting 8 compute racks with a total of 144 compute trays and 576 GPUs. Each plane of the compute network uses six spine and sixteen leaf HF6100-64ED or N9164E-NS4-O switches. The sixteen-leaf switches are divided into Rail 1, 2, 3, 4 with four leaf switches per rail-group. The 4 800G NVIDIA ConnectX-8 SuperNICs on each compute tray are each plugged with OSFPR-800G-DR8 modules allowing 2 400G ports per NIC where the two ports connect to two different planes. With 8 compute racks consisting of 144 compute trays, there are a total of 576 800G or 1152 400G ports, connecting 576 400G ports to compute network plane1 and 576 400G ports to compute network plane2. Within a plane, 36 400G ports connect to every leaf switch.

The converged North-South network uses four spine and eight leaf HF6100-64ED switches. The 2 400G NVIDIA Bluefield^®-3 DPU ports per compute tray of every rack connect to two different switches for redundancy. With 8 compute racks consisting of 144 compute trays, there are a total of 288 400G DPU ports evenly split between eight leaf switches.

A total of 12 Cisco EBox nodes are used as high-performance storage connected to 2 HF6100-32D storage leaf switches for a total of 48 200G downlinks and 20 400G uplinks evenly split between the two leaf switches. The number of Cisco EBox nodes can be increased to 20 for higher storage throughput and/or capacity with the use of QDD-400G-SR8 optics on storage leaf downlinks in 2 x 200G breakout mode.

The 8 compute rack’s ToR switches use a total of 32 100G uplinks connected to the converged network. Each Cisco EBox also requires 1 10G host management port and 1 1G server BMC port connected to a HF6100-60L4D OOB management leaf switch.

Topology with 8 compute racks (576 GPUs)

Topology with 16 compute racks (1152 GPUs)

Figure 11 shows the overall cluster topology interconnecting 16 compute racks with a total of 288 compute trays and 1152 GPUs. Each plane of the compute network uses nine spine and thirty-two leaf HF6100-64ED or N9164E-NS4-O switches. The thirty-two leaf switches are divided into Rail 1, 2, 3, 4 with eight leaf switches per rail-group. The 4 800G NVIDIA ConnectX-8 SuperNICs on each compute tray are each plugged with OSFPR-800G-DR8 modules allowing 2 400G ports per NIC where the two ports connect to two different planes. With 16 compute racks consisting of 288 compute trays, there are a total of 1152 800G or 2304 400G ports, connecting 1152 400G ports to compute network plane1 and 1152 400G ports to compute network plane2. Within a plane, 36 400G ports connect to every leaf switch.

The converged North-South network uses four spine and eight leaf HF6100-64ED switches. The 2 400G NVIDIA Bluefield^®-3 DPU ports per compute tray of every rack connect to two different switches for redundancy. With 16 compute racks consisting of 288 compute trays, there are a total of 576 400G DPU ports evenly split between eight leaf switches. Unlike prior topologies, each leaf has a 2:1 oversubscription with 72 400G downlinks and 36 400G uplinks.

A total of 12 Cisco EBox nodes are used as high-performance storage connected to 2 HF6100-32D storage leaf switches for a total of 48 200G downlinks and 32 400G uplinks evenly split between the two leaf switches. An additional pair of storage leaf switches with 32 400G uplinks are pre-provisioned to allow future expansion of high-performance storage servers. The 64 400G uplinks allow 12.5 gbps network bandwidth per GPU with the remaining network bandwidth meant for inter-storage server communication across the two pairs of storage leaf switches. The number of Cisco EBox nodes can be increased to 32 for higher storage throughput and/or capacity with the use of QDD-400G-SR8 optics on storage leaf downlinks in 2 x 200G breakout mode.

The 16 compute rack’s ToR switches use a total of 64 100G uplinks connected to the converged network. Each Cisco EBox also requires 1 10G host management port and 1 1G server BMC port connected to a HF6100-60L4D OOB management leaf switch.

Topology with 16 compute racks (1152 GPUs)

Cluster BOM

The BOM for clusters with varying compute racks (GPUs) using single compute plane is shown in Table 5.

Table 5.        BOM of clusters with GPU scale 144 to 1152 using single compute plane

 

The BOM for clusters with varying compute racks (GPUs) using dual compute plane is shown in Table 6.

Table 6.        BOM of clusters with GPU scale 144 to 1152 using dual compute plane

 

High-Performance Storage

Cisco has partnered with VAST Data to onboard their AI OS on Cisco UCS C225-M8N Rack Servers in EBox architecture: together, they provide the storage subsystem for this RA. This product is called Cisco EBox and it is NVIDIA-Certified high-performance storage for both NVIDIA and Cisco Enterprise Secure AI Factory. VAST Data supports a “Disaggregated Shared Everything” (DASE) architecture that allows for horizontally scaling storage capacity and read/write performance by incrementally adding servers to a single namespace. Additional features include native support for multitenancy, multiprotocol (NFS, S3, SMB), data reduction, data protection, cluster high availability, serviceability of failed hardware components etc.

Figure 12 shows the overall network connectivity and BOM with 12 storage servers. For data path, each server uses two NVIDIA BlueField-3 B3220L 2x200G SuperNICs – NIC0 is used for internal network within the servers, allowing any server to access storage drives from any other server, and NIC1 is used for external network supporting client traffic such as NFS, S3, and SMB. The 1G BMC and 10G x86 management ports are connected to a management leaf switch.

Block diagram and BOM of storage sub-system

Beside Cisco EBox, other NVIDIA-Certified storage partners can also be used in this reference architecture.

Software

To deploy and manage a high-scale AI cluster, a robust software stack is required with an automation-first approach. The use of controllers along with their programmability interfaces can tremendously simplify day-0 resource provisioning, day-1 configuration, and day-N operationalization. The following sub-sections cover the key software components involved in this reference architecture.

Network Controller

Cisco Nexus^® Hyperfabric controller is required to provision and manage the entire networking fabric. It fully manages the configuration target state, switch software versions etc. It ingests telemetry from switches and NICs on compute and storage nodes for end-to-end network visibility and optimization of network performance. Enteprises can also utilize the available programmability interfaces to fully integrate with the controller via their automation frameworks.

Compute Controller

NVIDIA Mission Control^™ paired with NVIDIA Base Command Manager and NVLink Management software manager are essential for deploying and managing GB300 NVL72 systems, providing centralized provisioning, configuration, autonomous hardware recovery, and deep observability. They automate firmware updates, monitor liquid cooling, and manage the high-speed NVLink fabric, maximizing uptime for the overall compute infrastructure.

Storage Controller

The Cisco EBox storage controller (also known as VAST Management Service) will be used for configuration and monitoring of the high-performance storage. RESTful APIs are supported for integration with automation frameworks.

NVIDIA AI Enterprise

This reference architecture includes NVIDIA AI Enterprise, deployed and supported on NVIDIA-Certified Supermicro GB300 NVL72 racks. NVIDIA AI Enterprise brings together optimized microservices, frameworks and libraries for AI development with advanced GPU orchestration and infrastructure management into a production-grade software suite. It enables the deployment of leading open-source tools and AI models, lowers infrastructure costs and reduces time to market while ensuring reliable, secure, and scalable operations.

NVIDIA NIM^™ gives developers ready-to-run containers and standard APIs for serving AI models on NVIDIA GPUs.

NVIDIA Spectrum-X

The NVIDIA Spectrum^™-X Networking technology significantly improves the performance and efficiency of Ethernet-based GPU and storage networks. Its benefits are available with Cisco Nexus Hyperfabric switches when connected to NVIDIA ConnectX-8 and BlueField-3 SuperNICs and Fine Grain Load Balancing (FGLB) license enabled. Spectrum-X License is not required when deploying East-West compute network with the use of N9164E-NS4-O switch.

Compute-server software stack

Customers can run their choice of OS distribution and software versions as per the NVIDIA AI Enterprise compatibility matrix published by NVIDIA.

Security

Security in AI infrastructure is very crucial to ensure confidentiality, integrity, and high availability against adversarial attacks by implementing robust access controls and host and network isolation to prevent unauthorized access or manipulation. A number of Cisco security technologies, as enumerated below, are available that can be deployed by Enterprises to configure, monitor, and enforce end-to-end security right from applications to overall infrastructure. The complete integration of these technologies into the end-to-end workflow is beyond the scope of this RA.

●     Cisco Secure Firewall

●     Cisco Isovalent

●     Cisco Hypershield

●     Cisco AI Defense

Observability

Observability is a key element of AI infrastructure to ensure continuous visibility and reliability and to provide high-performance by tuning as well as proper infrastructure scaling. It also facilitates debugging, aids security, and helps maintain trustworthy and effective AI systems. Cisco Splunk^® is an industry-leading observability solution for Enterprises to ingest significant amounts of telemetry and gain in-depth visibility. It’s integration within the end-to-end workflow is beyond the scope of this RA.

Testing and certification

The overall solution has been thoroughly tested on all aspects of management plane, control plane, and dataplane, combining compute, storage, and networking. The compute racks are NVIDIA-Certified Systems^™. The Cisco EBox high-performance storage solution has achieved NVIDIA-Certified Storage validation at the NCP level. Several benchmark test suites (such as HPC Benchmark, IB PerfTest, NCCL Test, MLCommons Training, and Inference benchmarks) have also been run to evaluate end-to-end performance and assist with tuning. Different elements and entities of the NVIDIA AI Enterprise ecosystem have been brought up and tested to evaluate several enterprise-centric customer use cases around fine-tuning, inferencing, and RAG.

Summary

In short, Cisco Nexus Hyperfabric Full Stack AI Infrastructure is a fully validated AI cluster solution, offering customers an integrated AI infrastructure powered by NVIDIA accelerated computing.

Appendix A – Control-node server specifications

The following table shows the minimum specifications of control, management, and support server.

Table 7.        Minimum specification of control, management, and support rack server