Cisco UCS C880A M8 Rack Server Getting Started Guide

 

1. About this document

This guide is for use after the Cisco UCS^® C880A M8 Rack Server is installed and cabled and its Baseboard Management Controller (BMC) is reachable at a known management address. It provides a practical first-server workflow for BMC configuration, firmware upgrade, operating system installation, host and Graphics Processing Unit (GPU) software installation, adapter configuration, final checks, monitoring, and support-data collection.

The examples in this guide use BMC firmware 4.0(1.260014) and Ubuntu Server 24.04 LTS. The procedures described here are based on the software versions available and tested when the guide was published; they are intended as guidance, not a permanent compatibility guarantee. Vendors can replace or withdraw firmware, drivers, packages, and repositories. Before deployment, use the current Cisco^® compatibility information and the release notes for the selected Cisco, NVIDIA, operating system, and adapter software.

This guide does not repeat rack installation, power and network cabling, or the initial assignment of a BMC address. Those tasks are covered in the Cisco UCS C880A M8 Installation and Service Guide.

1.1 Bring-up sequence

Follow this sequence for the first server:

1.     Sign in to the BMC and check platform identity, health, KVM, and virtual media.

2.     Upgrade the platform firmware to the release selected for the deployment.

3.     Install Ubuntu Server through Common Internet File System (CIFS) remote media and the BMC KVM.

4.     Configure the operating system and verify host inventory.

5.     Check host-side firmware and configure the selected adapter profile.

6.     Install and validate the NVIDIA GPU software stack.

7.     Validate the connected deployment network.

8.     Run the final platform checks.

9.     Configure monitoring and learn how to collect support data.

A comprehensive Redfish monitoring and automation reference is provided in the final appendix.

1.2 How to use the optional sections

Unmarked chapters and sections belong to the first-server path. Use sections marked Optional when the feature belongs to the deployment. Use sections marked Conditional only when the server contains the hardware or configuration named in the heading.

Complete link negotiation, peer traffic, RDMA over Converged Ethernet (RoCE), and InfiniBand checks after the server is connected to the deployment fabric.

For an unexpected result, use Chapter 11, “Troubleshooting,” and the Cisco UCS C880A M8 Rack Server Troubleshooting Guide. If the result remains unresolved, collect the support bundle as described in Chapter 10, “Monitor and Support the Server,” and open a Cisco Technical Assistance Center (TAC) case.

1.3 Software used in this guide

The out-of-band and host procedures in this guide were tested with the software listed in the following table.

Table 1.        Software used for testing in this guide

Firmware and driver compatibility can change. Before installing an operating system or software stack, confirm the target configuration in the Cisco C880 Hardware and Software Interoperability List and review the release notes supplied with the selected Cisco and NVIDIA software.

The Cisco UCS C880A M8 Rack Server supports multiple operating systems. This guide uses Ubuntu Server 24.04 LTS for the installation, driver, and validation procedures. Confirm the specific operating system release and platform configuration in the current Cisco compatibility information before deployment.

This guide covers:

●     Initial BMC access and management-plane configuration

●     BMC inventory, health, logs, alerting, KVM, virtual media, and support tools

●     Redfish discovery, monitoring, event delivery, boot control, and task monitoring

●     Firmware inventory and the Cisco UCS C880A M8 Rack Server out-of-band upgrade script

●     Direct BIOS policy, host adapter firmware, and storage firmware lifecycle

●     Interactive operating system installation and Redfish virtual media, one-time boot, and Preboot Execution Environment (PXE) handoff

●     Host, GPU, NVLink, and network validation

●     Reversible ConnectX-8 InfiniBand and Ethernet profiles and configuration-dependent ConnectX-7 or BlueField-3 north-south networking

●     Host networking-stack selection and Ethernet/RoCE configuration

●     Support data collection

2. Configure and verify the BMC

This chapter signs in to the BMC, configures the essential management settings, and checks inventory, health, KVM, virtual media, and power control. Use the BMC web interface throughout the chapter. Redfish equivalents are provided in Appendix D, “Redfish Monitoring and Automation Reference.”

2.1 Open the BMC

From a management workstation, open https://<bmc-address> and sign in with the active BMC account. Confirm that the dashboard loads and identifies the Cisco UCS C880A M8 Rack Server. Hardware installation, cabling, and initial BMC network configuration are covered by the Cisco UCS C880A M8 Rack Server Installation and Service Guide and Troubleshooting Guide.

2.2 Sign in and confirm platform identity

Open https://<bmc-ip-or-fqdn>/ in a supported browser and sign in with an authorized BMC account. A factory or self-signed certificate can produce a browser warning during initial deployment. Confirm that the certificate was presented by the intended BMC before continuing. Replace it with a site-trusted certificate after DNS and time are configured.

After sign-in, use the dashboard to confirm:

●     Product model

●     BMC firmware version

●     Host power state

●     Overall system health

●     GPU count

●     Active events or pending sensor deassertions

●     BMC uptime and recent management access

BMC dashboard summary

Any active critical states must be investigated before firmware maintenance or operating system installation. Use the sensor and event-log workflows in Chapter 2, “Configure and Verify the BMC,” to identify the affected component(s).

2.3 Optional: Adjust BMC network identity and DNS

Use this section when the assigned BMC address, host name, or DNS settings must change. Open Settings > Network and select the page for the setting being changed.

BMC network-configuration areas

1.     Configure the assigned IPv4 or IPv6 address, prefix or subnet mask, and default gateway.

2.     Set the BMC host name and management Fully Qualified Domain Name (FQDN).

3.     Configure the site DNS servers.

4.     Select Save or Apply on the active page.

5.     Reconnect by using the final management FQDN.

6.     From the management workstation, confirm DNS resolution and HTTPS access.

Changing the active address ends the current browser session. Keep a route to the new address before applying the change.

2.4 Configure BMC time and Network Time Protocol (NTP)

Open Settings > Date & Time.

BMC date and time settings

Enable NTP, configure the primary and secondary site NTP servers, and select the site time zone or UTC. Save the settings.

BMC NTP server fields

Refresh the page and confirm that the displayed time is current and both NTP servers remain configured. Section D.8 shows the Redfish readback and ETag-aware update method.

2.5 Optional: Install the BMC Transport Layer Security (TLS) certificate

Open Settings > SSL Settings. The BMC can display the current certificate, generate a certificate request, and upload a replacement certificate.

BMC certificate-management actions

Use a certificate whose subject alternative names include the management FQDN used by administrators and automation. After uploading the certificate:

1.     Reconnect to the BMC by the FQDN.

2.     Confirm that the certificate chain is trusted.

3.     Confirm that the requested FQDN is present in the certificate Subject Alternative Names (SANs).

4.     Test Redfish without curl -k.

5.     Update automation to require certificate verification.

An IP address connection can still report a name mismatch when the certificate contains only DNS names. Prefer the management FQDN after the certificate is deployed.

2.6 Configure accounts and management services

Open Settings > User Management.

BMC user-management page

Select an available user slot, enable the account, enter the user name, assign the required role, and select Save. Reopen the slot and confirm the account name, enabled state, and role. Create the accounts required by the operating model before the server enters service.

Open Settings > Services.

BMC network services and secure ports

Confirm that HTTPS, KVM, virtual media, and Redfish are available on the management network. Enable Secure Shell (SSH) or Serial over LAN (SOL) only when the operating model uses them. Save any change, reload the page, and test each required service from the management workstation.

Optional: Test Serial over LAN

If the deployment uses Serial over LAN, install ipmitool on the Ubuntu management workstation and confirm the BMC channel:

 

Sample output:

 

After the Ubuntu serial console is configured in Section 10.6, connect to it:

 

Expected session start:

 

Press ~. to close the SOL session.

The BMC firewall is under Settings > Firewall. Apply a firewall rule only after the required administrator, monitoring, DNS, NTP, and alerting source addresses have been identified.

2.7 Review inventory, sensors, and logs

Open Server Health > System Inventory and review the available categories:

●     System

●     Processor

●     Memory controller

●     Base board

●     Power

●     Thermal

●     PCIe function

●     Storage

●     Network

Confirm that the expected processor count, memory population, storage devices, PCIe functions, and network devices are present.

Open Server Health > FRU Information to inspect Field-Replaceable Units (FRU).

FRU inventory categories

Open Server Health > GPU Information to confirm that all eight GPU entries are present and report the expected model.

GPU inventory summary

Open Server Health > Sensor to inspect environmental and state sensors. Filter the table by status or sensor type and review:

●     Temperature

●     Fan speed and fan presence

●     Power-supply presence, power-good state, and output

●     Voltage and energy readings

●     Thermal-limit and throttling indicators

●     Component-presence signals

Use the dashboard for the first health summary, then use the sensor table and event log to find the source of a warning or critical state. A discrete sensor can report state rather than a numeric reading. A disabled sensor is not equivalent to a healthy measured value.

The Server Health > Event Log page provides severity, event, sensor, and time filters.

BMC event-log filters and export controls

Use the log sources shown in the following table together.

Table 2.        Logs and their primary uses

 

Download relevant logs before clearing them. Clearing a log is a state-changing action and can remove evidence required by Cisco TAC.

The system log controls support date and event filters.

System log filters

The audit log can be filtered and downloaded for security and change review.

Audit log filters and export controls

2.8 Verify KVM and virtual media

Open Remote Control. The page provides the HTML5 KVM viewer and Serial over LAN entry points.

BMC remote-control options

Use the HTML5 viewer for:

●     BIOS and POST observation

●     One-time boot-menu selection

●     Interactive operating system installation

●     Virtual CD/DVD and hard-disk media

●     Keyboard macros and special key sequences

●     Console screenshots or recordings approved by the site

Before a maintenance window, launch KVM and confirm that video and keyboard input work. Close inactive KVM sessions when they are no longer needed. Review Settings > Media Redirection > Remote Session when keyboard layout, attach mode, reconnect behavior, or monitor behavior must be changed.

Open Settings > Media Redirection. The available pages include general remote-media support, virtual media instance limits, remote session settings, and active redirections.

Media redirection settings

Enable remote-media support before attempting to insert an ISO.

Remote Media Support setting

The settings page supports one CD/DVD instance, one hard disk instance, one remote KVM CD/DVD instance, and one remote KVM hard disk instance. Wait for the page to finish processing before interpreting the displayed values; the counters can briefly display zero while data is loading.

Virtual media instance settings

Use Active Redirections to verify whether media is currently attached before starting a new session or disabling remote-media support.

Active media redirection sessions

2.9 Verify power control

Open Power Control to view the host state and available actions.

Host power control actions

The available choices can include:

●     Force Off

●     Force Restart

●     Graceful Restart

●     Graceful Shutdown

●     On

●     Power Cycle

Use the action required by the current procedure. Firmware activation can require a host power cycle or full power removal and restoration; follow the firmware bundle instructions.

2.10 Final BMC checks

Continue when the model and installed inventory are correct, all eight GPUs are present, no unexplained critical condition is active, BMC time is correct, KVM accepts input, virtual media is available, and the expected power actions are visible. Resolve hardware or sensor faults before the firmware upgrade.

3. Firmware upgrade

This chapter uses the Cisco UCS C880A M8 Rack Server firmware bundle and its upgrade script to discover the installed versions, update the required components, and verify the running firmware after activation. The procedure was tested with BMC firmware 4.0(1.260014).

3.1 Select the firmware bundle

Download the C880A M8 bundle selected for the deployment from Cisco Software Download. Read the bundle release notes and the upgrade script README before starting. Save the bundle filename, checksum, and script version with the upgrade log.

3.2 Prepare the firmware workstation

The upgrade script version 1.1 README requires Python 3, OpenSSL 3.2 or later, and the Python requests, prettytable, and urllib3 packages. Run this preflight before connecting the script to a server:

 

Pass is indicated when Python 3 is active, openssl version reports 3.2 or later, both checksums match the files approved for the change, and the import test prints Python package imports: OK. If the workstation OpenSSL version is earlier than 3.2, use a management workstation that meets the upgrade script README requirement.

3.3 Check pre-update health and inventory

Open Firmware Inventory and save the displayed platform and HGX component versions. Also capture the dashboard health state, active conditions, event log, and current host power state.

BMC platform firmware inventory

BMC HGX firmware inventory

The discovery command in Section 3.5 supplies the script-side inventory. The optional Redfish inventory procedure is in Section D.3.

3.4 Review the BMC update controls

Open Maintenance > Firmware Update.

BMC Firmware Update entry page

The Firmware Update page supports server platform and GPU baseboard component images. The selected image, force-update option, restore-default option, target component, host power requirement, and activation requirement must match the Cisco bundle instructions.

Uploading or starting an update is disruptive. Do not use this page merely to test connectivity.

3.5 Run discovery and approve the target

 

The script's documented discovery output includes these stages and fields:

 

Review the discovered product, host power state, GPU model, running versions, packaged versions, and Update Required result. Do not continue if the model, bundle, or firmware inventory does not match the intended target.

3.6 Run the update and activation cycles

Run an approved update:

 

Expected progress, abridged:

 

The exact component list and activation requirements come from the selected bundle. Every required component must finish with Completed and 100, and the post-activation discovery must report the packaged versions as running.

The script can require:

●     Host power cycling for BIOS and GPU activation

●     Full power cycling for selected Complex Programmable Logic Device (CPLD) and ERoT activation

●     Temporary loss of BMC HTTPS access

●     Additional time for BIOS POST after activation

Use --skip-power-cycle-confirmation only in a separately approved, unattended maintenance workflow that has equivalent preflight and failure controls.

3.7 Final firmware checks

Run discovery again after the final activation cycle. Confirm that every updated component reports the intended running version, platform health is normal, and the host completes POST without a firmware-related fault.

If a component fails, remains staged, or disappears from inventory, stop the rollout. Include the upgrade script support archive, task history, and firmware inventory in the Cisco support case.

3.8 Upgrade multiple servers

Use the same discovery, approval, update, activation, and verification sequence for each server. The supplied upgrade script accepts one target at a time, so a fleet wrapper should run one isolated process per BMC and preserve the exit status and output for that target. Limit concurrency to the capacity of the management network and maintenance window.

Create bmc-targets.txt with one BMC address or FQDN per line, then run:

Expected result: Each target completes discovery, update, activation, and post-update discovery before the wrapper starts the next target. The wrapper stops at the first nonzero script exit.

4. Install the operating system

This chapter installs Ubuntu Server 24.04 LTS from CIFS-backed BMC remote media and the HTML5 KVM. Appendix D covers Redfish virtual media and PXE automation. If an HTTPS repository does not mount, use Section 11.1.

4.1 Prepare the CIFS image repository

1.     Copy the supported Ubuntu Server 24.04 LTS ISO to a CIFS share reachable from the BMC management network.

2.     Verify the ISO checksum against the value published by the image provider.

3.     Verify that the filename identifies an amd64 or x86_64 image.

4.     Create or select a read-only repository account.

5.     Confirm name resolution, routing, and firewall access between the BMC and the CIFS server.

6.     Have the CIFS server address, share and image path, and optional domain ready for the BMC form.

4.2 Attach and verify the ISO

1.     Sign in to the BMC.

2.     Open Image Redirection.

3.     Select Remote Images.

Remote Images entry point

4.     Open Media General Settings.

5.     Enable Remote Media Support.

6.     In Server Address for CD/DVD Images, enter the CIFS server IP address or resolvable name.

7.     In Path in server, enter the share and directory that contain the ISO.

8.     Under Share Type for CD/DVD, select cifs.

9.     Enter the domain when the repository account requires one.

10.  Enter the read-only repository username and password.

11.  Set Retry Interval and Retry Count according to site policy.

12.  Select Save.

CIFS remote-media settings; repository and account values are masked

13.  Follow the Remote Media link.

14.  If the expected ISO is not listed, select Refresh Image List.

15.  Select the Ubuntu Server 24.04 LTS AMD64 ISO.

16.  Select the triangular Play button for the CD/DVD row.

Ubuntu Server ISO selected before starting redirection; unrelated repository entries are masked

17.  Wait several seconds for the BMC to start the redirection.

18.  If the displayed state does not update, select Sync Image Status.

19.  Confirm that Redirection Status reports Started.

CIFS-backed virtual media in the Started state

Continue when Redirection Status reports Started. Keep the repository available until installation has completed and the virtual media has been detached.

4.3 Select the virtual CD/DVD from the boot menu

1.     Confirm on the Remote Media page that the CIFS-backed virtual CD/DVD still reports Started.

2.     Launch the HTML5 KVM viewer.

3.     If the console reports Powered Off, open Power and select On.

Powering on a server from the HTML5 KVM viewer..

4.     If the host is already running, restart or power cycle it from the KVM or the BMC power controls, as described in Section 2.9.

5.     When the POST prompt appears, press F7 to open the boot menu.

6.     Confirm that the console reports Entering Boot Menu.

POST confirmation that the boot menu request was accepted

7.     Select UEFI: CISCO Virtual CDROM0 1.00.

CIFS-backed virtual CD-ROM selected for one-time UEFI boot; unrelated boot entries are masked

8.     Press Enter to boot from the virtual CD-ROM.

9.     When the GNU GRUB menu appears, select Try or Install Ubuntu Server and press Enter.

Ubuntu Server installation option in GNU GRUB..

The boot menu selection applies to the current startup only.

4.4 Complete the Ubuntu installation

1.     On the welcome screen, select English and press Enter.

Ubuntu Server installer language selection

2.     Select Ubuntu Server.

3.     Leave Search for third-party drivers cleared. The GPU and network software are installed explicitly after the base operating system.

4.     Select Done.

Standard Ubuntu Server installation selected.

5.     On the Network configuration screen, identify the connected management interface from the site cabling plan. In the screen shown, the first Intel X710 10GBASE-T port appeared as enp88s0f0np0.

6.     Edit the interface, set IPv4 Method to Manual, and enter the site-assigned values:

●     Subnet: /

●     Address:

●     Gateway:

●     Name servers: ,

●     Search domains: The site search domain, or leave blank when none is required

7.     Select Save, verify that the interface reports the intended address, and then select Done.

Manual IPv4 configuration for the host management interface

If archive-mirror validation reports that a release file is not valid yet, verify the installer clock before changing repository settings. From Help, open the installer shell and run:

 

Expected output, generalized:

 

The first date -u can show an incorrect value. The final value must match current UTC closely enough for repository metadata and TLS validation.

Enter the current UTC date and time, return to the mirror screen, and retry validation. After the installation, configure the approved site NTP sources and confirm that the operating system clock is synchronized.

Confirm that the installer reports This mirror location passed tests, and then select Done.

Successful Ubuntu archive mirror validation

If the installer offers to update itself, follow the site deployment policy. For this procedure, select Continue without updating so that the installation remains tied to the installer supplied by the selected Ubuntu Server media.

On the Guided storage configuration screen:

1.     Select Use an entire disk and confirm that the intended boot device is selected. For the standard redundant boot layout, select the logical device provided by the two 960-GB M.2 SATA drives and the Cisco boot-optimized RAID controller.

2.     Leave Set up this disk as an LVM group selected.

3.     For this procedure, leave LUKS encryption cleared.

4.     Select Done to generate the proposed file system layout. Review the layout before confirming the destructive storage operation.

Caution: Using an entire disk removes the existing data and partition layout from the selected disk. Before continuing, verify the device, capacity, and intended data-retention outcome.

Entire-disk guided storage with LVM selected; the drive identifier is masked

This layout is an installation example, not a storage requirement. Select a different boot device or Custom storage layout when that is part of the deployment design.

With guided LVM, the installer can allocate only part of the volume group to the root logical volume and leave the remaining space free in the volume group. Verify the displayed root capacity. If it does not meet the deployment requirement, edit the logical-volume size, use a custom layout, or plan and document a post-install extension before accepting the storage changes.

This procedure accepts the generated 100 GB root logical volume and leaves the remaining volume-group capacity free for later allocation.

Generated file system summary for the accepted guided LVM layout

At the Confirm destructive action dialog, verify that:

●     The intended installation disk is selected

●     Required data has been backed up or is no longer needed

●     The proposed file system layout is acceptable

Select Continue to format the selected disk and begin installation. The installer does not allow a return to the storage screens after this point.

Final confirmation before the installer formats the selected disk

On the Profile configuration screen:

1.     Enter the user's friendly name.

2.     Enter the server hostname.

3.     Enter the Linux username.

4.     Enter and confirm the password.

5.     Select Done.

The installer uses this password for local login and sudo.

Local administrator profile; password values are masked by the installer..

On the SSH configuration screen, select Install OpenSSH server when the deployment requires remote command-line administration. The host-shell commands in the following chapters can also be run from the KVM console. This procedure installs OpenSSH and does not import an SSH key during installation.

Optional OpenSSH server selected for remote administration

Allow the installer to configure storage, extract the operating system image, install packages, and configure the boot loader.

When the installer reports Installation complete!, allow the security update phase to finish. Do not select Cancel update and reboot unless the deployment procedure explicitly requires updates to be deferred.

When Reboot Now becomes available, the installation and update phases are complete.

Installation complete and ready to restart

Before restarting:

1.     Confirm that the boot loader was installed on the intended local boot device.

2.     In the BMC UI, open Image Redirection > Remote Media.

3.     Refresh or synchronize the image status. The Ubuntu installer can eject the virtual CD automatically. When this happens, the BMC reports Stopped - Device Ejected.

4.     If the image remains active, stop or eject it before continuing.

5.     Return to the KVM console and select Reboot Now.

6.     Allow the system to boot from local storage.

7.     Confirm the login prompt through KVM.

BMC confirmation that the installer ejected the virtual CD

After the restart, confirm that the system reaches the local Ubuntu login prompt. This verifies that the installation media is no longer controlling the boot and that the installed operating system starts from local storage.

Ubuntu login prompt after the first local boot

4.5 Alternative: PXE handoff

Press F7 during POST and select the UEFI: PXE IPv4 entry that names the cabled provisioning interface. Confirm the selected Network Interface Card (NIC) by its firmware boot-entry label, Bus:Device:Function (BDF) or MAC address, and cabling record before continuing.

Section D.12 describes the Redfish PXE boundary. BMC firmware 4.0(1.260014) advertises the generic Pxe target and a BootOptions collection, but the override request does not select a specific NIC. Configure the intended NIC in firmware boot order before using the generic Redfish PXE override.

4.6 Final operating system checks

Confirm that the virtual media is stopped or ejected, the server boots from local storage, the Ubuntu login prompt appears, and the selected KVM console or SSH management path is reachable.

5. Configure and verify the host

Use this chapter to update Ubuntu, confirm the installed hardware, and establish the host management path before installing NVIDIA software.

5.1 Verify OS, time, packages, and restart state

 

Sample output, abridged:

 

Installer-generated Dynamic Host Configuration Protocol (DHCP) definitions for disconnected adapters can leave systemd-networkd-wait-online.service failed. Sections 5.4 and 5.5 map the interfaces and correct that condition. The final validation must report 0 loaded units listed.

Confirm that:

●     The installed operating system release is supported

●     The expected kernel is active

●     The production host name is set

●     Time synchronization is active

●     No unexpected system services have failed

Configure redundant site NTP servers with a systemd-timesyncd drop-in:

 

Pass is indicated when NTPSynchronized=yes and timedatectl timesync-status shows an active server.

Sample output:

 

If the site requires an APT proxy, configure it according to local policy. Then update the package index and installed packages:

 

Sample result:

 

Pass is indicated when apt list --upgradable shows no remaining approved updates and apt-get check completes without dependency errors.

Check whether updated packages require a restart:

 

Expected result: No output means no restart is pending. When a restart is required, the first file reports *** System restart required *** and the second lists the packages that triggered it.

Schedule a controlled restart before installing kernel-coupled GPU or network drivers. After the restart, repeat the operating system, time, package, service, and network checks.

The first restart can take several minutes. Wait for the selected KVM console or SSH management path to return.

After access returns, verify:

 

Sample post-restart output:

The second active line is present only when OpenSSH was selected during installation.

5.2 Verify CPU, memory, storage, and PCIe inventory

Install the Ubuntu inventory and health utilities:

 

Expected output: One executable path for every command. On Ubuntu 24.04, rdma and dcb are supplied by iproute2, ibv_devices by ibverbs-utils, and ibstat by infiniband-diags.

 

Expected output structure, abridged:

 

CPU, memory, Non-Uniform Memory Access (NUMA), storage, and PCIe counts must agree with the purchased configuration and the BMC inventory.

Check the boot and data drives:

 

Expected result: findmnt shows the root, boot, and EFI mount points on the selected boot device. /proc/mdstat can report no active arrays when software RAID is not configured. nvme list and smartctl --scan enumerate only the devices visible through their supported management interfaces.

The exact storage tools depend on the configured boot controller and data devices.

5.3 Identify the host management interface

 

Expected output, abridged:

 

Identify the management interface from its active link, MAC address, PCI BDF address, and cabling plan. Verify the BDF-to-interface mapping in Section 5.4 before configuring production bonds, VLANs, RoCE policy, or data-network addresses.

5.4 Map PCI devices to Linux and RDMA names

Create the read-only interface map before changing Netplan. The PCI BDF joins PCI inventory, Linux interface names, driver information, and RDMA devices.

 

Expected output for each managed interface includes driver, firmware version, bus-info, and a sysfs path ending in the same BDF. The server has eight physical ConnectX-8 adapters. The active east-west profile exposes eight or sixteen ConnectX-8 PCI functions. The purchased configuration also has two north-south ConnectX-7, BlueField-3 B3220, or BlueField-3 B3240 cards. Runtime mlx5_X names can change; rediscover them after firmware, driver, or profile changes.

5.5 Complete the Netplan baseline

The Ubuntu installer can generate a DHCP configuration for every adapter it discovers. On a system with many unconnected data ports, this can leave systemd-networkd-wait-online.service failed even though the management interface is routable. Review the generated configuration:

 

Sample condition, abridged:

 

Use the map created in Section 5.4 to identify the active and unused interfaces.

Remove unintended DHCP configurations or mark unused interfaces optional. Regenerate and validate the Netplan configuration before the next restart. Run the change from KVM or another confirmed recovery session when the management interface is affected.

The following drop-in keeps the installer-generated definitions but prevents nonrequired data interfaces from blocking network-online.target. Include only interfaces that are not required during host startup. Do not include the active management interface:

 

Confirm the Netplan prompt only after a second session can reach the host management address. If the test is not confirmed within the timeout, Netplan restores the previous configuration.

Sample result, abridged:

 

If an interface will later participate in a bond, VLAN, bridge, storage network, RoCE fabric, or InfiniBand fabric, replace the temporary optional definition with the production network configuration.

5.6 Choose the host networking stack

Check the current driver source and installed networking packages:

 

Sample output, abridged:

 

Use the Ubuntu inbox stack when it supports the deployment requirements. For new NVIDIA networking deployments that require vendor drivers or DOCA features, use a supported DOCA-Host profile. NVIDIA is transitioning standalone MLNX_OFED to DOCA-OFED; the final standalone MLNX_OFED release remains in long-term support through October 2027.

Confirm the supported operating system, kernel, adapter firmware, and required features before selecting a profile.

Table 3.        Uses and scopes of host stacks and profiles

Follow the selected profile's installation and transition instructions. After the required restart, repeat the commands above and confirm every expected PCI function, network interface, RDMA device, and firmware version.

5.7 Final host checks

Confirm that the supported OS and kernel are active, time is synchronized, approved updates are complete, no restart is pending, no unexpected service is failed, hardware counts match the ordered configuration, every managed interface has a verified BDF mapping, and the management path remains routable.

6. Configure host firmware and adapters

This chapter checks host-managed storage and network adapter firmware, then configures the eight ConnectX-8 east-west adapters for either 8x800-Gb/s InfiniBand or 16x400-Gb/s Ethernet or InfiniBand. North-south procedures depend on whether the server contains ConnectX-7 adapters or BlueField-3 DPUs.

6.1 Install MFT and inventory network adapter firmware

The Cisco out-of-band upgrade workflow does not update the host-managed ConnectX-8 or north-south adapters. On Ubuntu 24.04, enable the NVIDIA CUDA repository and install the NVIDIA firmware tools (MFT):

 

Sample package source, abridged:

 

MFT 4.35.0.159-1 supplies mlxfwmanager, mlxconfig, mlxlink, and the Mellanox software tools (mst) service and command. mst exposes supported devices to MFT utilities. The iproute2 package supplies devlink.

Query adapter firmware and the exact parameter set IDs (PSIDs) from the operating system. A PSID identifies the board-specific firmware package for an adapter:

 

Table 4.        Expected firmware and PSID inventory

 

For each physical device:

1.     Check mlxfwmanager --query, devlink dev info, ethtool -i, and the corresponding mlxconfig -e ... q output.

2.     Obtain the firmware image from the Cisco approved bundle or the NVIDIA package explicitly approved for the C880A M8.

3.     Confirm the device type, exact PSID, image PSID, current firmware, target firmware, and activation requirement.

4.     Stop workloads and preserve out-of-band management.

5.     Update one adapter population at a time.

6.     Perform the specified adapter reset, host restart, or complete power cycle.

7.     Re-query every physical device and validate the link mode, PCI enumeration, driver binding, interface mapping, and health.

Use an exact-PSID update pattern:

 

Expected preflight result: The utility identifies the intended device and the image listing includes the expected device PSID. Do not run --update when the image is absent, the PSID differs, or the approved release instructions do not identify the device.

The host can also expose ConnectX-7 functions used by the internal HGX fabric path. On a ConnectX-7 north-south configuration, two additional external ConnectX-7 2x200-Gb/s cards are present. Use the product description, PSID, advertised port capability, and purchased configuration to distinguish them from the internal functions. Include the external cards in the north-south adapter update and validation workflow; leave the internal HGX functions with the HGX software and firmware stack.

6.2 Inventory and update host storage firmware

Install the NVMe and SMART management utilities:

 

Expected result: APT installs or confirms both utilities and the version commands complete successfully.

Capture model, running firmware, firmware slots, SMART state, and error logs:

 

Sample NVMe result, sanitized:

Expected result: Each selected SATA device reports SMART overall health PASSED and no reallocated, pending, or offline-uncorrectable sectors.

Use only a Cisco approved host firmware bundle or a vendor package identified for the exact model. Before an update, back up data, stop I/O, and confirm redundancy, device identity, and the active firmware slot. Follow the package instructions for download, commit, activation, and restart; do not infer a slot or activation action from another NVMe model.

When the approved package explicitly uses the standard NVMe interface, its workflow has this form:

 

Expected result: Download reaches 100 percent and commit reports successful firmware activation or the reset/power action required to activate it. The slot and action values are model- and package-specific. Do not substitute example numbers.

After activation, repeat nvme id-ctrl, nvme fw-log, nvme smart-log, and nvme error-log. Acceptance requires the approved active firmware, zero critical warnings, no new media error, and no unexplained error log entry.

6.3 Discover the east-west and north-south adapters

Discover the first PCI function (.0) of each of the eight physical ConnectX-8 adapters and query the complete adapter inventory:

 

If the server has ConnectX-7 north-south cards, confirm that mlxfwmanager --query lists two external 2x200-Gb/s cards with the expected PSIDs. If the server has BlueField-3, the command must list two B3220 or two B3240 devices.

Sample prechange state from a BlueField-3 configuration:

 

Use the discovered arrays in the commands that follow.

6.4 Select the ConnectX-8 profile

Choose the profile required by the deployment network design before running any mlxconfig set or mlxconfig reset command.

Table 5.        Choosing a procedure for the ConnectX-8 profile

 

If the current and next-boot values already match the selected profile, do not stage a change. Otherwise, create the backup in the next section and run only the matching procedure below.

The 800-Gb/s InfiniBand profile is the firmware default for the NVIDIA ConnectX-8 C8180P OCP 3.0 adapter used in this server. In the 16x400-Gb/s profile, a deployment can use all sixteen interfaces as two planes or use the odd or even interfaces as one plane.

6.5 Back up ConnectX-8 NVConfig

NVConfig is the nonvolatile adapter configuration that controls settings such as link protocol, port split, and PCI functions.

Create and verify a backup immediately before the first profile change:

 

The final find command should print nothing. Both count tests must succeed, and SHA256SUMS must contain one query file and one raw backup for each physical adapter.

6.6 Apply the selected ConnectX-8 profile

Run only the profile selected above. Each change is staged in Next Boot and requires a complete power-off and power-on cycle; an operating system restart is not sufficient.

6.6.1 Keep or return to 8x800-Gb/s InfiniBand

One 800-Gb/s InfiniBand port is the C8180P firmware-default profile. NVIDIA recommends resetting NVConfig to default when leaving a nondefault split.

After creating the backup above, reset each physical adapter through its first PCI function:

 

Expected output for each adapter:

 

mlxconfig reset returns all NVConfig values for the selected device to firmware defaults. It is not limited to the link type or split fields. Review the Default, Current, and Next Boot columns before proceeding.

For the documented C8180P profile, the default fields are:

 

Perform a complete power-off and power-on cycle. Then verify:

 

Expected result after the power cycle:

 

ibstat, rdma link show, and mlxlink must show the InfiniBand link layer and the expected port state. A production fabric must report the expected negotiated speed and active state.

Confirm the InfiniBand link layer, subnet-manager integration, expected port state, and fabric counters after connecting the host to the deployment fabric.

6.6.2 Apply the 16x400-Gb/s profile

Set the link protocol to 2 for Ethernet or 1 for InfiniBand, then apply the dual-port split to the first PCI function of every C8180P adapter:

 

Sample output for each changed adapter:

 

The command stages Next Boot values. It does not activate the new profile until the required complete power-off and power-on cycle.

Perform a complete host power-off and power-on cycle. An operating system restart alone is not sufficient for this C8180P profile change. Use the BMC power controls described in Section 2.9, confirm that the host reaches the off state, and then power it on.

After the power cycle, verify all eight devices:

 

Generate a compact aggregate check. First count the PCI functions:

 

Count the advertised 400-Gb/s capability and current cable state:

 

Count the Current and Next Boot profile values:

Sample aggregate result from the Ethernet profile:

 

Expected profile, abridged:

 

The operating system should expose sixteen C8180P functions in the selected protocol. For each adapter pair, devlink port show must report one function on physical port 0 and one on physical port 1. For InfiniBand, also confirm the link layer with ibstat and rdma link show.

6.7 Conditional: Check BlueField operating mode

If the server has BlueField-3 north-south cards, use the BLUEFIELD_DEVICES array created in Section 6.3 to query link type and operating mode ownership. If the server has ConnectX-7 cards, continue to Section 6.8.

 

Example output from a B3220 configuration, abridged:

 

To configure DPU mode with both network ports set to Ethernet:

●     INTERNAL_CPU_OFFLOAD_ENGINE=ENABLED(0) means DPU mode. The embedded Arm subsystem is active and owns the NIC resources through the embedded CPU function (ECPF).

●     INTERNAL_CPU_OFFLOAD_ENGINE=DISABLED(1) means NIC mode on BlueField-3. The Arm cores are inactive and the device behaves as a ConnectX adapter for the external host.

●     LINK_TYPE_P1 and LINK_TYPE_P2 independently select Ethernet or InfiniBand.

Leave the mode unchanged during initial bring-up unless the deployment design requires a different profile. For DPU mode, also verify the Arm-side Board Support Package (BSP) or DOCA version, management access, RShim state, service health, and port ownership.

If the deployment requires a mode change, create a raw NVConfig backup for each BlueField device as shown in Section 6.5, then stage the required mode:

 

Set the link protocol separately:

 

Expected output is Applying... Done!. Re-query both BlueField devices, confirm the requested Next Boot values, perform the activation action specified by MFT, and verify the runtime mode and link layer.

Verify the DPU management path

For a deployment that manages the BlueField Arm systems from the host, install RShim and verify both embedded Arm systems:

 

Sample result, abridged:

 

The RShim network interfaces use the same point-to-point subnet by default. Test one DPU at a time:

 

Sample result for both DPUs:

 

Open an Arm console when direct console access is needed:

 

Expected result: Each configured DPU console reaches its operating system login prompt. Check the BSP and DOCA versions, required services, port ownership, and health on each Arm system.

6.8 Final adapter and storage checks

Confirm that all eight ConnectX-8 physical adapters and the configured north-south cards are present, adapter and storage firmware match the approved set, every ConnectX-8 backup is nonempty, Current and Next Boot match the selected profile, and storage health reports no critical warning or new media error. If the server uses BlueField DPU mode, also confirm the RShim path and the required Arm-side software and services.

7. Install and validate the NVIDIA GPU stack

Install the NVIDIA driver, Fabric Manager, CUDA, NCCL, and DCGM, then verify that all eight GPUs and the NVLink fabric operate correctly.

7.1 Prepare the host and confirm GPU visibility

Install the build prerequisites for the running kernel:

 

Expected result: APT finishes without dependency or Dynamic Kernel Module Support (DKMS) errors. The running kernel's matching linux-headers package and the build tools are installed.

Check the operating system, kernel, Secure Boot, and GPU baseline:

 

The server exposes eight NVIDIA 3D-controller functions before the driver is installed:

 

The final value is the number of matching 3D-controller functions. Resolve missing PCIe devices before installing or changing drivers. If Secure Boot is enabled, use the site's approved module-signing and Machine Owner Key workflow.

7.2 Enable the NVIDIA repositories

Install the NVIDIA CUDA repository keyring for Ubuntu 24.04:

 

Expected output, abridged:

 

Set the approved software branches. The defaults below reproduce the tested stack:

 

Expected output: Each package shows an APT source and a non (none) Candidate version. Confirm that the driver, Fabric Manager, CUDA, NCCL, and DCGM branches are compatible before installation.

If APT reports an unresolved UCX dependency, enable the current NVIDIA DOCA repository selected for the same supported stack, run apt-cache policy ucx, and install the candidate supplied by that repository.

7.3 Install the driver and NVLink packages

Install the branch-pinning package before the driver so that the branch-coupled packages remain aligned:

 

Expected output, abridged:

Pass is indicated when DKMS completes and all branch-coupled packages use the selected NVIDIA branch.

Before restarting, confirm that the driver module was built for the running kernel:

 

Sample output:

 

If DKMS does not report installed for the running kernel, inspect the DKMS build log before restarting. Do not continue with a partially built driver.

Restart the host and allow the platform to complete POST:

 

Expected result: The SSH session closes as the host restarts. Redfish should continue to report power On; allow the platform to finish POST before testing SSH again.

7.4 Start and validate Fabric Manager

NVIDIA Fabric Manager configures and monitors the NVSwitch and NVLink fabric. Load the InfiniBand userspace MAD (ib_umad) module before starting the service:

 

Expected result: The module and UMAD device are present, Fabric Manager reports active, and nvidia-smi -L lists eight GPUs.

 

Validate the GPU fabric and NVLink topology:

 

Sample output, abridged:

 

The Completed and Success pair must appear for all eight GPUs. NV18 between each GPU pair indicates that the topology command sees the expected bonded NVLink paths.

Check the NVIDIA IMEX service:

 

IMEX is used for a configured multinode NVLink domain. On a standalone server, disable an unconfigured service:

 

Sample single-server result:

 

For a multinode NVLink domain, populate the IMEX node configuration for that domain and start the service instead. The single-server disable action leaves the package installed.

7.5 Install CUDA, NCCL, DCGM, and RDMA tools

Install the user-space packages:

 

Expected result: APT resolves the pinned CUDA 13.0 and NCCL versions, installs DCGM and RDMA utilities, and exits without dependency errors. The version query later in this section is the acceptance check.

Make the selected CUDA toolkit available to login shells and enable DCGM:

 

Expected output, abridged:

 

An existing enabled service may not print the symlink line. Confirm with systemctl is-enabled nvidia-dcgm and systemctl is-active nvidia-dcgm; both must return enabled and active.

Capture the installed package versions:

 

Sample output:

 

7.6 Validate CUDA and DCGM

Confirm the CUDA compiler and DCGM versions:

 

Sample output:

 

Run DCGM discovery, enable health watches, and run a level-1 diagnostic:

 

Sample output, abridged:

 

Health watches are runtime states. After restarting DCGM or the host, run dcgmi health -s a before dcgmi health -c. Otherwise DCGM can report that health watches are not enabled.

Level 1 is a short deployment diagnostic. For a broader GPU check, run a level 2 diagnostic during a maintenance window:

 

Sample output, abridged:

 

Compile and run a small CUDA program to validate the compiler, runtime, executable loader, and device discovery independently from NCCL:

 

Save the source as cuda_device_query.cu, then run:

 

Sample output, abridged:

 

This test proves that a compiled CUDA executable can discover and query all eight GPUs. It does not replace DCGM diagnostics or a multi-GPU communication test.

Capture persistence mode, peer access, I/O Memory Management Unit (IOMMU) state, and negotiated PCIe links:

 

Sample result, abridged:

 

For every PCIe device, LnkSta should match the expected negotiated generation and width. Investigate any downgraded links before performance qualification.

7.7 Run the NCCL all-reduce test

Fetch the exact NCCL test revision used for the sample output:

 

Expected result: The checkout enters detached-HEAD state and the final test returns zero. This guide used:

 

Build the tests for B300 compute capability 10.3:

 

Expected result: The compilation finishes without an error and creates $HOME/nccl-tests/build/all_reduce_perf. Compiler progress varies with the toolchain; verify that the executable exists before running it.

Run the eight-GPU all-reduce test from 8 MiB through 8 GiB:

 

Sample output, abridged:

 

Retain the full output, source revision, package versions, and test arguments when the result will be used as a performance baseline.

7.8 Verify restart persistence

Confirm persistence mode before and after the restart:

 

Sample result: GPUs 0 through 7 all reported Enabled.

If the deployment requires persistence mode and a GPU reports Disabled, enable it during the approved configuration window:

 

Expected result: The configuration command reports that persistence mode is enabled for every GPU, and the query returns Enabled eight times.

Use a second controlled restart to confirm that the kernel module and services start without manual intervention:

 

Expected result: The remote session closes. After the platform returns, the following checks must succeed without manually loading ib_umad or restarting NVIDIA services.

After the host returns, run:

 

Sample output, abridged:

The three enabled results and three active results correspond, in order, to Fabric Manager, NVIDIA Persistence Daemon, and DCGM. The Fabric state and status pair appeared for all eight GPUs.

Run a short eight-GPU NCCL test after the restart. Expected result: The test completes with zero validation errors. Investigate any change in GPU count, fabric state, service state, persistence mode, topology, or NCCL correctness before continuing.

7.9 Final GPU checks

Confirm that eight GPUs are visible, Fabric Manager reports Completed and Success, DCGM level 2 passes for GPU0 through GPU7, the CUDA query exits with zero, the NCCL all-reduce reports #wrong 0, and the same state returns after restart.

8. Validate networking

Complete this chapter when the external ConnectX-8 or configured north-south ports are connected to the deployment fabric. The inventory and configuration commands can still be run before the links are connected.

8.1 Conditional: Verify Ethernet and RoCE layers

RoCE validation covers the adapter, host, and fabric layers.

Table 6.        What to validate for each layer

 

Query the persistent adapter properties:

 

Sample Ethernet mode result, abridged:

 

This result establishes adapter configuration only. It does not show that the host or fabric QoS policy is complete.

With standard Linux DCB tools, query the host state:

 

Sample output for an unprovisioned Ethernet configuration:

 

These values are observations, not recommended production defaults. Configure trust, priority mapping, Priority Flow Control (PFC), Explicit Congestion Notification (ECN), differentiated services code point (DSCP) or Priority Code Point (PCP), and Maximum Transmission Unit (MTU) according to the deployment fabric design. Lossless treatment is normally limited to the selected RoCE priorities; enabling PFC on every priority can spread congestion to unrelated traffic.

The exact tool set depends on the selected host stack. A DOCA-Host profile can also provide NVIDIA utilities such as mlnx_qos, cma_roce_tos, ibdev2netdev, or lldptool. The Ubuntu inbox path uses dcb, rdma, ibv_devices, ibv_devinfo, ibstat, ethtool, mlxconfig, and mlxfwmanager.

After applying the production policy, repeat every query and verify traffic as described in Section 8.3. Compare host and switch counters before and after the test. Traffic validation requires a connected peer.

ConnectX-7 functions whose product description identifies the internal HGX communication path are managed with the HGX stack. Do not apply the external ConnectX-8 or north-south adapter workflow to those internal functions.

Authoritative references:

●     NVIDIA ConnectX-8 OCP port configurations

●     NVIDIA MFT “mlxconfig” backup and reset

●     NVIDIA BlueField modes of operation

●     NVIDIA DOCA-Host profiles

8.2 Conditional: Verify link state and GPU proximity

 

Expected result: Every deployed interface appears once, rdma link associates RDMA devices with their current netdevs, and the topology matrix shows the expected GPU-to-NIC proximity.

For each production link:

 

Expected output on a healthy Ethernet link:

 

Counter names vary by driver. Investigate increasing error, discard, Cyclic Redundancy Check (CRC), or unexpected pause counters rather than requiring every possible counter to be present.

Correlate the port, BDF, Linux interface, RDMA device, and GPU topology before applying bonds, VLANs, addresses, or RoCE policy.

A link-down result confirms tool access and configured speed capability but does not validate physical link negotiation or traffic. Production acceptance must also verify negotiated speed, bidirectional traffic, throughput, packet and error counters, and switch-side configuration.

For each cabled Ethernet/RoCE or InfiniBand path, run the peer traffic commands in Section 8.3 and verify switch-side speed, MTU, counters, and congestion configuration.

8.3 Conditional: Run peer network and RDMA tests

Install the peer traffic tools when they are part of the deployment test plan:

 

Expected result: APT installs the utilities and all three version commands complete. Installing the tools does not configure the fabric or start a listener.

Check inventory and configured state:

 

Expected inventory result, abridged:

 

Inventory validates PCIe discovery, driver binding, RDMA-device enumeration, NVConfig, and deterministic mapping. Validate the deployment fabric separately.

For an Ethernet or RoCE profile, check the host policy before traffic:

 

Expected result: MTU, pause, PFC, priority mapping, and counters match the approved fabric design.

For each Ethernet link:

 

Expected result:

 

For an InfiniBand link:

 

Expected result:

 

For RoCE or InfiniBand transport validation, run an approved test between two fabric-connected hosts. Start the receiver on the peer, then run the client:

 

For RoCE, include the required GID index and address-family options, such as -x <gid-index>, according to the deployment network design.

Expected result: The client and receiver establish the RDMA connection, the test completes without transport errors, and the reported bandwidth is consistent with the deployment baseline.

Validate IP reachability, MTU, and bidirectional Ethernet throughput when the network design uses an IP data path:

 

Run the iperf3 server on one endpoint and the client on the other, then reverse the direction. Expected result: No ping loss, no MTU error, throughput consistent with the site baseline, and no increase in CRC, discard, or error counters.

Use the commands in this section with the deployment network test plan. Verify host-side and switch-side speed, link layer, MTU, counters, and test results.

For connected interfaces, confirm that every required production link negotiates the intended speed and MTU, switch and host policy agree, bidirectional traffic succeeds, transport tests complete without errors, and counters remain stable.

9. Run final platform checks

Run these checks after completing Chapters 2 through 8 to confirm the server-local hardware, firmware, operating system, GPU stack, and services.

Expected result: The installed inventory matches the purchased configuration, firmware and services are healthy, storage and adapters report no errors, all eight GPUs and the NVLink fabric pass their checks, and the applicable local smoke tests complete successfully. Complete deployment-network checks when the required fabric links are available.

9.1 Verify host, CPU, memory, the configured boot device, and storage

Set the expected storage population for the purchased configuration:

In System Inventory > Storage, confirm the logical boot device and the physical members reported by the installed boot controller. The standard redundant boot configuration reports one healthy logical boot device backed by two healthy 960-GB M.2 SATA members. If another supported boot design was selected during installation, compare the logical device and member state with that design instead.

Expected result: The E1.S count matches the purchased configuration, the operating system file systems are on the intended logical boot device, the boot controller reports the expected logical device and member state, no unexpected service is failed, time is synchronized, no restart is pending, each NVMe device reports critical warning 0 and no new media error, every device discovered by smartctl reports a healthy state, and the kernel review contains no unexplained storage or PCIe error.

9.2 Verify GPU and NVLink state

Expected result: Eight GPUs are present, every fabric block reports Completed and Success, all three services are active, DCGM level 2 passes for GPU0 through GPU7, and NCCL reports #wrong 0 and Out of bounds values: 0 OK.

9.3 Verify adapter and storage firmware

 

Expected result: If the server has ConnectX-7 north-south cards, mlxfwmanager --query lists two external 2x200-Gb/s cards. If it has BlueField-3, the command lists two B3220 or two B3240 devices. The counts match, every expected device appears in the firmware inventory, versions match the approved set, and Current and Next Boot match the selected adapter profiles.

9.4 Run bounded post-configuration smoke tests

Install the validation tools when they are not already present:

 

Expected result: APT installs or confirms stress-ng and fio without dependency errors.

Run a 60-second CPU verification workload:

Sample output, abridged:

 

--cpu 0 uses all online logical CPUs. Reduce the worker count when site policy does not permit a full CPU load.

Run a bounded memory verification workload:

Sample output, abridged:

--cpu 0 uses all online logical CPUs. Reduce the worker count when site policy does not permit a full CPU load.

Run a bounded memory verification workload:

Sample output, abridged:

 

The example allocates 32 GiB in total. Select a workload that leaves sufficient memory for the operating system and any active services.

Run temporary direct I/O against a file system with at least 4 GiB of free space. The subshell removes its temporary directory on completion or error:

 

Sample output, abridged:

 

Expected result: err=0, the expected byte count is transferred, and no new storage or kernel error appears.

9.5 Verify the deployment network when connected

If the data interfaces are connected, run the host, peer, and switch checks in Chapter 8. If they are not connected, continue with server-local validation and complete the network tests when the deployment fabric is available.

9.6 Review final firmware and BMC/HGX health

Return to the BMC after the firmware, power cycle, adapter, GPU, and smoke test work. Run the discovery command in Section 3.5 again and review the final platform, BIOS, and HGX firmware inventory. Then review Dashboard, Sensor, GPU Information, and Logs & Reports.

Expected result: The running platform, BIOS, and HGX versions match the approved firmware set, the current platform and HGX health rollups are OK, and no active Warning or Critical condition affects the validated configuration.

10. Monitor and support the server

Configure the alert and monitoring paths used at the site, and use the support procedures in this chapter when a problem requires Cisco assistance.

10.1 Optional: Configure BMC alert delivery

The BMC separates event matching from delivery:

1.     Open Logs & Reports > Platform Event Filters > LAN Destinations and configure the Simple Network Management Protocol (SNMP) destination, or open Simple Mail Transfer Protocol (SMTP) and configure the email relay. Select Save and reopen the destination to confirm the values.

2.     Open Event Filters, select an available slot, enable it, choose the required sensor or event class and severity, and select Save.

3.     Open Alert Policies, select an available slot, enable it, associate the event filter with the intended action and destination, and select Save.

4.     Reopen the filter and policy and confirm that both remain enabled and that the destination number matches.

Platform Event Filter configuration areas

Platform event filter slots

Alert policy slots

SNMP LAN destination slots

SMTP alert settings

In LAN Destinations, select the configured destination and click Send Test Alert. The selected destination type determines whether the test uses SNMP or email; SMTP must be enabled for email delivery. Confirm that the site receiver records the test.

10.2 Optional: Back up the BMC configuration

Open Maintenance to access configuration backup and restore, system diagnostics, recovery image synchronization, factory reset, and BMC restart.

BMC maintenance tools

Open Maintenance > Configuration Backup:

1.     Check the current BMC firmware version.

2.     Select the configuration groups required for recovery.

3.     Enter the encryption value requested by the page.

4.     Generate and download the backup.

5.     Calculate its checksum after download:

 

Expected result: One SHA-256 digest and the backup filename. A restore can replace the active management configuration; exercise it only in the planned recovery procedure.

The System Diagnostics page generates the Cisco BMC support bundle.

BMC diagnostic log generation

The Host System Diagnostics area shows autonomous crash-dump records when data is available.

Host autonomous crash-dump records

Generate support data after reproducing an issue and before clearing logs or restarting the BMC. Section 10.5 lists the case artifacts.

10.3 Generate Cisco BMC support data

Open Maintenance > System Diagnostics. Select Generate Log, and wait for generation to finish. The operation can take several minutes. Do not restart the BMC, clear logs, or start another diagnostic collection while it is running.

The Last Log File area shown on the page changes from No Log Found to the generated archive when the bundle is ready. Download the archive without changing its contents, and calculate a checksum.

Also collect:

●     Dashboard health summary

●     System, GPU, FRU, and firmware inventory

●     Sensor table filtered to warning and critical states

●     Event log, system log, and audit log exports

●     Active media redirection state when the issue involves installation

●     Autonomous crash-dump data when present

●     Screenshots of the exact warning or failed task

10.4 Collect host diagnostics

Collect:

Expected result: Each command returns the corresponding operating system, service, PCIe, storage, GPU, network, or RDMA evidence. dcgmi diag -r 1 must finish with Pass for GPU0 through GPU7. Run the storage and networking commands that apply to the installed configuration.

Include the Cisco upgrade script log archive for firmware failures. Preserve generated support archives and their checksums for the Cisco support case.

10.5 Collect information for a Cisco support case

Include the problem time and symptom, BMC and BIOS versions, the pre-failure and current health states, relevant event and task records, the Cisco BMC support bundle and checksum, host journal and inventory, and the exact command or workload that exposed the failure.

10.6 Optional: Configure the installed OS Serial over LAN console

Use this procedure when the operating model requires a serial console for the installed Ubuntu system:

 

From the management workstation, open the console:

 

The channel maps to ttyS0 at 115200 baud. SOL should display the Ubuntu login prompt. Enter ~. to close the ipmitool SOL session. After the next restart, verify that /proc/cmdline contains console=ttyS0,115200n8 and that the login prompt returns.

10.7 Optional: Add the server to a monitoring system

Add the BMC to the site's HTTPS and Redfish monitoring system. Confirm that the collector can read the service root, system health, sensor resources, logs, and service conditions. The detailed collector and Redfish event examples are in Appendix D.

Test each configured SNMP or SMTP destination with the site's normal receiver. The Redfish SSE test in Appendix D validates the Redfish event stream; it does not by itself prove SNMP or SMTP delivery.

11. Troubleshooting

Use these focused checks when a procedure in this guide does not produce the expected result. For other faults, use the Cisco UCS C880A M8 Rack Server Troubleshooting Guide or collect the Chapter 10 artifacts for Cisco support.

11.1 HTTPS remote media does not mount

The installation in Chapter 4 uses CIFS. Use this procedure when an HTTPS repository is selected and the BMC cannot open or stream the ISO.

On the Ubuntu image server, install NGINX, OpenSSL, and the Basic Authentication utility:

 

Copy the ISO to /srv/rmedia, then create a certificate whose subject alternative name matches the exact IP address or FQDN entered in the BMC:

 

Create /etc/nginx/sites-available/rmedia:

 

Enable the site and restart NGINX:

 

Verify authentication, certificate matching, and byte-range delivery from a management workstation:

 

The first request returns HTTP 200 and the file length. The range request returns HTTP 206 Partial Content with Content-Range: bytes 0-1023/....

In Image Redirection > Remote Images > Media General Settings, select HTTPS and enter the same server address, port 443, ISO path, and Basic Authentication credentials. Save the settings, refresh the image list, select the ISO, and start redirection. Continue when Redirection Status shows Started.

For Redfish mounting, use the same URL and credentials in the InsertMedia request in Section D.10. A completed task must be followed by a readback that shows Inserted: true, ConnectedVia: "URI" and RedirectionStatus: "Redirection Started".

11.2 Fabric Manager does not start

If Fabric Manager reports that ib_umad is unavailable, verify the package, module, UMAD device, and service:

 

Expected result: ib_umad is loaded, at least one UMAD device is present, and Fabric Manager reports active. If the module or device is absent, include the service journal, PCIe inventory, and Chapter 10 support files in the Cisco support case.

11.3 Restore a ConnectX-8 NVConfig backup

Use the raw file created in Section 6.5 only with the matching PCI function. Apply it with the MFT set_raw command:

 

Confirm the prompt, then perform the power action reported by MFT. Query the device again and verify that Current and Next Boot contain the intended values.

Appendix A. Glossary and command prerequisites

The following table lists terms used in the host and adapter procedures.

Table 7.        Glossary of terms used in host and adapter procedures

Each procedure installs its required utilities before first use. The table below lists the main package groups.

Table 8.        Utilities package groups

Appendix B. BIOS, NUMA, and IOMMU reference

Use this appendix to review the active BIOS policy and confirm its effect from the operating system. Complete Section D.1 before running the Redfish commands.

Read the current BIOS attributes and the pending settings resource before an operating system, firmware, or performance policy change:

 

The first response is the current state. The Settings resource contains values staged for a later BIOS apply operation. A read-only baseline must not PATCH either resource.

Check the BIOS policy areas listed in the following table.

Table 9.        BIOS policy areas

Cross-check the BIOS view from the host:

 

Expected output: One executable path for each command.

 

Sample result, abridged:

 

Start with the factory default BIOS. Use iommu=pt only when the selected GPUDirect or application design requires IOMMU pass-through, then repeat the GPU and network checks that apply to the deployment.

Appendix C. Extended GPU qualification

Use these longer tests when the application team needs a repeatable GPU communication baseline beyond the Chapter 7 checks.

C.1 CUDA peer-to-peer sample

Run the CUDA peer-to-peer sample when a direct executable test is required. The CUDA toolkit and compiler must already be installed as described in Sections 7.5 and 7.6. Install CMake, download a fixed CUDA Samples release, verify its checksum, and build only the required target:

 

Sample release and checksum:

 

Sample output, abridged:

 

The connectivity matrix must show peer access for every expected GPU pair. Keep the full bandwidth and latency matrices when they will be used as a site baseline. The absolute values depend on software, topology, power state, and test release; the portable acceptance criteria are complete peer visibility and a successful exit without CUDA errors.

C.2 Additional NCCL profiles

Select additional NCCL profiles according to the question being answered.

Table 10.      Validation questions for selecting NCCL profiles

The result blocks below illustrate the output format. Bandwidth values vary by software and configuration; use successful completion, #wrong 0, and Out of bounds values: 0 OK as the portable acceptance criteria.

Run a two-GPU point-to-point profile:

 

Example result format:

 

Run four concurrent GPU pairs:

 

Example result format:

Run a fixed size sustained all reduce:

 

Example result format:

 

Pass is indicated when the command exits successfully, every GPU pair completes, GPU memory is released, and the kernel journal contains no new error.

Appendix D. Redfish monitoring and automation reference

Use Redfish to integrate the server with a monitoring or automation platform without relying on the BMC UI. This appendix covers authentication, resource discovery, inventory, health, sensors, logs, events, tasks, virtual media, one-time boot, PXE handoff, and support-data collection.

The examples were exercised against C880A M8 BMC firmware 4.0(1.260014). Discover member IDs and action targets on each system.

D.1 Authenticate and test the service root

On an Ubuntu management workstation, install the Redfish tools:

 

Expected result: APT completes without a dependency error and every version command identifies the installed utility.

Set the connection variables and reusable verified curl options:

 

Sample service root result, abridged:

 

Use the CA or chain that validates the certificate installed in Section 2.5. During initial certificate bootstrap only, -k can replace --cacert; return to the verified CURL array after the certificate is installed.

D.2 Discover implementation resource IDs

Discover resource IDs from each collection and follow their @odata.id links:

 

Sample output, abridged:

 

The examples in the table below use these discovered C880A M8 members.

Table 11.      Examples of implementation resource IDs

The literal resource IDs are part of this Redfish implementation and must be used as returned, including capitalization.

D.3 Capture inventory and health

The following commands collect an abridged baseline without changing the system:

Expected output, abridged:

 

Counts and model strings must match the purchased configuration. Use the host system summaries for CPU and memory. Use the GPU inventory resources for accelerator count. Evaluate the HGX baseboard by its current Status and the detailed HGX resources.

Interpret the main fields as shown in the table below.

Table 12.      Meanings of baseline fields

Use the detailed sensor and log resources when a rollup reports Warning or Critical.

Collect the complete firmware inventory:

 

Sample collection and representative members:

 

Pass is indicated when each member returns a version that matches the approved firmware set. Use the current resource health endpoints for state.

Collect the storage members exposed by the host System resource:

 

Use the BMC storage inventory or host storage tools for a configured boot logical device when its controller members are not exposed by Redfish.

D.4 Collect sensors and subsystem health

Discover the sensor collection:

 

Sample output, abridged:

 

Collect a compact status table from every member:

 

Example abridged sensor:

 

Not every sensor has a numeric reading. Preserve discrete state, health, and threshold fields in monitoring software. URL-encode sensor IDs that contain spaces or reserved characters.

Review the power and thermal subsystems separately:

 

Sample output, abridged:

 

Follow the links in PowerSupplies, Fans, and related collections for per-device readings. A healthy rollup does not replace per-member monitoring.

D.5 Collect logs and active conditions

Common Redfish log endpoints include:

 

Example:

 

Expected output structure, abridged:

 

An empty Members array is valid when the selected log has no entries. Severity can be OK, Warning, or Critical; retain all three.

Follow Members@odata.nextLink when present. A dashboard collector that reads only the first page can miss older or less severe events.

ServiceConditions is a Conditions array, not a member collection:

 

Follow every linked log and each unique origin:

 

Check the condition timestamp, linked log entry, and current origin-resource state. Keep current resource health and unresolved conditions as separate dashboard signals.

D.6 Monitor long-running tasks

Redfish actions can return 202 Accepted and a task URI in the Location response header. Use this bounded helper for the task examples in this appendix:

Call the helper with a relative task URI and a timeout:

 

Pass is indicated when the task reaches Completed with TaskStatus: OK. Treat a failed terminal state, a non-OK completed status, or the timeout as a failed operation.

D.7 Receive and test Redfish events

The EventService supports a Server-Sent Events stream and the SubmitTestEvent action on firmware 4.0(1.260014). Discover both from the service before use:

 

Start the listener in one terminal:

 

Keep this listener connected, or create an EventDestination subscription before calling SubmitTestEvent. With no active listener or subscription, the task ends in Exception with No Active Event Subscriptions.

In a second terminal, define a helper that submits one event and waits for its task:

 

Submit four message shapes, one at a time:

 

Each action returned:

 

Pass is indicated when the SSE listener receives all four event IDs with their matching MessageId and MessageArgs.

For persistent delivery, run an HTTP receiver that accepts Redfish event POSTs, then create an EventDestination:

 

Sample result:

 

Use the Location header as the subscription member URI. Submit the four test events above, waiting for each task before sending the next. The receiver should record four POST requests with the configured Context and matching event IDs.

Remove the test subscription:

 

Expected result:

 

Create the production EventDestination with the event receiver, registry filters, and retry policy selected for the monitoring platform.

D.8 Read and change the BMC configuration

 

Sample output:

 

Firmware 4.0(1.260014) requires a PATCH precondition. Read the current ETag before changing either resource:

 

The expected success status is HTTP 204. A request without If-Match returns HTTP 428 on firmware 4.0(1.260014).

After confirming that both site time sources return NTP responses from the management network, enable NTP:

 

Expected output:

 

Read the resource before every change. A PATCH that repeats values already applied can return HTTP 400 with SyncAgent.1.0.PatchValueAlreadyExists.

If synchronization fails, confirm DNS resolution, routing, firewall policy, and UDP port 123 reachability from the BMC management network. Correct the time before installing a production TLS certificate. ManagerNetworkProtocol reports whether NTP is enabled and which servers are configured. It does not expose a separate synchronization status. Poll Manager.DateTime and compare it with a trusted UTC reference.

D.9 Build a monitoring collector

Organize the collected data into four layers.

Table 13.      Layers for monitoring data

 

At each collection cycle:

1.     Fetch the system, manager, and chassis rollups.

2.     Fetch /redfish/v1/ServiceConditions and preserve every condition's severity, message ID, timestamp, origin, and linked log entry.

3.     Fetch the sensor members and normalize numeric and discrete readings.

4.     Fetch power and thermal rollups and their member collections.

5.     Fetch recent log entries, following pagination links.

6.     Collect firmware and software inventory on a slower schedule.

7.     Publish a collection timestamp, endpoint latency, and endpoint error list with the hardware data.

The following table shows examples of starting intervals. Adjust them for fleet size, BMC response time, and the monitoring service-level objective.

Table 14.      Examples of starting intervals

 

Treat an unreachable endpoint, stale data, timeout, or missing required collection as Unknown, not OK. Do not treat a null sensor health value as a failure by itself, and do not assume that an empty or old task collection represents an active fault.

The BMC also exposes TelemetryService:

 

Sample output, abridged:

 

Use the chassis sensor collection as the primary source for a near-real-time dashboard. Use TelemetryService only after the deployment defines and validates metric reports. Respect MinCollectionInterval; PT10M means 10 minutes and is not suitable for a 30-second metric-refresh loop.

D.10 Discover and insert virtual media

Inspect the BMC manager and action metadata:

 

Sample action metadata, abridged:

 

The action metadata allows the parameters shown in the table below.

Table 15.      Parameters for action metadata

 

Use the exact action targets returned by the manager:

 

Sample result: Both actions returned HTTP 200. The response body reports Ami.1.0.DelayInActionCompletion with severity OK and instructs the client to verify the updated property after approximately 4 to 5 seconds.

 

Re-query the standard collection:

 

Sample output:

 

Confirm that /redfish/v1/Managers/BMC/VirtualMedia/CD1 is present and advertises VirtualMedia.InsertMedia.

Firmware 4.0(1.260014) requires nonempty UserName and Password fields in the insert request, including when the source URL is otherwise readable without authentication. Use a read-only repository account.

 

Sample initial response:

 

The request returns HTTP 202 and a Location header containing the same TaskService URI as the response body. Extract and poll the task:

 

Expected terminal result:

 

Verify the media resource:

 

Successful abridged state:

 

Do not proceed when the task reports success but the media resource reports Inserted: false, ConnectedVia: "NotConnected", or RedirectionStatus: "Stopped - Unable to Open".

D.11 Stage one-time boot and monitor the reset task

Discover the future-state resource and reset action from the host system:

 

The action values on firmware 4.0(1.260014) are ForceOff, ForceRestart, FullPowerCycle, GracefulRestart, GracefulShutdown, On, and PowerCycle.

Read the Settings resource and capture its ETag:

 

After the ISO is inserted, stage the one-time CD target:

 

Expected result:

 

A PATCH without If-Match returns HTTP 428 on firmware 4.0(1.260014). The active System resource can continue to show Disabled and None before the restart; use the Settings resource to verify the staged values.

Read the captured power state. Use a graceful restart when the host is on, or power it on when it is off:

Sample terminal result:

 

Read the active System resource after the host returns:

 

Expected result after a normal boot:

 

Task completion confirms the reset action. Follow POST through KVM or the configured serial console. If the Settings resource still contains staged values after provisioning, PATCH it back to Disabled and None with its current ETag.

D.12 PXE and unattended-image handoff

Read the system, discover its Settings resource and reset action, and capture the current boot state:

 

Stage a one-time UEFI PXE boot and verify the Settings resource:

 

Expected result:

 

Use a graceful restart when the host is on, or power it on when it is off:

 

Redfish stages the generic Pxe target; this request does not select a specific provisioning NIC. Configure the intended interface first in firmware boot order before using this handoff. Use the KVM boot menu when a one-time, explicitly selected NIC is required. After PXE starts, confirm that the one-time override is consumed:

The active resource returns Disabled and None after the one-time override is consumed.

The PXE server, image, answer file, and completion callback are supplied by the deployment environment. Redfish provides the boot handoff, not the installer-completion signal.

D.13 Eject media and clean up

After the installer no longer needs the ISO:

 

Expected result: The action completes or creates a task. Re-querying CD1 must show Inserted: false, ConnectedVia: "NotConnected", and no active redirection. Ami.1.0.ActionNotExist means no redirection was active when the eject was requested.

Confirm Inserted: false, then disable remote media:

 

Expected result: HTTP 200 with Ami.1.0.DelayInActionCompletion. After approximately 4 to 5 seconds, Managers/BMC must report Oem.Ami.VirtualMedia.RMediaStatus: "Disabled" and the VirtualMedia collection can become empty.

If the BMC reports ActionInProgress, wait for ejection to complete, re-query CD1, and retry the disable action.

D.14 Generate Cisco BMC support data

Install the archive identification utility on the management workstation:

 

The Cisco BMC support data workflow uses the DiagnosticLog service. Discover the action and its accepted parameters instead of assuming that they are the same on every firmware release:

 

On BMC firmware 4.0(1.260014), the action target and representative parameter metadata are:

 

Use ALL when Cisco Support requests a complete Cisco BMC support bundle:

Expected response:

 

Poll the URI in the Location header. A typical task moves through New and Running before reaching Completed:

 

Representative running and terminal states:

 

Do not treat the initial 202 Accepted response as completion. Wait for the terminal task state.

After completion, enumerate the diagnostic entries:

 

Expected result: comm prints the URI added by the current operation. Use this set comparison instead of relying only on timestamps, because controller time can be incorrect. Retrieve the new member resource and read AdditionalDataURI:

 

Representative output:

 

Download the attachment from the returned URI:

 

Sample output:

 

D.15 Generate the HGX manager dump

The HGX BMC exposes a separate manager-level diagnostic collection. Discover its action metadata:

 

Sample parameter metadata:

 

Start the collection:

 

Expected result: The service returns 202 Accepted and a task URI. Poll that task as shown in Section D.6. After it completes, enumerate the manager dump entries:

 

Sample completed task:

 

Install the archive identification and Zstandard validation utilities:

 

Expected result: APT installs or confirms both utilities and the version commands complete successfully.

Retrieve the new entry and download its AdditionalDataURI with authenticated curl, as shown for the Cisco BMC bundle:

 

Representative entry metadata:

 

The attachment is Zstandard-compressed data even though the service can return a generic filename and application/json content type. Determine its format with file rather than from the HTTP content type.

Pass is indicated when the task reaches Completed, the entry provides an AdditionalDataURI, and zstd -t validates the downloaded archive. If the archive does not validate, collect the Cisco BMC support bundle from Section D.14 and open a Cisco support case.

D.16 Endpoint and action map

The examples in the table below use these discovered paths. Continue to discover them from parent resources so automation can detect changes.

Table 16.      Endpoints and use cases

Capture the endpoints shown in the table below.

Table 17.      Endpoints to capture for monitoring

Use timestamped directories and preserve HTTP status codes. Follow collection pagination and save the member resources referenced by each collection.

The common action targets in the table below are discovered from their parent resources.

Table 18.      Action targets discovered from parent resources

Read the action object and any linked @Redfish.ActionInfo resource before constructing the request.