Cisco UCS C880A M8 Rack Server Troubleshooting Guide

 

Revision History

Refer to the table below for changes made on this version of the Cisco UCS C880A M8 Rack Server Troubleshooting Guide.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table of Contents

Revision History. 2

Introduction.. 4

CPU SLED EXPLOSION VIEW... 5

Terminology. 9

Diagnostic Indicator. 10

3200W 50V Power Supply. 10

DC-SCM... 11

E1.S. 12

Running Diagnostic. 13

Introduction. 13

Create USB Drives from the Released ISO.. 13

User Manual. 14

Troubleshooting Hardware Issues. 21

Mother Board. 21

System Failure. 24

Server Management Software Issues. 30

Firmware Information. 30

Power Control. 33

Manual Fan Control. 36

BMC Time Collect and Configure. 37

BMC SEL Collect & Clear. 39

FRU Read and Write. 40

User Management. 41

Network Management. 43

Network Collection. 43

BMC IP Source and IP Address Configuration. 44

Hostname Configuration. 45

Maintenance. 46

Troubleshooting Operating System Issues. 53

How to Install the Operation System using Virtual Media - Ubuntu 24.04. 53

Nvidia GPU.. 56

BIOS Related Issue. 59

System Power-on issue. 59

POST issue. 59

Boot with no display. 59

OS boot issue. 60

Error occur after a BIOS setting is changed. 60

Failure occurs during ROM flash via BMC web. 61

See the SERR or PERR error on SEL. 61

PCI device not found / found but not plugged. 61

PCIe Slot Information. 66

POST code. 70

Appendix A: The way to output BIOS log.. 78

Documentation Feedback. 79

Legal Information.. 79

 

Introduction

 

 

CPU SLED EXPLOSION VIEW

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Terminology

 

 

Diagnostic Indicator

3200W 50V Power Supply

 

 

 

 

DC-SCM

 

 

 

 

 

 

 

 

 

 

E1.S

 

Running Diagnostic

 

Introduction

The Self-Diagnostic Test is used to perform function coverage testing. Running on Linux, it is a UI framework to merge/involve different test diagnostic programs including Open-Source tools (included BSD/GPL v2.0 license), Vendor Tools/Drivers, and some diagnostic tools from ODM.

The user can utilize selftest.py to merge various diagnostic utilities, packages, and scripts for conducting function testing. ODM can set up and build a package that includes selftest.py and diagnostic utilities for customers to perform interaction testing.

Create USB Drives from the Released ISO

 

Creating a bootable USB drive from an ISO image can be achieved through a variety of methods, depending on your operating system. We recommend using software tools to create a USB drive, such as Rufus.

We need two partitions: the first partition to write the ISO image, and the second partition to save log data.

Requirements

Rufus 4.5

Write the ISO Image to the Target Device

Step 1. Select target Device (Your USB KEY > 32GB).

Step 2. Select source iso.

Step 3. Partition scheme select GPT.

Step 4. Click Start.

 

User Manual

The program is executed with this test command by manual

Execute Self-Diagnostic Test command follow step as below:

Step 1. Left click Terminal.

 

 

 

 

 

 

 

 

 

 

Step 2. Type “cd /” and press Enter → Type “ls” and press Enter (check selfdiag.sh exists) → Type “sudo ./selfdiag.sh” and press Enter.

 

 

 

 

 

 

 

 

Step 3. Successfully executed program.

 

 

 

 

 

 

 

 

 

 

 

 

 

PS. If the system logs out automatically. You can use the following information to log in again.

User: The account is the Name you entered during installation.

Password: The Password is you entered during installation.

Main window

System Summary

It shows your “system”, “memory”, “CPU”, “Hard Disk”

 

 

Test

Follow steps as below:

1.      Left click “Test” → Select the testcase to test → select which SKU you want to test

 

 

 

 

 

 

 

Main window

System Summary

It shows your “system”, “memory”, “CPU”, “Hard Disk”.

Test

Follow steps as below:

1.      Left click “Test” → Select the testcase to test → select which SKU you want to test.

2.   Press Start button to start test during testing

 

 

 

 

 

 

 

 

 

 

 

Remark: Perform all test items which defined in “L10_74-XXXXXX.xml…”

 

 

 

 

 

 

 

 

Test Item: BMC Login Info

This test case will prompt a window prompting the user to input the BMC ID and password, which is used for redfish command related test case.

Test Status

Check test status and you can also press Cancel button to stop test during testing.

 

Test Log

Log – Login for unlocking Log function

 

Log – Input Username and Password for Login

Username: admin

Password: superuser

 

 

Unzip password: superuser

Log - Process Log

Log - Error Log

 

You can initially judge the possible problematic part through the text in [] in front of each test item. For detailed Fail items, please refer to the Test Log to DEBUG.

Troubleshooting Hardware Issues

Mother Board

G3 to S5

After providing AC power, you can refer to postcode debug LED on MB and check if LED has been lighting on or not.

 

If yes, please refer to table 1. and check the corresponding post code number.

If not, please re-program MB CPLD FW and check if the scenario can be solved or not.

Table 1.           G3àS5 Postcode

FPGA postcode error code check

When system post hangs, please note the number on postcode debug LED and refer to Table 2. below to check the root cause.

Table 2.           Error code

S5 to S0(POST)

Please observe the postcode to understand the current boot progress and status.

Table 3.           S5àS0 Postcode

System Failure

Troubleshooting under OS issue

Loss PCIE Device Check

Prerequisite

PCIE Device does not detect under OS.

Troubleshooting steps:

1.      Under OS and execute the command to check which device lost & corresponding slot.

Location

●     Check PCIe Tree and device address: lspci -tv | less

●     Check which slot ID cannot recognize:

lspci -v | grep -i “slot:” |sort “

2.      After get the lost slot location, apply below system PCIE topology Mapping table to check the path of lost device. (Figure 4-1 to 4-3 and Table 4-1 to 4-3)

3.      Referring to the system cable routing to check the connection between cable/board and the device through the cable ID.

Multiple devices cannot be detected due to cable faults or poor plugging Therefore, we recommend applying HW Check first.

 

 

 

 

 

 

 

 

 

 

 

 

Table 4.           PCIE Bus Mapping Table

 

 

 

 

 

 

 

 

 

 

Table 5.           CPU Sled Mapping Table

 

Table 6.           B300 Baseboard Mapping Table

●     GPU Mapping Table

 

●     CPU SLED Upper Layer with E1.S & Riser Cage Modules

●     CPU SLED Lower Layer with M.2, OCP & DCSCM Modules

 

 

Cannot detect single E1.S device

1.      Re-plug the E1.S.

2.      If you still cannot read the device, change the new E1.S SSD.

3.      3.  If still cannot read, please re-plug the E1.S Cable Module and make sure the cable and connector are no bent pins/ pollution.(*1)

4.      4.  If there are no bent pins or pollution, try to change a new E1.S Module.(*1)

 

Note:       *1 Please refer to User Manual – Chapter 3 Assembly SOP.

Server Management Software Issues

Firmware Information

BMC Firmware Revision

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Managers/BMC

 

Property Name: FirmwareVersion

Description: Represents the revision of the BMC

 

This figure provides a reference example for the BMC Firmware Revision results.

IPMI:

In-Band:

ipmitool mc info

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} mc info

 

This figure provides a reference example for the BMC Firmware Revision results.

BIOS Firmware Revision

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Systems/DGX

 

Property Name: BiosVersion

Description: Represents the revision of the BIOS

 

This figure provides a reference example for the BIOS Firmware Revision results.

DC-SCM CPLD Firmware Revision

 

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/CPLDDCSM_0

 

Property Name: Version

Description: Represents the revision of the DC-SCM CPLD

 

This figure provides a reference example for the DC-SCM CPLD Firmware Revision results.

 

MB CPLD Firmware Revision

 

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/CPLDMB_0

 

Property Name: Version

Description: Represents the revision of the MB CPLD

 

This figure provides a reference example for the MB CPLD Firmware Revision results.

E1.S CPLD Firmware Revision

 

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/CPLDE1SBP_0

 

Property Name: Version

Description: Represents the revision of the E1.S CPLD

 

This figure provides a reference example for the E1.S CPLD Firmware Revision results.

 

Backplane CPLD Firmware Revision

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/CPLDBACK_0

 

Property Name: Version

Description: Represents the revision of the Backplane CPLD

 

This figure provides a reference example for the BackPlane CPLD Firmware Revision results.

PCIe Switch Firmware Revision

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/PCIeSwitch_0

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/PCIeSwitch_1

 

Property Name: Version

Description: Represents the revision of the PCIe Switch

 

This figure provides a reference example for the PCIe Switch 0 Firmware Revision results.

This figure provides a reference example for the PCIe Switch 1 Firmware Revision results.

 

GPU Firmware Revision

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/UpdateService/FirmwareInventory/{GPU_Instance}

 

GPU_Instance:

HGX_FW_BMC_0,

HGX_FW_ConnectX_0 ~ HGX_FW_ConnectX_7,

HGX_FW_ConnectX_SMA_0 ~ HGX_FW_ConnectX_SMA_3,

HGX_FW_ERoT_BMC_0,

HGX_FW_ERoT_FPGA_0,

HGX_FW_ERoT_NVLinkManagementNIC_0,

HGX_FW_ERoT_NVSwitch_0, ~ HGX_FW_ERoT_NVSwitch_1,

HGX_FW_FPGA_0,

HGX_FW_GPU_0 ~ HGX_FW_GPU_7,

HGX_FW_NVLinkManagementNIC_0,

HGX_FW_NVSwitch_0 ~ HGX_FW_NVSwitch_1,

HGX_FW_SXM_SMA_0 ~ HGX_FW_SXM_SMA_7,

HGX_InfoROM_GPU_0 ~ HGX_InfoROM_GPU_7

 

This figure provides a reference example for the HGX_FW_BMC_0 Firmware Revision results.

 

Power Control

Power Status

 

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Systems/DGX

Property Name: PowerState

Description: Represents the power status of the system

 

This figure provides a reference example for the power status.

IPMI:

In-Band:

ipmitool power status

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} power status

 

This figure provides a reference example for the power status.

 

Power On

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/Systems/{System_Instance}/Actions/ComputerSystem.Reset -H "Content-Type:application/json" -d '{"ResetType": "On"}'

 

This figure provides a reference example for executing power-on.

 

 

 

 

 

 

IPMI:

In-Band:

ipmitool power on

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} power on

 

This figure provides a reference example for executing power-on.

Power Off

 

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/Systems/{System_Instance}/Actions/ComputerSystem.Reset -H "Content-Type:application/json" -d '{"ResetType": "ForceOff"}'

 

This figure provides a reference example for executing power-off.

 

 

 

 

 

 

IPMI:

In-Band:

ipmitool power off

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} power off

 

This figure provides a reference example for executing power-off.

 

Power Cycle

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/Systems/{System_Instance}/Actions/ComputerSystem.Reset -H "Content-

Type:application/json" -d '{"ResetType": "PowerCycle"}'

 

This figure provides a reference example for executing power-cycle.

 

IPMI:

In-Band:

ipmitool power cycle

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} power cycle

 

This figure provides a reference example for executing power-cycle.

 

 

AC Cycle

IPMI:

In-Band:

ipmitool raw 0x3c 0x99

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} 0x3c 0x99

 

This figure provides a reference example for executing ac-cycle.

 

 

Manual Fan Control

IPMI:

In-Band:

ipmitool raw 0x30 0x20 0x01 {PWM}

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} raw 0x30 0x20 0x01 {PWM}

 

Note: The PWM range 0x14 to 0x64 in hexadecimal represents the range 20 to 100 in decimal.

This figure provides a reference example for executing manual fan control.

 

 

FAN Configurations Diagram

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

BMC Time Collect and Configure

BMC Date & Time Collection

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Managers/BMC

 

Property Name: DateTime

Description: Represents the current date and time of the BMC.

This figure provides a reference example for date and time of the BMC.

IPMI:

In-Band:

ipmitool sel time get

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} sel time get

 

This figure provides a reference example for date and time of the BMC.

 

BMC Date & Time Configure

Redfish:

1.      Disable NTP

curl -sku {user}:{password} -X PATCH https://{BMC IP}/redfish/v1/Managers/BMC/NetworkProtocol -H 'content-type: application/json' -H 'If-None-Match: "123"' -d '{"NTP":{"ProtocolEnabled":false}}'

2.      Configure BMC Date & Time

curl -sku {user}:{password} -X PATCH https://{BMC IP}/redfish/v1/Managers/BMC -H 'content-type: application/json' -d '{"DateTime": "yyyy-mm-ddThh:mm:ss+00:00"}'

This figure provides a reference example for executing BMC Date & Time Configuration.

 

IPMI:

In-Band:

ipmitool sel time set "mm/dd/yyyy hh:mm:ss"

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} sel time set "mm/dd/yyyy hh:mm:ss"

 

This figure provides a reference example for executing BMC Date & Time Configuration.

BMC Time Zone Collection

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Managers/BMC

 

Property Name: DateTimeLocalOffset

Description: Represents the current time zone of the BMC.

This figure provides a reference example for the time zone of the BMC.

BMC Time Zone Configure

Redfish:

1.      Disable NTP

curl -sku {user}:{password} -X PATCH https://{BMC IP}/redfish/v1/Managers/BMC/NetworkProtocol -H 'content-type: application/json' -H 'If-None-Match: "123"' -d '{"NTP":{"ProtocolEnabled":false}}'

2.      Configure BMC Date & Time

curl -sku {user}:{password} -X PATCH https://{BMC IP}/redfish/v1/Managers/BMC -H 'content-type: application/json' -d '{"DateTime": "yyyy-mm-ddThh:mm:ss+00:00"}'

 

This figure provides a reference example for executing BMC time zone Configuration.

 

BMC SEL Collect & Clear

BMC SEL Collection

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Managers/BMC/LogServices/SEL/Entries

 

This figure provides a reference example for BMC SEL collection.

IPMI:

In-Band:

ipmitool sel list

or

ipmitool sel elist

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} sel list

or

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} sel elist

 

This figure provides a reference example for BMC SEL collection.

BMC SEL Clear

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/Managers/BMC/LogServices/SEL/Actions/LogService.ClearLog -H "Content-Type:application/json" -d {}

This figure provides a reference example for executing BMC SEL clear.

IPMI:

In-Band:

ipmitool sel clear

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} sel clear

 

This figure provides a reference example for executing BMC SEL clear.

 

FRU Read and Write

These IDs are used to access and retrieve FRU information.

 

FRU Read

IPMI:

In-Band:

ipmitool fru print {FRU ID}

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} fru print {FRU ID}

This figure provides a reference example for BMC FRU collection.

FRU Write

In-Band:

ipmitool fru write {FRU ID} {FRU bin file}

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} fru write {FRU ID} {FRU bin file}

 

This figure provides a reference example for executing BMC FRU write.

 

User Management

User Collection

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/AccountService/Accounts/{Account ID}

This figure provides a reference example for collecting BMC user account.

Add User

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/AccountService/Accounts -H "Content-Type:application/json" -d '{"Enabled": {User Access}, "Password": "{Password}", "UserName": "{Username}", "RoleId": "{Privilege}"}'

 

Username: The username for the account.

Password: The password for this account

Privilege: The role for this account. Value: Administrator, Operator, ReadOnly.

User Access: It indicates whether an account is enabled. If `true`, the account is enabled and the user can log in. If `false`, the account is disabled and, in the future, the user cannot log in

This figure provides a reference example for creating BMC user account.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

User Configuration

Redfish:

curl -sku {user}:{password} -X PATCH https://{BMC IP}/redfish/v1/AccountService/Accounts/{Account_Instance} -H "Content-Type:application/json" -H 'If-Match:*' -d '{"{Property}": "{Value}"}'

 

Property:

Password: Password to be modified

RoleId: RoleId to be disabled or enabled

Enabled: It indicates whether an account is enabled.

This figure provides a reference example for modifying BMC user account setting.

 

Delete User

Redfish:

curl -sku {user}:{password} -X DELETE https://{BMC IP}/redfish/v1/AccountService/Accounts/{Account_Instance} -H "Content-Type:application/json" -H 'If-Match:*'

This figure provides a reference example for deleting BMC user.

Network Management

Network Collection

Redfish:

curl -sku {user}:{password} -X GET https://{BMC IP}/redfish/v1/Managers/BMC/EthernetInterfaces/eth0

This figure provides a reference example for BMC Network collection.

 

 

 

BMC IP Source and IP Address Configuration

Redfish:

curl -sku {user}:{password} -X PATCH https://{BMC IP}/redfish/v1/Managers/BMC/EthernetInterfaces/eth0  -H 'Content-Type: application/json' -H 'If-Match:*'  -d '{"DHCPv4": {"DHCPEnabled": false},"IPv4StaticAddresses": [{"Address": "'{New BMC IP}'","Gateway": "{GATEWAY}","SubnetMask": "{SUBNETMASK}"}]

}'

This figure provides a reference example for executing change IP source to 'static' and change IP address.

 

 

 

 

 

 

 

 

After changing IP source and changing IP address.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

●     Configure IP source to DHCP

 

Redfish:

curl -sku {user}:{password} -X PATCH https:// {BMC IP}/redfish/v1/Managers/BMC/EthernetInterfaces/eth0 -H 'Content-Type: application/json' -H 'If-Match:*' -d '{"DHCPv4":{"DHCPEnabled": true}}'

 

This figure provides a reference example for executing change IP source to 'DHCP'.

 

 

 

 

 

 

Hostname Configuration

Redfish:

curl -sku {user}:{password} -X PATCH https:// {BMC IP}/redfish/v1/Managers/BMC/EthernetInterfaces/eth0 -H 'content-type: application/json' -H 'If-Match:*' -d '{"FQDN":"{New_Hostname}.in.megatrends.com","HostName": "{New_Hostname}"}'

 

This figure provides a reference example for executing change hostname.

 

 

 

Maintenance

BMC Firmware Update

●     Redfish

curl -g --http0.9 -ku {user}:{password} -X POST "https://{BMC IP}/redfish/v1/UpdateService/upload" -F "UpdateFile=@{Firmware Image Path};type=application/octet-stream" -F 'UpdateParameters={"Targets":["/redfish/v1/UpdateService/FirmwareInventory/BMC"],"ForceUpdate":true};type=application/json'

 

Note:       The firmware image must be in the .fwpkg format.

This figure provides a reference example for executing BMC firmware update.

Monitoring BMC Firmware Update Progress:

curl -ku {user}:{password} -X GET https://{BMC IP}/redfish/v1/TaskService/Tasks/{Task ID}

 

Note:       The Task ID can be retrieved from the 'Location' field in the response after executing the BMC firmware update.

This figure provides a reference example for checking BMC firmware update progress.

 

●     Web UI

Please log in to the BMC web and go to the Maintenance > Firmware Update.

Click "Choose File" to select the BMC Firmware Image, check "Force Update".

Click 'Start firmware update' to begin the BMC firmware update process. Once the update is complete, a message stating 'Firmware update is completed' will appear, and the BMC will automatically reboot.

BIOS Firmware Update

Web UI

Perform the following:

1.        Login BMC Web UI.

2.        Go to Maintenance > Firmware Update.

 

3.        Select the BIOS firmware image and tick "Force Update".

4.        Click Start firmware update to begin the BIOS firmware update process.

Redfish Firmware Update

1.        Follow the instructions below to execute the BIOS firmware update.

curl --http1.0 -k -X POST --user <Account>:<Password> -H 'Content-Type:multipart/form-data' --insecure https://<BMC IP>/redfish/v1/UpdateService/upload -F 'UpdateFile=@<bios_image.fwpkg>' -F 'UpdateParameters={"Targets":{}};type=application/json' | jq

Note:       The firmware image must be in the .fwpkg format

2.        Follow the instructions below to check the BIOS firmware update process.

curl --http1.0 -k -X GET --user <Account>:<Password> -H 'Content-Type:multipart/form-data' --insecure https://<BMC IP>/redfish/v1/TaskService/Tasks/<Task Id> | jq

DC-SCM CPLD Firmware Update

Redfish:

curl -g --http0.9 -ku {user}:{password} -X POST "https://{BMC IP}/redfish/v1/UpdateService/upload" -F "UpdateFile=@{Firmware Image Path};type=application/octet-stream" -F 'UpdateParameters={"Targets":["/redfish/v1/UpdateService/FirmwareInventory/ CPLDDCSM_0"],"ForceUpdate":true};type=application/json'

 

Note:       The firmware image must be in the .fwpkg format.

Monitoring DC-SCM CPLD Firmware Update Progress:

curl -ku {user}:{password} -X GET https://{BMC IP}/redfish/v1/TaskService/Tasks/{Task ID}

 

Note:       The Task ID can be retrieved from the 'Location' field in the response after executing the DCSCM CPLD firmware update.

●     Web UI

Refer to Section WebUI Update Method and select DC-SCM CPLD Firmware Image.

MB CPLD Firmware Update

Redfish:

curl -g --http0.9 -ku {user}:{password} -X POST "https://{BMC IP}/redfish/v1/UpdateService/upload" -F "UpdateFile=@{Firmware Image Path};type=application/octet-stream" -F 'UpdateParameters={"Targets":["/redfish/v1/UpdateService/FirmwareInventory/ CPLDMB_0"],"ForceUpdate":true};type=application/json'

 

Note:       The firmware image must be in the .fwpkg format.

Monitoring MB CPLD Firmware Update Progress:

curl -ku {user}:{password} -X GET https://{BMC IP}/redfish/v1/TaskService/Tasks/{Task ID}

 

Note:       The Task ID can be retrieved from the 'Location' field in the response after executing the MB CPLD firmware update.

●     Web UI

Refer to WebUI Update Method and select MB CPLD Firmware Image.

E1.S CPLD Firmware Update

Redfish:

curl -g --http0.9 -ku {user}:{password} -X POST "https://{BMC IP}/redfish/v1/UpdateService/upload" -F "UpdateFile=@{Firmware Image Path};type=application/octet-stream" -F 'UpdateParameters={"Targets":["/redfish/v1/UpdateService/FirmwareInventory/ CPLDE1SBP_0"],"ForceUpdate":true};type=application/json'

 

Note:       The firmware image must be in the .fwpkg format.

Monitoring E1.S CPLD Firmware Update Progress:

curl -ku {user}:{password} -X GET https://{BMC IP}/redfish/v1/TaskService/Tasks/{Task ID}

 

Note:       The Task ID can be retrieved from the 'Location' field in the response after executing the E1.S CPLD firmware update.

●     Web UI

Refer to Section WebUI Update Method and select E1.S CPLD Firmware Image.

BackPlane CPLD Firmware Update

Redfish:

curl -g --http0.9 -ku {user}:{password} -X POST "https://{BMC IP}/redfish/v1/UpdateService/upload" -F "UpdateFile=@{Firmware Image Path};type=application/octet-stream" -F 'UpdateParameters={"Targets":["/redfish/v1/UpdateService/FirmwareInventory/ CPLDBACK_0"],"ForceUpdate":true};type=application/json'

 

Note:       The firmware image must be in the .fwpkg format.

Monitoring BackPlane CPLD Firmware Update Progress:

curl -ku {user}:{password} -X GET https://{BMC IP}/redfish/v1/TaskService/Tasks/{Task ID}

 

Note:       The Task ID can be retrieved from the 'Location' field in the response after executing the E1.S CPLD firmware update.

●     Web UI

Refer to Section WebUI Update Method and select BackPlane CPLD Firmware Image.

HGX Firmware Update

Redfish:

curl -g --http0.9 -ku {user}:{password} -X POST "https://{BMC IP}/redfish/v1/UpdateService/upload" -F "UpdateFile=@{Firmware Image Path};type=application/octet-stream" -F 'UpdateParameters={"Targets":["/redfish/v1/UpdateService/FirmwareInventory/ HGX_0"],"ForceUpdate":true};type=application/json'

Note:       The firmware image must be in the .fwpkg format.

Monitoring HGX Firmware Update Progress:

curl -ku {user}:{password} -X GET https://{BMC IP}/redfish/v1/TaskService/Tasks/{Task ID}

 

Note:       The Task ID can be retrieved from the 'Location' field in the response after executing the E1.S CPLD firmware update.

●     Web UI

Refer to Section WebUI Update Method and select HGX Firmware Image.

BMC Reset

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/Managers/BMC/Actions/Manager.Reset -H 'content-type: application/json' -d '{"ResetType":"ForceRestart"}'

This figure provides a reference example for executing BMC reset.

IPMI:

In-Band:

ipmitool mc reset

 

Out-of-Band:

ipmitool -I lanplus -H {BMC IP} -U {User} -P {Password} mc reset

This figure provides a reference example for executing BMC reset.

 

BMC to Factory Default

Redfish:

curl -sku {user}:{password} -X POST https://{BMC IP}/redfish/v1/Managers/BMC/Actions/Manager.ResetToDefaults" -H 'content-type: application/json' -d '{"ResetType":"ResetAll"}'

This figure provides a reference example for executing BMC to Factory Default.

 

 

 

 

 

 

Preserve Configuration

To check if the configuration is preserved:

Redfish:

curl -sku {user}:{password} https://{BMC IP} /redfish/v1/Managers/BMC/Oem/Ami/PreserveConfigurations

This figure provides a reference example for getting preserve configuration.

To set the preserve configuration properties:

●     Redfish:

curl -sku {user}:{password} -X PATCH  -H 'content-type: application/json' -H 'If-Match:*' -d ' {"PreserveConfigurations": "{properties} ": {true or false}}' https://{BMC IP} /redfish/v1/Managers/BMC/Oem/Ami/PreserveConfigurations

This figure provides a reference example for editing preserve configuration.

Troubleshooting Operating System Issues

This section helps you troubleshoot operating system issues in your system.

NOTE: If the problem persists, contact Wistron Technical Support for further assistance.

How to Install the Operation System using Virtual Media - Ubuntu 24.04

Steps:

A.     Prerequisites

1.      Ensure that the server supports the operating system you intend to install.

2.      Make sure you have the OS source media available for the server.

3.      Prepare a storage location where you can store the necessary installation files.

B.      Access the BMC WebUI interface by entering the server's IP address in a web browser.

Note:       The default username is admin and Password is admin.

C.      Go to Image Redirection to Mount OS Source media into the Virtual Media drive

1.      Select Image Redirection -> Remote Images

2.      Click here to go to Media General Settings.

3.      Click and Enable Remote Media Support.

3.        Share Type for CD/DVD by nfs, cifs, https, smbe.

4.        Set and Save.

4.      After Setup the share type, go back to Remote Media, and then started the media.

5.      Reboot the system and boot into BIOS to select virtual media CD/DVD and install operation system. Save the Changes, Reboot the Server, and Boot into the OS Installation Image.

6.      In the BIOS setup utility or boot menu, navigate to the "Save & Exit" section or a similar option.

7.      Select the "UEFI: Linux File-CD Gadget 0515" or similar entry from the available boot devices.

8.      Save the changes and exit the BIOS setup utility or boot menu.

9.      The server will now boot from the selected boot device, which is the OS installation image on the virtual media drive.

 

 

 

 

 

 

 

 

 

 

 

 

 

Once the installation completes, the server will reboot again, this time booting from the installed operating system on the server's local disk.

●     GPU Mapping Table (Please refer to System Failure > Loss PCIE Device check).

 

Nvidia GPU

Installing and installing Nvidia BLACKWELL HGX B300 NVL8 driver

 

Solution:

Ubuntu 24.04 (Noble Numbat) works well with these drivers via the official repositories or NVIDIA's CUDA toolkit. Below are step-by-step instructions using the recommended methods.

Prerequisites

Ensure your system has an NVIDIA B300 GPU installed and recognized: Run lspci | grep -i nvidia to confirm (look for something like "VGA compatible controller: NVIDIA Corporation Device [3182]" where 3182 is a Blackwell device ID).

1.      Update your system:

sudo apt update && sudo apt upgrade -y

2.      Install required build tools (for DKMS kernel modules):

sudo apt install build-essential linux-headers-$(uname -r) dkms

3.      Disable Secure Boot in your BIOS/UEFI if enabled (Blackwell drivers often require it off for installation).
If using Nouveau (open-source driver), blacklist it:

sudo bash -c "echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf"

sudo bash -c "echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf"

sudo update-initramfs -u

sudo reboot

●     Method 1: Using Ubuntu's Built-in "Additional Drivers" Tool (Easiest for GUI Users)

This auto-detects your B300 and installs the recommended proprietary driver from Ubuntu's repos (typically 575+ on 24.04.2+).

1.        Open "Software & Updates" from the Applications menu.

2.        Go to the Additional Drivers tab.

3.        Wait for it to scan your hardware—it should list options like nvidia-driver-575 or nvidia-driver-590 (select the highest version with "recommended").

4.        Select the driver and click Apply Changes.

5.        Enter your password and let it install (it may prompt a reboot).

6.        Reboot: sudo reboot.

7.        Verify: Run nvidia-smi—it should show your B300 GPU details.

If no Blackwell-compatible driver appears, proceed to Method 2.

●     Method 2: Command-Line Installation via ubuntu-drivers (Recommended for Servers/CLI)

This tool automatically selects and installs the best driver for your B300.

1.      Install the tool if needed:

sudo apt install ubuntu-drivers-common

2.      List available drivers:

ubuntu-drivers devices

Look for your B300 and the recommended driver (e.g., nvidia-driver-590).

3.      Install the recommended one:

sudo ubuntu-drivers autoinstall

Or specify: sudo ubuntu-drivers install nvidia:590 (replace 590 with the version from step 2).

4.      Reboot: sudo reboot.

5.      Verify: nvidia-smi.

For open kernel modules (often required for Blackwell stability): Use nvidia-driver-590-open instead.

●     Method 3: Installing via NVIDIA's CUDA Repository (For AI/HPC Users)

1.        Add the NVIDIA CUDA repo.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb

sudo dpkg -i cuda-keyring_1.1-1_all.deb

sudo apt update

2.        Install the driver + CUDA.

sudo apt install cuda-drivers nvidia-driver-590  # Or latest version

For open modules: sudo apt install cuda-drivers-open nvidia-driver-590-open.

3.        Reboot: sudo reboot.

4.        Verify: nvidia-smi (shows GPU) and nvcc --version (for CUDA).

 

Troubleshooting

●     "No devices found" in nvidia-smi: Ensure Nouveau is blacklisted and reboot. If persists, purge old drivers: sudo apt purge '^nvidia-*' && sudo apt autoremove, then reinstall.

●     Kernel mismatch: Update to the latest kernel (6.8+): sudo apt install linux-generic-hwe-24.04.

●     Blackwell-specific issues: Use open drivers (-open suffix) as some Blackwell firmware mandates them. Check NVIDIA forums for beta drivers if needed.

●     Boot loop/black screen: Boot to recovery mode, purge drivers, and try an older version like 575.

●     Test: Run prime-select query (for laptops) or glxinfo | grep NVIDIA for OpenGL.

 

For the absolute latest drivers, download the .run file from NVIDIA's site (e.g., version 590.xx for Blackwell) and install manually:

chmod +x NVIDIA-Linux-x86_64-590.xx.run

sudo ./NVIDIA-Linux-x86_64-590.xx.run –dkms

But stick to repo methods for easier updates.

If issues persist, provide output from lspci -v | grep VGA and dmesg | grep nvidia for more help.

How to isolate HW Failures on Nvidia BLACKWELL HGX B300 NVL8 products

Solution:

If you encounter the hardware failure of NVIDIA BLACKWELL HGX B300 NVL8, you can use partnerdiag diagnostics to isolate HW failures.

1.      Installation

Please follow these steps:

1.        Copy 629-PPPPP-KKKK-FLD-DDDDD.tgz to your system.

2.        tar xfz 629-PPPPP-KKKK-FLD-DDDDD.tgz.

2.      Running partnerdiag diagnostics

Please follow these steps:

1.        cd 629-PPPPP-KKKK-FLD-DDDDD

2.        Run partner ddiag

- Mfg  : sudo ./partnerdiag --mfg --run_spec=spec_blackwell-hgx-b300-nvl8_partner_mfg.json --run_on_error --no_bmc

  - Field: sudo ./partnerdiag --field --run_on_error --no_bmc

3.        PASS/FAIL/RETEST will be displayed when partnerdiag finishes execution.

4.        Run ./partnerdiag --help for more details on options.

If issues persist, provide logs to NVIDIA AE team for more help.

BIOS Related Issue

When an issue occurs (error or loss), please refer to the following sections for initial troubleshooting.

 

System Power-on issue

Symptom:

●     The system has no response when the power button is pressed.

●     The system power/health LED is abnormal.

 

Action:

●     Check if there’s improperly seated component (e.g., PSU, CPU, DIMMs, cables..).

●     Ensure the system FWs comply with the FW table. If not, please do firmware update.

●     If the issue still exists, please contact R&D.

 

POST issue

 

Symptom:

●     The system hangs in POST.

●     The system can’t complete POST.

 

Action:

●     Check the 80 port and POST error messages, refer to POST code table to do the initial troubleshooting.

●     Shut down the system and try to break the    system    down to the minimum hardware configuration first, then power on the system.

●     Clear CMOS (power off the system and remove the CMOS battery, discharge for 30 seconds, and then reinstall it) and reboot the system.

●     Ensure the system FWs comply with the FW table. If not, please do firmware update.

●     If the issue still exists, please contact R&D.

 

Boot with no display

 

Symptom:

●     The system boots, but has no video output.

 

Action:

●     Unplug mini-DP and plug again.

●     If the issue still exists, please contact R&D.

 

OS boot issue

 

Symptom:

●     The    system    can’t boot an installed OS.

 

Action:

●     Check if the OS is corrupted, replace it to another OS device.

●     System only supports UEFI mode, check if user uses the legacy OS.

●     Check if secure boot is enabled, if yes, please disable it in BIOS menu and reboot the system.

●     If system can’t boot via OS path in Boot Override, try from the embedded EFI SHELL as below:

1.      Find your OS partition and access it.

ex. Shell> fs1:

2.      Access to the EFI folder.

ex. FS1> cd EFI

3.      Access to the BOOT folder.

ex. FS1\EFI> cd BOOT

4.      Execute the OS boot path

ex. FS1\EFI\BOOT> BOOTX64.EFI

 

Error occur after a BIOS setting is changed

 

Symptom:

●     The    system    can’t boot normally after settings were changed.

Action:

●     Clear CMOS to load default settings, and reboot system.

●     Flash BIOS to restore the system to default settings.

●     Check the system logs to determine the changes made, and change the settings back to the original configuration.

●     If the issue still exists, please contact R&D.

 

Failure occurs during ROM flash via BMC web

 

Symptom:

●     Any abnormal/error windows pop up during flash.

 

Action:

●     Check if the BMC connection is normal.

●     Check if the network cable is loose.

●     Make sure to use the right ROM file.

●     If there’s an interrupted during a ROM flash, or the ROM image is corrupted and the server does not start, re-flash the BIOS.

●     If user still can't access BMC web to flash BIOS, please use copy machine to flash it (e.g., SF600)

●     If the issue still exists, please contact R&D.

 

See the SERR or PERR error on SEL

Symptom:

●     See the PERR/SERR error the same as example below:

Action:

●     Find the EventData that we can decode to find which PCIe slot/device triggered the event.

●     The way to decode:

So, 8220 indicates the PCIe device/slot 82/04/00.

●     Please contact R&D and inform the PCIe slot/device that triggered the event.

 

PCI device not found / found but not plugged

Symptom:

●     PCI device plugged but not found in PCI Subsystem Settings.

●     PCI device found in PCI Subsystem Settings but not plugged in the system.

Action:

●     Prepare the following information and contact R&D.

1.      Take screenshots of PCI subpage.

●     Enter PCI Subsystem Settings (Setup Page→Advanced menu)

 

●     Take screenshots of the subpage.

 

●     Enter and take screenshots of the wrong-detected PCI device, example:

 

2.      Dump PCIe Information from Linux OS:

●     Make the following commands to dump lspci and save files

sudo lspci -tv > FILENAME_tv.log

sudo lspci -vvv > FILENAME_vvv.log

sudo lspci -xxx > FILENAME_xxx.log

●     Example for tv.log

 

 

 

 

 

 

 

 

 

 

 

●     Example for vvv.log

 

 

●     Example for xxx.log

 

 

 

 

 

 

 

 

 

 

 

3.      Dump PCIe Information from UEFI: Built-in EFI Shell.

●     Switch to the USB. Make the following command to dump pci and save file

FS1:

pci > FILENAME_pci.log

●     Example for pci.log

 

PCIe Slot Information

Fixed PFA for PCIe Slots

When an issue occurs (error or loss), please refer to the table below to find out the problematic devices.

E1.S

Table 7.           E1.S PCIE Bus Mapping Capture

BF3

Table 8.           BF3 PCIE Bus Mapping Capture

OCP

Table 9.           OCP PCIE Bus Mapping Capture

GPU

Table 10.       GPU Bus Mapping Capture

PCIE information in the Setup Page

The GPU server provides the status of all PCIe slots so that users can easily see if a PCIe device is plugged in or not.

POST code

AMI BIOS POST code

SEC Phase

 

PEI Phase

 

DXE Phase

 

 

S3 and Recovery

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix A: The way to output BIOS log

Physical UART

1.      Plug the cable between HOST and SUT.

2.      Open Tera Term/putty (Below use Tera Term to do the explanation).

3.      Choose Serial and Enhanced COM Port.

4.      Select File > Log > Enter the file path you would like to keep the log and also the log name > Press Save.

 

 

 

 

 

 

 

 

 

 

 

5.      It will start to log the serial log once you finish step 5.

6.      After above steps, please kindly send us the log for debugging.

 

 

 

 

 

 

 

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, send your comments to apic-docfeedback@cisco.com. We appreciate your feedback.

Legal Information

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

© 2026 Cisco Systems, Inc. All rights reserved.