C220M4 and C240M4 Option ROM issues

Introduction

 The intent of this document is to address the manufacturing issue related to the Peripheral Component Interconnect Express (PCIe) Option ROM settings as seen on some Cisco C220 M4 and C240 M4 servers. A number of the C-Series servers have shipped to customers with the incorrect PCIe Option ROM settings, preventing them from booting to many PCIe based devices (Including but not limited to RAID Controller / Serial-Attached SCSI (SAS)  Host Bus Adapter (HBA), Modular LAN on Motherboard (MLOM) or other networking cards or HBAs. The instructions below will assist you in resolveing customers impacted by this.

Potentially Impacted Systems (~11000 servers):

C220 M4: Systems shipped between Feb 14th and Mar 28th

C240 M4: Systems shipped between Feb 14th and Mar 28th

Customer Visibility/Impact:

Customers are unable to boot from any PCIe Slot based device, as the "All PCIe Slots OptionROM" flag has been disabled. Customers will need to toggle this BIOS token, and can do so using any of the workaround methods mentioned below.

Work-Around Options:

Work-Around #1 (Recommended):

Log in to the IMC, and navigate to the Summary / BIOS / Configure BIOS / Advanced Tab as shown below.

Set the “All PCIe Slots OptionROM” flag to “enabled”. Save changes and REBOOT the host.

Work-Around #2:

Connect crash cart / launch vKNM to the system. Power the system on and use the F2 menu to set the “All PCIe Slots OptionROM” flag to ENABLED.

1. Navigate to the Advanced LOM and PCIe Slots Configuration tab, and press ENTER
2. Scroll down and set the "All PCIe Slots OptionROM" to ENABLED
3. Save changes by pressing F10 and reboot the server.

Work-Around #3:

NOTE: This work-around assumes that the Cisco IMC is online and the IP is known. This script also uses default IMC credentials of admin/password, so if these have been changed users will need to modify the $pass field as required.

This solution is for customers who used DHCP to IP the IMC(s). Use XML API to log into one or more system and set the “all PCIe Slots OptionROM” flag correctly. A sample script has been provided below:

Step 1: Download IMC PowerTool module from the following page:

IMC PowerTool

Save & Run the following file as a <filename>.ps1 file:

Import-Module CiscoUcsPs

$multiimc = Set-UcsPowerToolConfiguration -SupportMultipleDefaultUcs $true

# The tool will prompt user to enter IP’s when run

$imclist = Read-Host "Enter Cisco IMC IP or list of IMC IPs separated by commas"

[array]$imclist = ($imclist.split(",")).trim()

$user = 'admin'

# The default password is on the next line (update as needed)

$pass = ConvertTo-SecureString –String "password" –AsPlainText -Force

$cred = New-Object –TypeName System.Management.Automation.PSCredential –ArgumentList $user, $pass

$out = Connect-Imc -Credential $cred $imclist

Set-ImcRackUnit -RackUnit 1 -AdminPower soft-shut-down -Force

Get-ImcBiosSettings | Get-ImcBiosVfPCIOptionROMs | Set-ImcBiosVfPCIOptionROMs -VpPCIOptionROMs "Enabled" -Force

sleep 25

Set-ImcRackUnit -RackUnit 1 -AdminPower up -Force

$out = Disconnect-Imc

Work-Around #4 (Live Linux USB with UCScfg):

This work-around involves creating a live bootable USB, that includes the tools necessary to correct the OptionROM settings. This requires a blank USB that is at least 4GB in size. Note: the steps below were performed on Windows 7.

1. Download the live .iso image here: Live CD
2. Download Universal USB Installer here: http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3/
3. Launch UUI, and select CentOS as the desired Linux Distribution
4. Point UUI to the .iso downloaded in step 1.
5. Select desired USB drive to install to, and then click “Create”. Wait for the process to complete, then USB is ready to use.

6. Plug in USB to affected server and select it as the desired boot device.

Note: The USB loaded with the above image will boot to CentOS, and then launch a series of UCScfg commands. This particular .iso corrects the optionROM settings AND sets the MLOM Port 0 as the first boot device. After the configuration changes complete, the server will power off. Changes will be reflected at next power on. UCScfg can be used to customize BIOS/IMC settings as desired, and the process shown above is just an example, and will correct the issues mentioned in this article only.