Troubleshoot DIMM Memory Issues in UCS

Introduction

This document describes how to troubleshoot memory modules and related issues in the Cisco Unified Computing System (UCS) solution.

Prerequisites

Requirements

Cisco recommends knowledge of Cisco Unified Computing System (UCS).

Components Used

This document is not restricted to specific software and hardware versions.

However, this document addresses:

• Cisco UCS B-Series Blade Servers
• UCS Manager
• UCS uses Dual In-line Memory Module (DIMM) as RAM modules.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Troubleshoot Methodology

This section covers several parts of UCS memory issues.

• Memory placement
• Troubleshoot DIMMs via UCSM and CLI
• Logs to check in technical support

Terms and Acronyms

Memory Placement

Memory placement is one of the most notable physical aspects of the UCS solution.

Typically, the server comes with memory pre-populated with a requested amount.

However, when in doubt, refer to the hardware installation guide.

For memory population rules, refer to B-series technical specifications for the specific platform.

B-series technical specifications link: 

Data Sheets

Memory Errors

• DIMM Error
  • Multibit = Uncorrectable
  • POST is mapped by BIOS; OS does not see DIMM.
  • Runtime usually causes OS reboot.
  • Singlebit = Correctable
  • OS continues to see the DIMM.
• Error Correcting Code (ECC) Error
• Parity Error
• Serial Presence Detect (SPD) Error
• Configuration Error
  • Not supported DIMMs
  • Not supported DIMM population
• Unpaired DIMMs
• Mismatch errors
• Identity unestablishable error
• Check and update the catalog.

Correctable versus Uncorrectable Errors

Whether a particular error is correctable or uncorrectable depends on the strength of the ECC code employed within the memory system.

Dedicated hardware is able to fix correctable errors when they occur with no impact on program execution.

The DIMMs with correctable errors are not disabled and are available for the OS to use. The Total Memory and Effective Memory  are the same.

These correctable errors are reported in the UCSM operability state as Degraded while overall operability is Operable with correctable errors.

Uncorrectable errors make it impossible for the application or operating system to continue execution.

The DIMMs with uncorrectable errors are disabled and OS does not see them. UCSM operState change to Inoperable in this case.

Troubleshoot DIMM’s via UCSM and CLI

Check Errors from GUI

To obtain statistics, navigate to Equipment > Chassis > Server > Inventory > Memory, then right-click Memory and select show navigator.

Check Errors from CLI

These commands are useful when troubleshooting errors from CLI.

scope server x/y -> show memory detail
scope server x/y -> show memory-array detail
scope server x/y -> scope memory-array x -> show stats history memory-array-env-stats detail

From memory array scope, you can also get access to DIMM.

scope server X/Y > scope memory-array Z > scope DIMM N

From there, then you can obtain per-DIMM statistics or reset the error counters.

UCS/chassis/server/memory-array/dimm # reset-errors                
UCS /chassis/server/memory-array/dimm* # commit-buffer               
UCS /chassis/server/memory-array/dimm # show stats memory-error-state

If you see a correctable error that matches this information, the problem can be corrected by resetting the BMC instead of resetting the blade server.

Use these Cisco UCS Manager CLI commands:

(Resetting the BMC does not impact the OS running on the blade.)

To reset memory-error counters on a Cisco UCS C-Series Rack Server operating in standalone mode, run the script on the CLI:

UCS-C# scope reset-ecc
UCS-C/reset-ecc # set enabled yes
UCS-C/reset-ecc *# commit

For colusa servers:

UCS# scope chassis
UCS /chassis # scope server x
UCS /chassis/server # reset-ecc

With UCS releases 2.27, and 3.1 and above, the thresholds for memory corrected errors have been removed.

Therefore, memory modules are no longer reported as Inoperable or Degraded solely due to corrected memory errors.

As per whitepaper Managing Correctable Memory Errors on Cisco UCS Servers

Industry demands for greater capacity, greater bandwidth, and lower operating voltages lead to increased memory error rates.

Traditionally, the industry has treated correctable errors in the same way as uncorrectable errors, requiring the module to be replaced immediately upon alert.

Given extensive research that correctable errors are not correlated with uncorrectable errors, and that correctable errors do not degrade system performance, the Cisco UCS team recommends against immediate replacement of modules with correctable errors. 

Customers who experience a Degraded memory alert for correctable errors are advised to reset the memory error and resume operation. This recommendation helps to avoid unnecessary server disruption. Future enhancements to error management distinguish among various types of correctable errors, and identify the appropriate actions, if any, needed.

At minimum, use version 2.1(3c) or 2.2(1b) which has enhancement with UCS memory error management

Log Files to Check in Tech Support

UCSM_X_TechSupport > sam_techsupportinfo provides information about DIMM and memory array.

Chassis/server tech support

CIMCX_TechSupport\tmp\CICMX_TechSupport.txt -> Generic tech support information about sever X.
CIMCX_TechSupport\obfl\obfl-log -> OBFL logs provide an ongoing logs about status and boot of server X.
CIMCX_TechSupport\var\log\sel -> SEL logs for server X.

Based on the platform/version, navigate to the files in tech support bundle.

var/nuova/BIOS > RankMarginTest.txt

var/nuova/BIOS > MemoryHob.txt

var/nuova/var/nuova/ BIOS > MrcOut_*.txt

These files provide information about memory as seen from BIOS level.

Information there can be cross-referenced again with DIMM states report tables.

 Example:

/var/nuova/BIOS/RankMarginTest.txt

• Useful for showing the test results from BIOS Training test MEMBIST.

• Look for errors.
• Look to see if any DIMMs are mapped out.
• Show DIMM specific information (Vendor/speed/PID).

DIMM |GB|R|MfgDate|Mod ID |DRAM ID   |Reg ID    |CtW Tck CLS Taa V|Freq|Part#
A1 18| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
A2 26| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
B1 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
B2 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
C1 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
C2 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
D1 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
D2 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
E1 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
E2 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
F1 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9 
F2 01| 8|2|2009W48|Samsung|Samsung 00|Inphi   03|5550 0C 003C 69 0|1333|M393B1K70BH1-CH9

The first column has two values:

DIMM locator (F2)

DIMM status (01)

Here is a brief description for each status:

0x00 // Not Installed (No DIMM)

0x01 // Installed (Working)

//// 0x02-0F (Reserved)

//// Failed                                 

0x10 // Failed Training

0x11 // Failed Clock Training

//// 0x12-17 (Reserved)

0x18 // Failed MemBIST 

//// 0x19-1F (Reserved)

//// Ignored

0x20 // Ignored (Disabled from debug console)

0x21 // Ignored (SPD Error reported by BMC)

0x22 // Ignored (Non-RDIMM)

0x23 // Ignored (Non-ECC)

0x24 // Ignored (Non-x4)

0x25 // Ignored (Other PDIMM in same LDIMM failed)

0x26 // Ignored (Other LDIMM in same channel failed)

0x27 // Ignored (Other channel in LockStep or Mirror failed)

0x28 // Ignored (Invalid PDIMM population)

0x29 // Ignored (PDIMM Organization Mismatch)

0x2A // Ignored (PDIMM Register Vendor Mismatch)

//// 0x2B-7F (Reserved)

var/nuova/BIOS > MemoryHob.txt

Shows effective and failed memory installed on the server.

 +++ BEGINNING OF FILE
Memory Speed     = 1067 MHz
Memory Mode      = 00
RAS Modes        = 03
MRC Flags        = 0000000A
Total Memory     = 98304 MB
Effective Memory = 90112 MB
Failed Memory    = 8192 MB
Ignored Memory   = 0 MB
Redundant Memory = 0 MB
|---------------------------------|
| Memory  | Channel | DIMM Status |
| Channel | Status  |   1    2    |
|---------------------------------|
|   A     |    01   |   01   01   |
|   B     |    01   |   01   01   |
|   C     |    01   |   01   01   |
|   D     |    01   |   01   01   |
|   E     |    01   |   01   01   |  
|   F     |    01   |   01   18   |
|---------------------------------|

18h - DIMM status is marked as failed when it fails in MemBist test.  Replace with a known good DIMM.

DIMM Status Description

00h Not Installed (No DIMM)

01h Installed (Working)

02h-0Fh Reserved

10h Failed (Training)

11h Failed (Clock training)

12h-17h Reserved

18h Failed (MemBIST)

19h-1Fh Reserved

20h Ignored (Disabled from debug console)

21h Ignored (SPD Error reported by BMC)

22h Ignored (Non-RDIMM)

23h Ignored (Non-ECC)

24h Ignored (Non-x4)

25h Ignored (Other PDIMM in same LDIMM failed)

26h Ignored (Other LDIMM in same channel failed)

27h Ignored (Other channel in LockStep or Mirror)

28h Ignored (Invalid memory population)

29h Ignored (Organization mismatch)

2Ah Ignored (Register  vendor  mismatch)

2Bh- 7Fh Reserved

80h Ignored ( Workaround  -Looping)

81h Ignored  (Stuck I2C bus)

82h – FFh Reserved

DIMM Blocklisting

In Cisco UCS Manager , the state of the Dual In-line Memory Module  (DIMM) is based on SEL event records.

When the BIOS encounters a noncorrectable memory error during memory test execution, the DIMM is marked as faulty.

A faulty DIMM is a considered a nonfunctional device.

If you enable DIMM blocklisting, Cisco UCS Manager monitors the memory test execution messages and blocklists any DIMMs that encounter memory errors in the DIMM SPD data. 

DIMM Blocklisting was introduced as an optional global policy in UCSM 2.2(2).

Server firmware must be 2.2(1)+ for B-series blades and 2.2(3)+ for C-series rack servers to properly implement this feature.

In UCSM 2.2(4), the DIMM, Blocklisting is enabled.

Open the tech support file …/var/log/DimmBL.log

Open the file /var/nuova/BIOS/MrcOut.txt if it is available

Find the DIMM Status table. Look for DIMM Status:

DIMM Blocklisted =  1E

Find the DIMM Status table. Look for DIMM Status:

        DIMM Status:

                    00 - Not Installed

                    01 - Installed

                    10 - Failed (Training failure) clear

                    1E - Failed (DIMM Blocklisted by BMC)

                    1F - Failed (SPD Error)

                    25 - Disabled (Other DIMM failed in same channel.)

Example:

        DIMM Status:

        |=======================|

        | Memory  | DIMM Status |

        | Channel |  1   2   3  |

        |=======================|

        |    A    |  25  1F  25 |

        |    B    |  01  01  01 |

        |    C    |  1F  25  25 |

        |    D    |  01  01  01 |

        |    E    |  01  01  01 |

        |    F    |  25  25  1E |

        |    G    |  01  01  01 |

        |    H    |  01  01  01 |

        |=======================|

          DIMM Status:

            01 - Installed

            1E - Failed (DIMM Blocklisted by BMC)

            1F - Failed (SPD Error)

            25 - Disabled (Other DIMM failed in same channel)

Methods to Clear DIMM Blocklisting Errors

UCSM GUI

UCSM CLI

UCS-B/chassis/server # reset-all-memory-errors

Related Information

• Cisco UCS Manager GUI Configuration Guide, Release 2.2

• Field Notice: FN - 63651 - UCS-B M3-Series Blade Servers Might Get Memory Errors Due to Voltage Regulator Setting - BIOS/Firmware Upgrade Recommended

Notable Bugs

Cisco bug ID CSCug93076 B200M3-DDR voltage regulator has excessive noise under light load

Cisco bug ID CSCup07488 IPMI DIMM fault sensor is setting Dimm Degraded with no error count.

Cisco bug ID CSCud22620 Improved accuracy at identifying Degraded DIMMs

Cisco bug ID CSCuw44524 C460M4, B260M4 or B460M4 IVB clear CMOS can cause memory UECC Error

Cisco bug ID CSCur19705  ECC/UECC Errors observed on B200M3

Cisco bug ID CSCvm88447Reset ECC steps documentation are missing for Standalone Colusa Servers