Field Notice: FN74254 - Limited Number of Cisco Network Convergence System 540 and 560 Series Routers May Experience Memory Failure - Replace on Failure

A limited number of DIMM (Dual In-line Memory Modules) and DRAM (Dynamic Random Access Memory) components in Cisco Network Convergence System (NCS) 540 Series Routers and NCS 560 Series Routers that were shipped from Cisco are impacted by an issue in the manufacturing process of the memory supplier. This issue may lead to an increased rate of hardware failure following a reboot. 

A manufacturing deviation affected certain DIMM and DRAM components within a specific date range. Since identifying this issue, additional controls have been implemented in the manufacturing process to prevent similar variations in future components.

The impacted Cisco NCS 540 and NCS 560 Series Routers may reload or fail to boot. The following failure signatures may also be seen on the routers that reboot. 

Problem Symptoms for Cisco NCS 540 Series Routers 

Units that are affected by this issue will show a memory controller error that states that an uncorrectable hardware error is detected, which can cause a device to reload.

  1. Uncorrected errors can occur and processor context can be corrupted.

In the following log examples, the status "fe001bce001000c0" with the highlighted data f indicates that there are uncorrected errors and that processor context has been corrupted.

The following log can be seen from a pstore/dmesg-mtdoops-*  file:

<0>[11334.444107] mce: [Hardware Error]: CPU 2: Machine Check Exception: 5 Bank 9: fe001bce001000c0

 The following log can also be seen from a pstore/dmesg-mtdoops-*  file, indicating the error:

<0>[11334.803057] mce: [Hardware Error]: CPU 0: Machine Check Exception: 5 Bank 9: fe001bce001000c0<0>[11334.812586] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffffb800bfdc> {default_idle+0x1c/0xb0}<0>[11334.822314] mce: [Hardware Error]: TSC f873a16bcf8 ADDR 2b0685000 MISC 908402808003e8c<0>[11334.831268] mce: [Hardware Error]: PROCESSOR 0:50663 TIME 1737946143 SOCKET 0 APIC 0 microcode 7000013<0>[11334.841663] mce: [Hardware Error]: Run the above through 'mcelog --ascii'<0>[11334.849249] mce: [Hardware Error]: Machine check: Processor context corrupt<0>[11334.979803]  Kernel panic - not syncing: Fatal Machine check 

2. An uncorrectable error can cause the device to reload, as shown in the following example:

sysadmin-vm:0_RP0# show sdr default-sdr reboot-history 
      Reboots
                   Since
Location     Created   Reason
---------------------------------------------------------------------
0/RP0/VM1    20       
                       01/22/2025 13:45:40 CARD_SHUTDOWN
                       01/22/2025 12:17:00 CARD_SHUTDOWN
                       01/22/2025 10:48:19 CARD_SHUTDOWN

The Cisco NCS 540 Series Routers that were assembled with impacted DRAMs may exhibit the following errors in the console:

[   98.238034] Unhandled fault: synchronous external abort (0x92000210) at 0x0000aaaaf7d2df2c
[   98.246614] Bad mode in Error handler detected on CPU3, code 0xbf000002 -- SError
[   98.246626] Unhandled fault: synchronous external abort (0x96000210) at 0xffff8001d2c6fc80
[   98.246629] Internal error: : 96000210 [#2] SMP
[   98.246639] Modules linked in: m25p80 spi_nor cisco_fpga_setup(O) shmem_pg(O) mtdoops uio_pci_proxy fuse
[   98.246643] CPU: 0 PID: 505 Comm: systemd Tainted: G      D W  O    4.8.28-WR9.0.0.20_cgl #1
[   98.246644] Hardware name:  , BIOS 2018.03.1002-6.0.0 04/29/2020

Problem Symptoms for Cisco NCS560 Series Routers

1. Units that are affected by this issue will show a memory controller error, which will result in a log as shown in the examples below:

The following log can be seen from a kern.log file:

Aug 29 02:58:10 host kernel: [85572629.773552] mce: [Hardware Error]:Machine check events logged

The following log can be seen from an mcelog.log file: 

It’s a Hardware event. This is not a software error.MCE 0CPU 2 BANK 10MISC 90004000400008c ADDR 5f2afb000TIME 1724900290 Thu Aug 29 02:58:10 2024MCG status:MCi status:Corrected errorMCi_MISC register validMCi_ADDR register validMCA: MEMORY CONTROLLER MS_CHANNEL1_ERRTransaction: Memory scrubbing errorMemCtrl: Corrected patrol scrub errorSTATUS 8c000040000800c1 MCGSTATUS 0MCGCAP 1000c16 APICID 4 SOCKETID 0CPUID Vendor Intel Family 6 Model 86

2. Impacted modules exhibit removal/insertion loop without physical intervention. This uncorrectable error can lead to bootup failure, as shown in the following example:

0/RP1/ADMIN0:Nov 10 22:18:16.716 JST: shelf_mgr[28743]: %INFRA-SHELF_MGR-5-CARD_REMOVAL : Location: 0/RP0, Serial#: FOC2545NXXX0/RP1/ADMIN0:Nov 10 22:18:27.101 JST: shelf_mgr[28743]: %INFRA-SHELF_MGR-5-CARD_INSERTION : Location: 0/RP0, Serial #: FOC2545NXXX

Solution

Cisco recommends replacement of Cisco NCS 540 and NCS 560 Series Routers that exhibit the failure symptoms mentioned in the field notice and match the impacted serial number. Serial numbers can be identified using the serial number validation section of this field notice.

To determine if a unit is affected, see the Serial Number Validation section of this field notice.