THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Affected Product Name | Description | Comments |
---|---|---|
N540-24Q8L2DD-SYS | NCS540-2x400G QSFP-DD+8x50G+24x25G | |
N540-24Q8L2DD-SYS= | NCS540-2x QSFP-DD 400G+8x50+24x25G Spare | |
N540-24Z8Q2C-M | NCS 540 Series 24x1/10GE, 8x10/25GE, 2x100GE Fixed Chassis | |
N540-24Z8Q2C-M= | Cisco NCS 540 Series Fixed Router 24x10G, 8x25G, 2x100G | |
N540-24Z8Q2C-SYS | N540-24Z8Q2C Base HW Flex. Consumption Need Smart Licensing | |
N540-ACC-M | NCS 540 Series 24x1/10GE, 8x10/25GE, 2x100GE Fixed Chassis | |
N540-ACC-SYS | NCS540 24x1/10G SFP+, 8x1/10/25G SFP+/SFP28, 2x100G QSFP28 | |
N540-ACC-SYS-V | NCS540 24x1/10G SFP+, 8x1/10/25G SFP+/SFP28, 2x100G QSFP28 | |
N540-ACC-SYS= | NCS540 24x10G SFP+, 8x1/10/25G SFP+/SFP28, 2x100G QSFP28 | |
N540X-6Z18G-SYS-A | NCS540 18x1G SFP + 6x1/10G SFP+ Dual-AC iTEMP Conformal Coat | |
N540X-6Z18G-SYS-A= | NCS540 18x1G SFP + 6x1/10G SFP+ Dual AC iTEMP Conf-coat Spre | |
N540X-6Z18G-SYS-D | NCS540 18x1G SFP + 6x1/10G SFP+ Dual DC iTEMP Conformal Coat | |
N540X-6Z18G-SYS-D= | NCS540 18x1G SFP + 6x1/10G SFP+ Dual DC iTEMP Conf-Coat Spar | |
N540X-8Z16G-SYS-A | N540 12/20(CSFP)x1G + 4x1GCu +8x1/10G DualAC iTEMP Conf-Coat | |
N540X-8Z16G-SYS-A= | N540 12/20(CSFP)x1G + 4x1GCu + 8x1/10G DualAC iTEMP Conf Spr | |
N540X-8Z16G-SYS-D | N540 12/20(CSFP)x1G + 4x1GCu +8x1/10G DualDC iTEMP Conf-Coat | |
N540X-8Z16G-SYS-D= | N540 12/20(CSFP)x1G + 4x1GCu + 8x1/10G DualDC iTEMP Conf Spr | |
N560-4-RSP4 | NCS 560 Series Router 4RU Route Switch Processor 4, 800G | |
N560-4-RSP4= | NCS 560 Series Router 4RU Route Switch Processor 4, 800G | |
N560-4-RSP4E | NCS 560 Series Router 4RU Route Switch Processor 4E, 800G | |
N560-4-RSP4E= | NCS 560 Series Router 4RU Route Switch Processor 4E, 800G | |
N560-RSP4 | NCS 560 Route Switch Processor 4 - L scale, 800G | |
N560-RSP4-E | NCS 560 Route Switch Processor 4 Enhanced- XL scale, 800G | |
N560-RSP4-E= | NCS 560 Route Switch Processor 4 Enhanced- XL scale, 800G | |
N560-RSP4= | NCS 560 Route Switch Processor 4 - L scale, 800G |
Defect ID | Headline |
CSCwc01361 | NCS540 - Some DIMMs failing at higher than expected rate in NCS540 |
CSCwo25212 | RSP's may experience "Failed to allocate resources for sdr", Uncorrectable error & or kernel crash |
CSCwo41741 | NCS540 - Some DDR Devices failing at higher than expected rate |
A limited number of DIMM (Dual In-line Memory Modules) and DRAM (Dynamic Random Access Memory) components in Cisco Network Convergence System (NCS) 540 Series Routers and NCS 560 Series Routers that were shipped from Cisco are impacted by an issue in the manufacturing process of the memory supplier. This issue may lead to an increased rate of hardware failure following a reboot.
A manufacturing deviation affected certain DIMM and DRAM components within a specific date range. Since identifying this issue, additional controls have been implemented in the manufacturing process to prevent similar variations in future components.
The impacted Cisco NCS 540 and NCS 560 Series Routers may reload or fail to boot. The following failure signatures may also be seen on the routers that reboot.
Problem Symptoms for Cisco NCS 540 Series Routers
Units that are affected by this issue will show a memory controller error that states that an uncorrectable hardware error is detected, which can cause a device to reload.
1. Uncorrected errors can occur and processor context can be corrupted.
In the following log examples, the status "fe001bce001000c0" with the highlighted data f indicates that there are uncorrected errors and that processor context has been corrupted.
The following log can be seen from a pstore/dmesg-mtdoops-* file:
<0>[11334.444107] mce: [Hardware Error]: CPU 2: Machine Check Exception: 5 Bank 9: fe001bce001000c0
The following log can also be seen from a pstore/dmesg-mtdoops-* file, indicating the error:
<0>[11334.803057] mce: [Hardware Error]: CPU 0: Machine Check Exception: 5 Bank 9: fe001bce001000c0<0>[11334.812586] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffffb800bfdc> {default_idle+0x1c/0xb0}<0>[11334.822314] mce: [Hardware Error]: TSC f873a16bcf8 ADDR 2b0685000 MISC 908402808003e8c<0>[11334.831268] mce: [Hardware Error]: PROCESSOR 0:50663 TIME 1737946143 SOCKET 0 APIC 0 microcode 7000013<0>[11334.841663] mce: [Hardware Error]: Run the above through 'mcelog --ascii'<0>[11334.849249] mce: [Hardware Error]: Machine check: Processor context corrupt<0>[11334.979803] Kernel panic - not syncing: Fatal Machine check
2. An uncorrectable error can cause the device to reload, as shown in the following example:
sysadmin-vm:0_RP0# show sdr default-sdr reboot-history
Reboots
Since
Location Created Reason
---------------------------------------------------------------------
0/RP0/VM1 20
01/22/2025 13:45:40 CARD_SHUTDOWN
01/22/2025 12:17:00 CARD_SHUTDOWN
01/22/2025 10:48:19 CARD_SHUTDOWN
The Cisco NCS 540 Series Routers that were assembled with impacted DRAMs may exhibit the following errors in the console:
[ 98.238034] Unhandled fault: synchronous external abort (0x92000210) at 0x0000aaaaf7d2df2c
[ 98.246614] Bad mode in Error handler detected on CPU3, code 0xbf000002 -- SError
[ 98.246626] Unhandled fault: synchronous external abort (0x96000210) at 0xffff8001d2c6fc80
[ 98.246629] Internal error: : 96000210 [#2] SMP
[ 98.246639] Modules linked in: m25p80 spi_nor cisco_fpga_setup(O) shmem_pg(O) mtdoops uio_pci_proxy fuse
[ 98.246643] CPU: 0 PID: 505 Comm: systemd Tainted: G D W O 4.8.28-WR9.0.0.20_cgl #1
[ 98.246644] Hardware name: , BIOS 2018.03.1002-6.0.0 04/29/2020
Problem Symptoms for Cisco NCS560 Series Routers
1. Units that are affected by this issue will show a memory controller error, which will result in a log as shown in the examples below:
The following log can be seen from a kern.log file:
Aug 29 02:58:10 host kernel: [85572629.773552] mce: [Hardware Error]:Machine check events logged
The following log can be seen from an mcelog.log file:
It’s a Hardware event. This is not a software error.MCE 0CPU 2 BANK 10MISC 90004000400008c ADDR 5f2afb000TIME 1724900290 Thu Aug 29 02:58:10 2024MCG status:MCi status:Corrected errorMCi_MISC register validMCi_ADDR register validMCA: MEMORY CONTROLLER MS_CHANNEL1_ERRTransaction: Memory scrubbing errorMemCtrl: Corrected patrol scrub errorSTATUS 8c000040000800c1 MCGSTATUS 0MCGCAP 1000c16 APICID 4 SOCKETID 0CPUID Vendor Intel Family 6 Model 86
2. Impacted modules exhibit removal/insertion loop without physical intervention. This uncorrectable error can lead to bootup failure, as shown in the following example:
0/RP1/ADMIN0:Nov 10 22:18:16.716 JST: shelf_mgr[28743]: %INFRA-SHELF_MGR-5-CARD_REMOVAL : Location: 0/RP0, Serial#: FOC2545NXXX0/RP1/ADMIN0:Nov 10 22:18:27.101 JST: shelf_mgr[28743]: %INFRA-SHELF_MGR-5-CARD_INSERTION : Location: 0/RP0, Serial #: FOC2545NXXX
Solution
Cisco recommends replacement of Cisco NCS 540 and NCS 560 Series Routers that exhibit the failure symptoms mentioned in the field notice and match the impacted serial number. Serial numbers can be identified using the serial number validation section of this field notice.
To determine if a unit is affected, see the Serial Number Validation section of this field notice.
Cisco provides the Serial Number Validation Tool to verify whether a device is impacted by this issue. To check the device, enter the serial number in the Serial Number Validation Tool.
Important: For security reasons, you must click the Serial Number Validation Tool link that is provided in this section. Do not copy and paste the link into a browser. Use of the Serial Number Validation Tool URL external to this field notice will fail.
Version | Description | Section | Date |
1.0 | Initial Release | — | 2025-MAR-27 |
For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:
To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications.
Unleash the Power of TAC's Virtual Assistance