Field Notice: FN72464 - Cisco Nexus 9300 Switches and APIC Servers Can Experience Memory Failures - Hardware Upgrade Available

A limited number of dual inline memory modules (DIMMs) shipped from Cisco are impacted by a known deviation in the memory supplier's manufacturing process. This deviation can result in a higher-than-expected rate of failure.

DIMM manufacturers compose their DIMMs of multiple memory modules to reach the desired capacity. In this case, a manufacturing deviation in specific modules impacts 8GB DIMMs and 16GB DIMMs. This deviation was contained to a specific date range, and the DIMMs that were manufactured during the middle to end of 2020.

Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to help prevent future DIMMs from experiencing early failure due to this process variation.

The DIMMs with this manufacturing deviation will exhibit persistent correctable memory errors. If not replaced, the DIMMs can eventually encounter an uncorrectable memory event. If encountered during runtime, uncorrectable errors will cause an unexpected switch reset.

Various DIMM Reliability, Availability, and Serviceability (RAS) features or even operating system features can mask the extent of these correctable errors. It is recommended to check your DIMMs for exposure using the Serial Number Validation Tool described in the Serial Number Validation section of this field notice. Only specific DIMMs are impacted by this issue.

Solution

Customers should replace the hardware DIMM to avoid the potential for unexpected switch or server failure. For information about requesting a replacement, see the Upgrade Program Information section of this field notice after validating the serial number(s) as described in the How To Identify Affected Products section.

Important: For an updated list of impacted products, re-check serial numbers for all products that are listed in this field notice. All 8 GB and 16 GB are included in the Serial Number Validation tool.

All serial numbers are the switch or server serial number, not the DIMM serial number.

DIMM Replacement

Cisco is offering field services free of charge for DIMM replacement through a qualified Cisco third-party Field Engineer.

After the replacement DIMMs have arrived onsite and you are ready to schedule the replacement(s), send an email to ciscodimmswap@parkplacetech.com to engage the field services team.

Impacted DIMMs can be identified based on the Product ID (PID) and their serial number. To determine if a serial number is affected, see the Serial Number Validation section of this field notice.

Note: Cisco recommends notifying our engineers for onsite service to schedule repairs or replace the device. See the Additional Information section.

Switch Logs That Contain Memory Errors

When running NX-OS Standalone, a unit that experiences this issue can show these messages in the syslogs on the device:

• %DAEMON-3-SYSTEM_MSG: Location: SOCKET:0 CHANNEL:? DIMM:? [] - mcelog
• %DEVICE_TEST-3-MCE_24HR_FAIL: Module 1 has exceeded MCE 24 hour correctable threshold of 100 with #### correctable errors within 24 hours.
• %DAEMON-3-SYSTEM_MSG: corrected Socket memory error count exceeded threshold: #### in 24h - mcelog

or

• %DAEMON-3-SYSTEM_MSG: MESSAGE : corrected DIMM memory error count exceeded threshold: #### in 24h - mcelog
• %DAEMON-3-SYSTEM_MSG: MESSAGE_Location: /var/log/mcelog - mcelog

Note: Syslogs that reference the count of errors will be generated whenever 100 errors are corrected and the cumulative count of errors will be printed.

These errors indicate that correctable errors are being generated, which should not impact switch performance. If errors continue, an uncorrectable error can be experienced, and the device will undergo a kernel panic.

When running ACI software, a unit that experiences this issue can display errors in the /mnt/pss/bootlogs/current/dmesg archived log or in the latest logs in dmesg -T in the CLI. For example, the following logs confirm that DIMMs are bad and in which DIMM-0 is bad:

• [  167.751610] sbridge: HANDLING MCE MEMORY ERROR
• [  167.751614] CPU 0: Machine Check Exception: 0 Bank 7: 8c00004000010091
• [  168.415928] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_Channel#1_DIMM#0 (channel:1 slot:0 page:0x53232 offset:0xfc0 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0091 socket:0 channel_mask:2 rank:0)

Replacement Procedure for Switch or APIC Server with Outage Symptoms

To replace DIMM memory onsite, use the procedure that is described in the Upgrade Program Information and Workaround/Solution sections of this field notice when the following device conditions exist:

• A Nexus switch shows only signs of memory errors without impact to services or the system.
• An APIC server shows only signs of memory errors without current impact (system is stable).

However, if switch or server is affected by this issue and has failed or shows signs of service or system outage, use the standard Return Materials Authorization (RMA) replacement process through a Cisco Technical Assistance Center (TAC) support case when the following device conditions exist:

• A Nexus switch shows impact (device failure or signs of both memory errors and service or system crash, outage, or reset).
• An APIC server shows repeatable impact (server is down or has multiple occurrences of service or system crash, outage, or reset).

Switch Special Notes

• Onsite replacements are usually done during maintenance windows and typically scheduled.
• Some switches are originally assembled with 24GB of memory, this is one 16GB and one 8GB DIMM. Because of this, we send both DIMMs on those devices. When the switch cover is removed, both DIMMs are replaced even though the 8GB is known to be extremely reliable. This is a proactive action decision made by the Cisco team.
• Any switch that was repaired previously has had all of its memory replaced. There is no need for the memory to be replaced again.
• For Cisco N9K-C93360YC-FX2 or N9K-C93216TC-FX2, the DIMMs are not field replaceable. Cisco will replace the switch, not DIMMs, using this field notice to create the switch RMA. 
• Cisco highly recommends that you take advantage of the field service offering of a field engineer. Please note that if you decided to change the DIMMs on your own there are some switch models that have a more extensive process to gain access to the DIMM location. It is highly recommended for a Cisco Field Engineer to complete the replacement for switches with high screw counts.

Switch modules and the number of screws involved are listed in this table:

APIC Server Special Notes

• The APIC server serial numbers were added to Serial Number Validation as of Revision 1.2 of this field notice.
• APIC-SERVER-M3 and spare has 6 DIMMs, ask for all 6 when you fill out your form for each server. You will replace ALL the memory.
• APIC-SERVER-L3 and spare has 12 DIMMs, ask for all 12 when you fill out your form for each server. You will replace ALL the memory.
• An onsite Field Engineer is available for free if needed. Customers can choose to do this themselves.
• There is a very important Field Notice CIMC BIOS. See the customer documentation for your current release on how to upgrade Application Centric Infrastructure (ACI) software or CIMC BIOS. The document is titled "Cisco APIC Installation and ACI Upgrade and Downgrade Guide".

The CIMC BIOS issue is noted in UCS field notice FN72272. This BIOS issue will show higher EC errors counts that are potentially higher than the actual EC error count. You can see Uncorrectable errors because of older BIOS.