Field Notice: FN72464 - Nexus 9300 Switches And APIC Servers Can Experience Memory Failures - Hardware Upgrade Available

A limited number of Dual In-line Memory Modules (DIMMs) shipped from Cisco are impacted by a known deviation in the memory supplier's manufacturing process. This deviation can result in a higher rate of failure.

In Revision 1.2 of this field notice, the Application Policy Infrastructure Controller (APIC) products were moved from Fix on Fail to Proactive DIMM replacement. 

It is required to replace the DIMMS and update from the older Cisco Integrated Management Controller (CIMC) BIOS (Version 4.1(3c) or earlier) in the same maintenance window.

DIMM manufacturers compose their DIMMs of multiple memory modules to reach the desired capacity. In this case, a manufacturing deviation in specific modules impacts 16GB DIMMs. This deviation was contained to a specific date range, and the DIMMs which use these chips were manufactured during the middle to end of 2020.

Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to help prevent future DIMMs from experiencing this process variation.

Most DIMMs with this manufacturing deviation will exhibit persistent correctable memory errors. If left untreated, the DIMMs can eventually encounter an uncorrectable memory event. If encountered during runtime, uncorrectable errors will cause an unexpected switch reset.

Various DIMM Reliability, Availability, and Serviceability (RAS) features or even operating system features can mask the extent of these correctable errors. It is recommended to check your DIMMs for exposure using the Serial Number Validation Tool described in the Serial Number Validation section of this field notice. Only specific DIMMs are impacted by this issue.

Customers should replace the hardware DIMM to avoid the potential for unexpected switch/server failure. For information about requesting a replacement, see the Upgrade Program Information section of this field notice after validating the Serial Number(s) as described in the How To Identify Affected Products section.

All Serial Numbers are the Switch or Server Serial Number, not the DIMM Serial Number.

DIMM Replacement

Cisco is offering field services free of charge for DIMM replacement through a qualified Cisco 3rd party Field Engineer.

After the replacement DIMMs have arrived onsite and you are ready to schedule the replacement(s), send an email to ciscodimmswap@centricsit.com to engage the field services team.

Impacted DIMMs can be identified based on the Product ID (PID) and their serial number. You will need to use the Serial Number Validation Tool described in the Serial Number Validation section of this field notice to identify impacted product.

Note: Cisco recommends notifying our engineers for onsite service to schedule repairs or replace the device. See the Additional Information section.

Switch Logs That Contain Memory Errors

When running NX-OS Standalone, a unit that experiences this issue can show these messages in the syslogs on the device:

• %DAEMON-3-SYSTEM_MSG: Location: SOCKET:0 CHANNEL:? DIMM:? [] - mcelog
• %DEVICE_TEST-3-MCE_24HR_FAIL: Module 1 has exceeded MCE 24 hour correctable threshold of 100 with #### correctable errors within 24 hours.
• %DAEMON-3-SYSTEM_MSG: corrected Socket memory error count exceeded threshold: #### in 24h - mcelog

or

• %DAEMON-3-SYSTEM_MSG: MESSAGE : corrected DIMM memory error count exceeded threshold: #### in 24h - mcelog
• %DAEMON-3-SYSTEM_MSG: MESSAGE_Location: /var/log/mcelog - mcelog

Note: Syslogs referencing the count of errors will be generated whenever 100 errors are corrected and the cumulative count of errors will be printed.

These errors indicate that correctable errors are being generated, which should not impact switch performance. If errors continue, an uncorrectable error can be experienced, and the device will undergo a kernel panic.

Switch Special Notes

This field notice is to replace memory on site, and it is important to note that if your impacted switch/server has failed or has memory errors and is degraded, use the standard RMA replacement.

Onsite replacements are usually done during maintenance windows and typically scheduled.

Some switches are originally assembled with 24GB of memory, this is one 16GB and one 8GB DIMM. Because of this, we send both DIMMs on those devices. When the switch cover is removed, both DIMMs are replaced even though the 8GB is known to be extremely reliable. This is a proactive action decision made by the Cisco team.

“Cisco highly recommends that you take advantage of the field service offering of a field engineer. Please note that if you decided to change the DIMMs on your own there are some switch models that have a more extensive process to gain access to the DIMM location."

It is highly recommended for a Cisco Field Engineer to complete the replacement for switches with high screw counts.

You can view a list of the switch module and the number of screws involved in this table.

Additionally, switches designed with 8GB low-density DIMM memory are impacted, but the failure rate is extremely low. These products are fixed upon failure.

APIC Server Special Notes

• The APIC server serial numbers were added to Serial Number Validation as of Revision 1.2 of this field notice.
• APIC-SERVER-M3 and spare has 6 DIMMs, ask for all 6 when you fill out your form for each server. You will replace ALL the memory.
• APIC-SERVER-L3 and spare has 12 DIMMs, ask for all 12 when you fill out your form for each server. You will replace ALL the memory.
• An onsite Field Engineer is available for free if needed. Customers can choose to do this themselves.
• There is a very important Field Notice CIMC BIOS. See the customer documentation for your current release on how to upgrade Application Centric Infrastructure (ACI) software or CIMC BIOS. The document is titled "Cisco APIC Installation and ACI Upgrade and Downgrade Guide".

  The CIMC BIOS issue is noted in UCS field notice FN72272. This BIOS issue will show higher EC errors counts that are potentially higher than the actual EC error count. You can see Uncorrectable errors because of older BIOS.