Field Notice: FN - 72145 - Nexus ACI 9000 Will Fail With SSD Read-Only Filesystem – Power Cycle Required - BIOS/Firmware Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.5	18-Sep-22	Updated the Background Section
1.4	28-Oct-21	Updated the Problem Symptom Section
1.3	13-Jun-21	Updated the Workaround/Solution Section
1.23	03-Jun-21	Updated the Workaround/Solution Section
1.22	01-Jun-21	Updated the Defect Information Section
1.21	24-May-21	Updated the Problem Description and Background Sections
1.2	14-May-21	Updated the How to Identify Affected Products Section and Added the Serial Number Validation Section
1.1	21-Apr-21	Updated the Workaround/Solution Section
1.0	29-Mar-21	Initial Release

Products Affected

Affected Product ID	Comments
N9K-C9396TX
N9K-C9396PX
N9K-C93128TX
N9K-C9332PQ
N9K-C9372PX
N9K-C9372TX
N9K-C93120TX
N9K-C9372PX-E
N9K-C9372TX-E
N9K-C93180YC-EX
N9K-C93108TC-EX
N9K-C93180LC-EX
N9K-SUP-B+
N9K-SUP-B
N9K-SUP-A+
N9K-SUP-A
N9K-C9396TX=
N9K-C9396PX=
N9K-C93128TX=
N9K-C9332PQ=
N9K-C9372PX=
N9K-C9372TX=
N9K-C93120TX=
N9K-C9372PX-E=
N9K-C9372TX-E=
N9K-C93180YC-EX=
N9K-C93108TC-EX=
N9K-C93180LC-EX=
N9K-SUP-B+=
N9K-SUP-B=
N9K-SUP-A+=
N9K-SUP-A=
N9K-C9336PQ
N9K-C9336PQ=

Defect Information

Defect ID	Headline
CSCvx19640	Nexus 9000 switch in read-only mode with M500IT SSD

Problem Description

Due to a flaw in the Solid State Drive (SSD) firmware, the SSD will no longer respond after approximately 3.2 years of cumulative operation. Restart the system in order to allow the drive to operate for another 1008 hours (approximately six weeks) before it will no longer respond again.

Background

After approximately 3.2 years (28,224 accumulated Power On Hours (POH)), a memory buffer overrun condition occurs which triggers the firmware event in the SSD.

This causes the drive to become unresponsive until the drive is power-cycled. No data loss will occur when the memory buffer overrun firmware event occurs. A power-cycle restores normal operation of the drive.

The drive continues to operate normally for approximately six weeks (1008 additional accumulated POH), at which time the drive will become unresponsive again.

Power-cycle the system in order to temporarily recover from this problem. However, this failure will reappear after 1008 hours of operation.

Problem Symptom

The SSD goes into read-only mode, which prevents normal switch functionality. When this happens the behavior of the switch is unpredictable, can cause operational impact, and an unexpected reload. Diagnostic test ssd-acc will fail repeatedly and fault F1222 will be raised on the switch until it is reloaded.

The failure conditions can be identified with these outputs:

• Presence of F1222 on the switch with the reason “Failed to write to file”:

  Fault Code           : F1222
Cause                : diag-failed
Affected Object      : topology/pod-1/node-350/sys/diag/rule-ssd-acc-trig-forever/subj-[topology/pod-1/node-350/sys/ch/supslot-1/sup]/rslt-2019-03-11T00:32:59.000+04:00
Description          : Diagnostics test failed. reason:Failed to write to file

• The diagnostic test ssd-acc fails repeatedly with the reason “Failed to write to file”.

  leaf101# show diagnostic result module 1 test 24 detail
  
  
  Current bootup diagnostic level: bypass
  Module 1: 48x10/25G  (Active)
  
          Test results: (. = Pass, F = Fail, I = Incomplete,
          U = Untested, A = Abort, E = Error disabled)
  ----------------------------------------------------------------------
  
          24) ssd-acc F
  
                  Error code ------------------> DIAG TEST FAIL
                  Total run count -------------> 216
                  Last test execution time ----> 2019-03-11 00:32:59
                  First test failure time -----> 2019-03-11 00:32:59
                  Last test failure time ------> 2019-03-11 00:32:59
                  Last test pass time ---------> 2019-03-11 00:27:59
                  Total failure count ---------> 1
                  Consecutive failure count ---> 1
                  Last failure reason ---------> Failed to write to file
                  Next Execution time ---------> 2019-03-11 00:37:59
  
          ----------------------------------------------------------------------

• Enter this command in order to verify /bootflash is in read-only mode (flag set to “ro”):

  leaf101# mount | grep bootflash
  /dev/sda4 on /bootflash type ext4 (ro,nodev,noexec,noatime,data=ordered)
  /dev/sda4 on /bootflash type ext4 (ro,nodev,noexec,noatime,data=ordered)

Workaround/Solution

Workaround

Restart the system in order to temporarily recover from this problem. However, this failure will reappear after 1008 hours of operation.

Solution

Upgrade the firmware of the SSD.

In order to prevent this issue and disruption to the network and operations, Cisco recommends to upgrade the firmware of the SSD proactively before the uptime reaches 28,224 hours. See the How to Identify Affected Products section and follow the firmware upgrade procedure accordingly.

If the system is already impacted, the SSD firmware upgrade will permanently resolve this defect.

Notes:

• It is strongly recommended to upgrade the firmware in a Maintenance Window.
• A Return Material Authorization (RMA) is not recommended as the upgrade process will resolve the issue.

Option 1. Upgrade the Application Centric Infrastructure NX-OS Version

First, verify if the /bootflash directory is already in the failed state (read-only state). If so, the switch must first be reloaded prior to the software upgrade.

The new firmware with the fix for this issue will be packaged in 13.2(10), 14.2(7), 15.1(4), 15.2(1), and later Application Centric Infrastructure (ACI) NX-OS versions.

When the switch is upgraded using the fixed (ACI) NX-OS version, the SSD firmware version will be automatically upgraded. See the Cisco APIC Installation, Upgrade, and Downgrade Guide for more information.

Option 2. Upgrade the SSD Firmware with SSDUpgrader

Use this procedure to download the SSDUpgrader app from the Cisco DC App Center and upgrade the SSD firmware.

1. Log in to the Cisco DC App Center as an end user.
2. Locate the SSDUpgrader app and click Download.
3. Review the license agreement and click Agree and download.
   The app is downloaded to your local machine.
4. Log in to the Cisco Application Policy Infrastructure Controller (APIC) user interface.
5. From the menu bar, choose Apps > Apps.
6. In the work pane, choose Add Application and complete these steps:
   1. Browse and choose the Cisco-SSDUpgrader-1.X.aci app image.
   2. Click Download.
7. Once the app has completed installation, click Enable.
8. On the SSDUpgrader app, click Open.
9. In the navigation pane, choose Settings > Connectivity and complete these steps:
   1. Enter the SSH admin username and password credentials.
   2. Enter the root token (must be provided by the Cisco Technical Assistance Center (TAC)).
   3. In the work pane, click Save.

10. In the nagivation pane, click Devices.
11. In the work pane, choose any node and click Check for Upgrade.
12. For any nodes that say Upgrade required, in the work pane choose the target node(s) and click Upgrade.

Notes:

• Administrator credentials are required for "Check for Upgrade". The root token is required for "Upgrade".
• Customers who purchase directly from Cisco but do not hold a Cisco service contract, and customers who make purchases through third-party vendors but are unsuccessful in obtaining a resolution for this field notice through their point-of-sale, should contact the Cisco Technical Assistance Center (TAC) in order to obtain an upgrade.
  Customers should have the product serial number available and be prepared to provide the URL of this field notice as evidence of entitlement to support and resolution of this field notice. Additionally, customers can only expect this exception for software which has a valid license, procured from Cisco directly or through a Cisco authorized reseller or partner. In most cases, this will be a maintenance upgrade to software that was previously purchased. Free updates related to this field notice do not entitle customers to a new software license, additional software feature sets, or major revision upgrades.

How To Identify Affected Products

Check the model and firmware revision of the SSD.

• Enter this command in the APIC CLI.

  apic1# moquery -c eqptFlash -f 'eqpt.Flash.model*"Micron_M500IT"'
  
  Total Objects shown: 2
  
  
  # eqpt.Flash
  
  acc          : read-write
  
  cap          : 61057
  
  childAction  :
  
  cimcVersion  :
  
  deltape      : 9
  
  descr        : flash
  
  dn           : topology/pod-1/node-101/sys/ch/supslot-1/sup/flash
  
  gbb          : 0
  
  id           : 1
  
  lba          : 0
  
  lifetime     : 1
  
  majorAlarm   : no
  
  mfgTm        : 2021-01-27T14:28:18.226+00:00
  
  minorAlarm   : no
  
  modTs        : 2021-02-22T14:27:10.100+00:00
  
  model        : Micron_M500IT_MTFDDAT064SBD
  
  monPolDn     : uni/fabric/monfab-default
  
  operSt       : ok
  
  peCycles     : 565
  
  readErr      : 0
  
  rev          : MC02.00
  
  rn           : flash
  
  ser          : MSA241001K2
  
  status       :
  
  tbw          : 2.881872
  
  type         : flash
  
  vendor       : Micron
  
  warning      : no
  
  wlc          : 0

• If the switch runs ACI Version 13.2(5) or later (excluding the 14.0(x) train), this command can be used to collect similar information and the Power_On_Hours.

  leaf101# cat /mnt/pss/smartctl_full_dump.log | tail -n 103 | egrep "Device Model|Firmware Version|ATTRIBUTE_NAME|Power_On_Hours"
  	  
  Device Model:     Micron_M500IT_MTFDDAT064SBD
  
  Firmware Version: MC02.00
  
  ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  
    9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       3347

From the output of the previous commands, both of these conditions are true on an affected switch:

• The Serial Number Validation tool (see the Serial Number Validation section) indicates that the serial number of the affected PIDs has the status “Affected” or the previous command shows that Device Model starts with “Micron_M500IT”.
• The Firmware Version is either MU01.00 or MC02.00.

If any of the conditions are not true, then the switch is not in the affected list and no action is required.

Power_On_Hours from the output can be used to calculate how much time is left before this issue occurs.

Serial Number Validation

This field notice provides the ability to determine if the serial number(s) of a device is impacted by this issue. In order to verify your serial number(s), enter it in the Serial Number Validation tool at https://snvui.cisco.com/snv/FN72145.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.