Field Notice: FN - 72162 - MDS 9396T and MDS 9148T Might Reload During or Not Pass Traffic After a Nondisruptive Software Update - Software Upgrade Recommended

Available Languages

Updated:June 10, 2021
Document ID:FN72162

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision Publish Date Comments
1.0
15-Apr-21
Initial Release
1.1
10-Jun-21
Updated the Products Affected and Problem Symptom Sections

Products Affected

Affected Product ID Comments
DS-C9148T-24IK9
DS-C9148T-24PITK9
DS-C9148T-48PITK9
DS-C9148T-24PITK9
DS-C9148T-24IK9
DS-C9148T-48PITK9
DS-C9148T-48PETK9
DS-C9148T-24EK9
DS-C9148T-24PETK9
DS-C9148T-24EK9
DS-C9148T-24PETK9
DS-C9148T-48PETK9
DS-C9148T-24PETK9
DS-C9148T-24EK9
DS-C9148T-48PETK9
DS-C9148T-48PITK9
DS-C9148T-24IK9
DS-C9148T-24PITK9
DS-C9148T-K9
DS-C9396T-48ETK9
DS-C9396T-48EK9
DS-C9396T-96ETK9
DS-C9396T-K9
DS-C9396T-96ETK9
DS-C9396T-48EK9
DS-C9396T-48ETK9
DS-C9396T-96ITK9
DS-C9396T-48IK9
DS-C9396T-48ITK9
DS-C9396T-48IK9
DS-C9396T-48ITK9
DS-C9396T-96ITK9
DS-C9396T-48ITK9
DS-C9396T-48IK9
DS-C9396T-96ITK9

Defect Information

Defect ID Headline
CSCvx52194 MDS 9396T, 9148T reloads during ISSU or internal switching errors or some ports fail loop back tests

Problem Description

A Multilayer Director Switch (MDS) updates the supervisor BIOS automatically as part of an NX-OS In Service Software Upgrade (ISSU) and, under limited circumstances, as part of an In Service Software Downgrade (ISSD).

The MDS 9148T (DS-C9148T) and MDS 9396T (DS-C9396T) Fibre Channel fabric switches do not operate correctly after a nondisruptive upgrade to certain BIOS versions. Any one of these symptoms will occur:

  • “f32xbar_usd” service crash and switch disruptively reloads during ISSU
  • “f32mac” service crash and switch disruptively reloads during ISSU
  • Arbiter is not detected and the switch disruptively reloads during ISSU
  • Burst of frame errors inside the switch during ISSU
  • Port loopback test failures after ISSU
  • No symptom seen and the switch appears to operate normally

Background

This issue is due to installation of BIOS code bundled in NX-OS 8.4(2b) and 8.5(1) software releases. The versions of BIOS in these two releases are not ISSU-compatible with earlier versions. The process of nondisruptively updating the supervisor CPU BIOS to these versions causes the CPU to enter an indeterminate state. The behavior of the CPU in this state is unpredictable and one of the symptoms described in the Problem Symptom section will occur.

This table compares the working and ISSU-incompatible BIOS versions.

Working BIOS Version ISSU-Incompatible BIOS Version
0.01 to 0.16 0.17 to 1.00

 

This table shows which supervisor CPU BIOS version is bundled in each NX-OS version.

NX-OS Version Supervisor CPU BIOS Version
8.3(1) 0.12
8.3(2) 0.12
8.4(1) 0.15
8.4(1a) 0.15
8.4(2) 0.16
8.4(2a) 0.16
8.4(2b) 0.18
8.5(1) 0.19

 

The issue affects the switch only once – the first time the BIOS is updated to an ISSU-incompatible version by an ISSU. The first ISSU with these BIOS versions to NX-OS versions will trigger this issue.

Current BIOS Version Issue Triggered by ISSU/D to
0.01 to 0.16 8.4(2b), 8.5(1)

 

A switch might not run the bundled BIOS version when it is shipped from the factory. Always check the current BIOS version as described in the How to Identify Affected Products section.

Problem Symptom

Any one of these symptoms might occur when an ISSU/ISSD is executed from the working version of the BIOS to the ISSU-incompatible version:

  1. The “f32xbar_usd” service crashes during ISSU. The last reset information in the show version command display shows a switch reset by “f32xbar_usd hap reset”: 
    Last reset at 364747 usecs after Sun Feb 21 01:51:59 2021
  Reason: Reset triggered due to HA policy of Reset
  System version: 8.4(2b)
  Service: f32xbar_usd hap reset
  2. The “f32mac” service crashes during an ISSU. The last reset information in the show version command display shows a switch reset by “f32mac hap reset”: 
    Last reset at 767868 usecs after Sun Feb 21 02:15:19 2021
  Reason: Reset triggered due to HA policy of Reset
  System version: 8.4(2b)
  Service: f32mac hap reset
  3. The arbiter is not detected during the ISSU. This diagnostic failure syslog is logged to show logging nvram: 
    %MODULE-2-MOD_DIAG_FAIL: hardware is not accessible in device DEV_TOR_ARB

    This is a fatal error which leads to a switch reset. The last reset information in the show version command output shows a reset caused by a “lc_port_mgr hap reset”:

    Last reset at 547800 usecs after Sun Feb 21 02:12:21 2021
  Reason: Reset triggered due to HA policy of Reset
  System version: 8.4(2b)
  Service: lc_port_mgr hap reset
  4. The ISSU is successful, but there is internal frame corruption. Either:
     
    1. A burst of internal frame corruption occurs during the ISSU itself. This can be seen after the ISSU as nonzero "egr EPR" counts with the show hardware internal errors command: 
      86  egr EPR2: F32_EPR2_CNT_BAD_SF_OUT         0000000000000142 1-16 -
489 egr EPR2: F32_EPR2_CNT_BAD_PKT_OUT        0000000000000142 1-16 -
492 egr EPR2: F32_EPR2_CNT_BAD_PKT_DROP       0000000000000163 1-16 -
494 egr EPR2: F32_EPR2_CNT_INVALID_KEY_OUT    0000000000000163 1-16 -
502 egr EPR2: F32_EPR2_CNT_OTHER_DROP         0000000000000163 1-16 -
514 egr EPR3: F32_EPR3_CNT_ERR_FI             0000000000000259 1-16 -
516 egr EPR3: F32_EPR3_CNT_ERR_CRC_ERR        0000000000000322 1-16 -
517 egr EPR3: F32_EPR3_CNT_ERR_CSUM_ERR       0000000000000169 1-16 -
518 egr EPR3: F32_EPR3_CNT_ERR_PSR            0000000000000164 1-16 –
    2. Random ports no longer pass traffic after the ISSU due to internal frame corruption. This causes diagnostic port loopback test failure syslog messages as shown by the show logging logfile command: 
      %SYSTEMHEALTH-2-OHMS_MOD_PORT_LB_TEST_FAILED: Module 1 Port 49 has failed loop back tests.
  5. The switch might hang during the ISSU/ISSD. This will result in the switch being unresponsive to out-of-band management and also a cessation of traffic switching. It will not recover even after a power-cycle.
  6. No symptom seen during or after an ISSU/ISSD. The switch appears to operate normally.

Note: Do not confuse this issue with these three defects. These defects have very similar symptoms, but different causes and fixes. The issue described in this field notice is only caused by an upgrade to a BIOS to the identified versions and only on the identified platforms. If these symptoms are seen in circumstances other than described in this field notice, then refer to these defects for the solution.

  • Cisco bug ID CSCvv49843 - MDS 9396T disruptively reboots during ISSU/D
  • Cisco bug ID CSCvv77164 - ISSU is disruptive in MDS 9132T, 9148T, 9396T after 'HA policy of reset'
  • Cisco bug ID CSCvr54401 - Evaluation of n7k/MDS platform for Intel 2019.2 IPU

Workaround/Solution

For Switches That Have Been Affected

Switches that have been affected by this issue must be cold booted with the reload CLI command or power-cycled. This will restore the CPU to a normal state. It needs to be completed only once after the BIOS upgrade.

If the switch has already been cold booted or power-cycled after the ISSU for any reason (which includes symptoms 1, 2, or 3 in the Problem Symptom section), then it is not necessary to repeat this step. The issue will not reoccur after this.

A nondisruptive reload, such as another ISSU, will not resolve this issue.

For Switches That Have Not Been Affected

New versions of NX-OS will address this issue and allow software updates for these platforms to be nondisruptive. The first such version is Cisco NX-OS Release 8.4(2c) which is due for release in the second calendar quarter of 2021.

Switches that have not been upgraded to an ISSU-incompatible BIOS version should not be nondisruptively upgraded to NX-OS Release 8.4(2b) or 8.5(1). An ISSU/ISSD to other versions of NX-OS as listed in the Background section will be nondisruptive since BIOS will not be updated to an ISSU-incompatible version.

If it is unavoidable to upgrade to an ISSU-incompatible BIOS version of Cisco NX-OS (such as for specific code fixes) it must be done disruptively, as follows:

  1. Download the kickstart and system image to the switch bootflash.
  2. Enter configuration mode with the configure command.
  3. Clear the system boot variable with the no boot system command.
  4. Set the kickstart boot variable (change the filename as appropriate) with the boot kickstart m9300-s2ek9-kickstart-mz.8.4.2b.bin command.
  5. Set the system boot variable (change the filename as appropriate) with the boot system m9300-s2ek9-mz.8.4.2b.bin command.
  6. Save the new boot variables with the copy running-config startup-config command.
  7. Disruptively reload the switch with the reload command.

Note: BIOS is updated automatically only during an ISSU. This procedure will not update the BIOS version as it does not use an ISSU. Due to this, the switch can still be affected by this issue at the next ISSU if the target NX-OS version does not have the fix.

The switch will function normally after it has been cold booted, even with an ISSU-incompatible BIOS version.

How To Identify Affected Products

In order to determine if a switch will be affected by this issue during an ISSU, verify the currently installed BIOS version with the show version CLI command. The highlighted lines in this example show that BIOS Version 0.19 is currently installed:

Software
  BIOS:      version 0.19.0

In order to determine if a switch has been affected by an ISSU, verify the BIOS version history with the show logging onboard boot-uptime CLI command. An entry is made with both the primary and secondary BIOS versions each time the switch boots. The highlighted lines in this example show that the switch booted at 16:23:50 with BIOS Version 0.15 and was upgraded by an ISSU to Version 0.19 at 14:16:46:

Wed Mar 17 14:16:46 2021:  Boot Record
-------------------------------------------------------
Boot Time..........:  Wed Mar 17 14:16:46 2021
Slot Number........:  1
Serial Number......:  JAE2208....
Bios Version.......:  v0.19.0(09/29/2020)
Alt Bios Version...:  v0.19.0(09/29/2020)
Firmware Version...:  8.5(1) [build 8.5(1)]

Wed Mar 10 16:23:50 2021:  Boot Record
-------------------------------------------------------
Boot Time..........:  Wed Mar 10 16:23:50 2021
Slot Number........:  1
Serial Number......:  JAE2208....
Bios Version.......:  v0.15.0(09/29/2020)
Alt Bios Version...:  v0.15.0(09/29/2020)
Firmware Version...:  8.4(1a)) [build 8.4(1a)]

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco