Field Notice: FN - 72433 - Nexus 3000 and 9000 Switch Traffic Disruptions and Control Plane Instability - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.1	01-Sep-22	Updated the Problem Description, Problem Symptom, and Workaround/Solution Sections
1.0	20-Jul-22	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	NX-OS System Software	9	9.2(1), 9.2(2), 9.2(3), 9.3(1), 9.3(2), 9.3(3), 9.3(4), 9.3(5), 9.3(6), 9.3(7), 9.3(7a), 9.3(8)
NON-IOS	NX-OS System Software	10	10.1(1), 10.1(2), 10.2(1), 10.2(2)

Defect Information

Defect ID	Headline
CSCvz65993	tahoe0 brought down resulting in inband connectivity failure

Problem Description

Nexus 9000 switches might experience traffic disruptions and control plane instability when running release 9.2(1) through 9.3(8) or 10.1(1) through 10.2(2).

In affected releases later than Release 9.3(1), the device might undergo a kernel panic due to a L2ACLRedirect or Rewrite-engine GOLD test failure.

Background

A known fix for Cisco bug ID CSCvz65993 is available in the software releases identified in this software field notice. Cisco recommends that you upgrade to the fixed version or later.

Problem Symptom

The impact seen for Nexus 3000 and Nexus 9000 devices that experience this issue is detailed in this section.

Cloudscale Nexus 9200/9300 - Suffixes EX, FX, FX2, FX3, and GX

These syslogs will be seen on the device when the issue occurs:

%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_INTERMEDIATE_FAIL: Warning! Module:1 Test:L2ACLRedirect Loopback failed 5 consecutive times. Upon reaching consecutive failure threshold corrective action would be taken.

%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <L2ACLRedirect> has been disabled as a part of default EEM action

%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_FAIL: Module:1 Test:L2ACLRedirect Loopback failed 10 consecutive times. Faulty module:  Error:Loopback test failed. Packets lost on the SUP in the receive direction

In order to recover the device from this condition, a manual reload of the device must be performed.

When this issue occurs in affected releases after Release 9.3(1), control plane traffic will be impacted until the unit undergoes a diagnostic-triggered kernel panic. When this occurs, the previous syslogs and the next syslog will be seen on the device:

%KERN-1-SYSTEM_MSG: [19791125.834518] BUG: unable to handle kernel NULL pointer dereference at (null) - kernel

These reset reasons will be seen on the device:

----- reset reason for Supervisor-module 1 (from Supervisor in slot 1) ---

At 503414 usecs after Tue Jan 1 00:00:00 2022

      Reason: Kernel Panic

      Service:

      Version: 9.3(6)

  At 126444 usecs after Tue Jun 1 00:00:00 2022

      Reason: Reset due to critical failure detected by Online diagnostic

      Service: GOLD Packet Test Failure

      Version:

In affected releases earlier than Release 9.3(1), devices will encounter indefinite control plane instability and will not kernel panic. In order to avoid this, configure this Embedded Event Manager (EEM) to do a forced reload. The EEM applet is only a workaround to avoid the indefinite control plane instability and will not resolve this issue permanently. For a permanent solution, an upgrade to the fixed version shown in the Workaround/Solution section is the only option.

event manager applet gold_l2acl override __L2ACLRedirect

  action 1 syslog priority emergencies msg L2ACL_test_failed_reloading

  action 2 reload force

Nexus 9500 - All Generations of LCs

When this issue occurs on Nexus 9500s, control plane instability will be seen on the device. The internal management channels (EOBC/EPC) will be impacted which might result in modules within the chassis being shut off.

In affected releases, kernel panic might occur on the active supervisor which results in supervisor switchover. This syslog will be seen on the device when the issue occurs:

%PLATFORM-2-MOD_ALL_PWRDN_NOSC: All fabric, line-card modules powered down due to non-availabilitySS of system controller modules

If the device does not automatically recover from this condition, a manual restart of the device should be performed.

Nexus 3000 R-Series

When this issue occurs on Nexus 3000 R-Series, these syslogs will be seen on the device:

%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_INTERMEDIATE_FAIL: Warning! Module:1 Test:L2ACLRedirect Loopback failed 5 consecutive times. Upon reaching consecutive failure threshold corrective action would be taken.

%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <L2ACLRedirect> has been disabled as a part of default EEM action

%DIAG_PORT_LB-2-L2ACLREDIRECT_LOOPBACK_TEST_FAIL: Module:1 Test:L2ACLRedirect Loopback failed 10 consecutive times. Faulty module:  Error:Loopback test failed. Packets lost on the SUP in the receive direction

Indefinite control plane impact might be seen on devices that run the impacted versions. This EEM can be configured to force a reload. The EEM applet is only a workaround to avoid the indefinite control plane instability and will not resolve this issue permanently. For a permanent solution, an upgrade to the fixed version shown in the Workaround/Solution section is the only option.

event manager applet gold_l2acl override __L2ACLRedirect

  action 1 syslog priority emergencies msg L2ACL_test_failed_reloading

  action 2 reload force

Nexus 3100 and First-Generation Merchant-Silicon 9300

When this issue occurs on Nexus 3100 Series and first-generation merchant-silicon Nexus 9300 on affected releases, these syslogs will be seen on the device:

%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <RewriteEngineLoopback> has been disabled as a part of default EEM action

%DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:1 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:  Error:Loopback test failed. Packets lost on the SUP in the receive direction

When this issue occurs in affected releases after Release 9.3(1), control plane traffic will be impacted until the unit undergoes a diagnostic triggered kernel panic. When this occurs, the previous syslogs plus the next syslog will be seen on the device:

%KERN-1-SYSTEM_MSG: [24014796.045164] BUG: unable to handle kernel NULL pointer dereference at (null) - kernel

In affected releases before Release 9.3(1), devices will encounter indefinite control plane instability and will not kernel panic. In order to avoid this, configure this EEM to do a forced reload. The EEM applet is only a workaround to avoid the indefinite control plane instability and will not resolve this issue permanently. For a permanent solution, an upgrade to the fixed version shown in the Workaround/Solution section is the only option.

Note: This EEM will work only for Nexus 3100 and first-generation Nexus 9000.

event manager applet goldrewriteacl override __RewriteEngineLoopback

  action 1 syslog priority emergencies msg Rewrite_engine_failed_reloading

  action 2 reload module 1 force

Nexus 3000

When this issue occurs on Nexus 3000 Series on affected releases, these syslogs will be seen on the device:

%DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test <RewriteEngineLoopback> has been disabled as a part of default EEM action

%DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:1 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:  Error:Loopback test failed. Packets lost on the SUP in the receive direction

When this issue occurs in affected releases, control plane traffic will be impacted until the unit is recovered by a manual reload.

Workaround/Solution

Customers should upgrade impacted Product Identifiers (PIDs) to Release 9.3(9) or Release 10.2(3) in order to address this condition.

PID	Release
N3K-C3048	9.3(9)
N3K-C3064	9.3(9)
N3K-C3132C-Z	9.3(9)
N3K-C3132Q	9.3(9)
N3K-C3164Q	9.3(9)
N3K-C3172PQ	9.3(9)
N3K-C3172TQ	9.3(9)
N3K-C3172PQ-XL	9.3(9)
N3K-C3172TQ-XL	9.3(9)
N3K-C3232C	9.3(9)
N3K-C3264C-E	9.3(9)
N3K-C3264	9.3(9)
N3K-C3524	9.3(9)
N3K-C3524-X	9.3(9)
N3K-C3524-XL	9.3(9) or 10.2(3)
N3K-C3548	9.3(9)
N3K-C3548-XL	9.3(9) of 10.2(3)
N3K-C31108PC-V	9.3(9)
N3K-C31108TC-V	9.3(9)
N3K-C31128PQ	9.3(9)
N3K-C36180YC-R	9.3(9) or 10.2(3)
N3K-C3636-R	9.3(9) or 10.2(3)
N9K-C9372PX	9.3(9)
N9K-C9372PX-E	9.3(9)
N9K-C9372TX	9.3(9)
N9K-C9372TX-E	9.3(9)
N9K-C9332PQ	9.3(9)
N9K-C93120TX	9.3(9)
N9K-C9396PX	9.3(9)
N9K-C9396TX	9.3(9)
N9K-C93120TX	9.3(9)
N9K-C93128TX	9.3(9)
N9K-C92160YC-X	9.3(9)
N9K-C9236C	9.3(9)
N9K-C92300YC	9.3(9)
N9K-C92304QC	9.3(9)
N9K-C9272Q	9.3(9)
N9K-C92348GC-X	9.3(9) or 10.2(3)
N9K-C93180YC-EX-24	9.3(9) or 10.2(3)
N9K-C93180YC-EX	9.3(9) or 10.2(3)
N9K-C93180LC-EX	9.3(9) or 10.2(3)
N9K-C93108TC-EX	9.3(9) or 10.2(3)
N9K-C93108TC-EX-24	9.3(9) or 10.2(3)
N9K-C93108TC-FX	9.3(9) or 10.2(3)
N9K-C93108TC-FX-24	9.3(9) or 10.2(3)
N9K-C93180YC-FX	9.3(9) or 10.2(3)
N9K-C93180YC-FX-24	9.3(9) or 10.2(3)
N9K-C9348GC-FXP	9.3(9) or 10.2(3)
N9K-C9364C	9.3(9) or 10.2(3)
N9K-C9332C	9.3(9) or 10.2(3)
N9K-C9336C-FX2	9.3(9) or 10.2(3)
N9K-C93240YC-FX2	9.3(9) or 10.2(3)
N9K-C93360YC-FX2	9.3(9) or 10.2(3)
N9K-C93216TC-FX2	9.3(9) or 10.2(3)
N9K-C9336C-FX2-E	10.2(3)
N9K-C93180YC-FX3	9.3(9) or 10.2(3)
N9K-C93180YC-FX3S	9.3(9) or 10.2(3)
N9K-C93108TC-FX3P	9.3(9) or 10.2(3)
N9K-C93600CD-GX	9.3(9) or 10.2(3)
N9K-C9364C-GX	9.3(9) or 10.2(3)
N9K-C9316D-GX	9.3(9) or 10.2(3)
N9K-C9504-FM	9.3(9)
N9K-C9508-FM	9.3(9)
N9K-C9516-FM	9.3(9)
N9K-C9504-FM-E	9.3(9)
N9K-C9508-FM-E	9.3(9)
N9K-C9516-FM-E	9.3(9)
N9K-C9508-FM-E2	9.3(9) or 10.2(3)
N9K-C9516-FM-E2	9.3(9) or 10.2(3)
N9K-C9504-FM-R	9.3(9) or 10.2(3)
N9K-C9508-R	9.3(9) or 10.2(3)
N9K-C9504-FM-G	10.2(3)
N9K-C9508-FM-G	10.2(3)

Additional Information

For information on how to upgrade the software, see:

• Recommended Cisco NX-OS Releases for Cisco Nexus 9000 Series Switches
• Cisco Nexus 9000 and 3000 ISSU Support Matrix
• 9.3(x) Software Upgrade Guide
• 10.2(x) Software Upgrade Guide

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.