Field Notice: FN72495 - Nexus 9000 Series Switch Stuck in Boot Loop Due to policyelem hap Reset After Conversion to ACI - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Defect Information

Problem Description

Nexus 9000 Series switches shipped from the Service Depot via a Return Material Authorization (RMA) might fail when an upgrade to Application Centric Infrastructure (ACI) Release 4.2 and later is performed.

If you plan to upgrade an active network that is an earlier release of ACI to the impacted Release 4.2 or later, you must replace the switch prior to upgrade or this device will NOT work with the new software.

New switches received directly from manufacturing are not affected.

Nexus Operating System (NX-OS) software does not use this certificate. Switches that use NX-OS are not affected.

Background

There was a change in the manufacturing process which was not reflected at the same time in the Service Depots. As such, Nexus 9000 Series switches that were handled by Service Depots during this time via an RMA went through a different certificate provisioning process.

Problem Symptom

If you have an affected switch that runs any ACI release earlier than Release 4.2, no issue is exhibited.

If you upgrade an affected switch to ACI Release 4.2 or later, the "policyelem hap reset" issue is exhibited.

[  515.607768] @@@cctrli: wrote 16 to scratch RR

[  515.660368] nvram_klm wrote rr=16 rr_str=policyelem hap reset to nvram

[  515.736919] Collected 9 ext4 filesystems

[  515.787134] Freezing filesystems

[  515.870371] Collected 0 ubi filesystems

[  515.917365] Freezing filesystems

[  515.958101] Done freezing filesystems

[  516.004045] Putting SSD in stdby

[  516.977462] Done putting SSD in stdby 0

[  517.024456] Done offlining SSD

[  517.062066] Writing reg=0x84 val=0x80000000

If your switch continually exhibits the "hap reset" error only on Release 4.2 and later, complete these steps in order to verify if it is affected:

1. Reload the switch and break into the loader.
2. Enter these commands in order to boot the switch and quell the constant crashing:

   cmdline no_hap_reset
   boot

3. Verify that cert_extraction looks for 2099 (= TRUE):

   Leaf# egrep 2099 /mnt/pss/bootlogs/current/cert_extract.log
   
   ...
   
   2099 SUDI CERT present = TRUE
   
   ...

4. Verify these Policy Element (PE) logs:

   Leaf# cat /tmp/logs/dme_logs/svc_ifc_policyelem.log*

14754||2021-11-24T09:31:20.680537357+00:00||ifm||DBG4||co=ifm||Using regular cert's.||../dme/common/src/ifm/./IFMSSL.cc||287

14754||2021-11-24T09:31:20.681056311+00:00||ifm||DBG4||co=ifm||Failed to match Switch Regex ||../dme/common/src/ifm/./PeerVerificationUtils.cc||163

14754||2021-11-24T09:31:20.681094585+00:00||ifm||DBG4||co=ifm||Switch Certificate & SN mismatch||../dme/common/src/ifm/./IFMSSL.cc||309

14754||2021-11-24T09:31:20.681155894+00:00||ifm||ERROR||co=ifm||Failed to load the default SSL Engine||../dme/common/src/ifm/./IFMSSL.cc||208

14754||2021-11-24T09:31:20.681196907+00:00||log||CRIT||co=ifm||UNCONDITIONAL ASSERT (PANIC!) (!"failed to initialize openssl") failed @ ../dme/common/src/ifm/./Connection.cc:339

Workaround/Solution

If your switch is impacted, and you will use it to upgrade to ACI Release 4.2, call the Technical Assistance Center (TAC) for a replacement.

A replacement is ONLY required if the switch is used in ACI mode in the network and there are plans to upgrade the switch to Release 4.2.

How to Identify Affected Products

In order to validate if your product is impacted, use the Serial Number Validation Tool described in the Serial Number Validation section.

Serial Number Validation

Cisco provides a tool to verify whether a device is impacted by this issue. In order to check the device, enter the device's serial number in the Serial Number Validation Tool.

Note: For security reasons, you must click on the Serial Number Validation Tool link provided in this section to check the serial number for the device. Use of the Serial Number Validation Tool URL external to this field notice will fail.

Revision History

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.