Field Notice: FN74007 - Cisco UCS 6400 and 6500 Series Fabric Interconnects unexpectedly restart after remote user login - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Defect Information

Problem Description

Cisco UCS 6400 Series and 6500 Series Fabric Interconnects that are running Cisco UCS Infrastructure and UCS Manager (UCSM Infrastructure) Software Bundle Release 4.2(2d) or Release 4.2(3b) may unexpectedly restart. When reviewing the reason for the reset, you will see Reset triggered due to HA policy of Reset, as seen in the following example:

1) At [Time] usecs after [Date and Time]
    Reason: Reset triggered due to HA policy of Reset
    Service: sysmgr stateful recovery

The fabric interconnect will recover after the unexpected reboot. Upgrading to Cisco UCSM Infrastructure Software Bundle Release 4.2(2e) or later or Release 4.2(3d) or later will resolve the issue.

Background

When using a Cisco UCS 6400 Series and 6500 Series Fabric Interconnect that is running Cisco UCSM Infrastructure Software Bundle Release 4.2(2d) or Release 4.2(3b) and logging in remotely as an administrator, a temporary user is created within the Cisco UCSM framework. A function call to synchronize the admin password to the remote fabric extenders is made. In rare situations, when this password synchronization function completes, the function can create a deadlock condition. This situation requires a system reset to clear.

Problem Symptom

During normal operation, a Cisco UCS 6400 Series and 6500 Series Fabric Interconnect may experience an unexpected reboot. 

The output of the NX-OS command show system reset-reason may show the service sysmgr stateful recovery, as in the following example:

1) At [Time] usecs after [Date and Time]
    Reason: Reset triggered due to HA policy of Reset
    Service: sysmgr stateful recovery <--

The output of the NXOS command show logging nvram may show the following information, which indicates that the fex service may have crashed:

2023 Feb  7 06:56:03  %$ VDC-1 %$ %SYSMGR-2-SERVICE_CRASHED: Service "fex" (PID 21783) hasn't caught signal 6 (core will be saved).
2023 Feb  7 06:56:03  %$ VDC-1 %$ %SYSMGR-2-HAP_FAILURE_SUP_RESET: Service "fex" in vdc 1 has had a hap failure

Workaround/Solution

Logging in to a Cisco UCS 6400 Series and 6500 Series Fabric Interconnect as a remote user triggers user creation and password synchronization with the fabric extender, potentially causing the fabric interconnect to reboot. Avoid using remote users (such as LDAP accounts) to access Cisco UCSM through the GUI, the XML API, or the CLI. Use a local account instead.

Logging in to Cisco UCSM through Cisco Intersight cross-launch does not trigger this issue and is a safe, viable alternative.

Customers should upgrade to Cisco UCSM Infrastructure Software Bundle Release 4.2(2e) or later or Release 4.2(3d) or later to resolve this problem.

How to Identify Affected Products

Follow these steps to identify the Cisco UCSM Infrastructure Bundle and the fabric interconnects that you are using.

For Cisco UCSM
To check the software bundle, follow these steps:

1. Log in to the Cisco UCS Manager GUI.
2. Click the Equipment icon at the top on the far left.
3. Choose Fabric Interconnects from the drop-down menu.
4. Choose the parent Fabric Interconnects menu.
5. In the right pane, choose Installed Firmware.
6. Open the UCS Manager drop-down menu.
7. Open the Fabric Interconnects drop-down menu.
8. Find the primary Fabric Interconnect (A or B) row and look in the Package Version column to determine what version the fabric interconnect is running.

If the package version is Release 4.2(2d) or Release 4.2(3b), you are using the affected release.

To check the fabric interconnect model, follow these steps:

1. Log in to the Cisco UCS Manager GUI.
2. Click the Equipment icon at the top on the far left.
3. Choose Fabric Interconnects from the drop-down menu.
4. Choose Fabric Interconnect A or Fabric Interconnect B.
5. In the Properties section, look for Product Name. If the product ID (PID) is one of the following three PIDs, you are using the affected hardware:
   • UCSC-FI-6454
   • UCS-FI-64108
   • UCS-FI-6536

For Cisco Intersight
To check the software release and hardware version, follow these steps:

1. Log in to Cisco Intersight.
2. From the Overview menu, choose the Operate drop-down menu from the left pane.
3. Choose Fabric Interconnects.
4. Look in the Model column. If the PID is one of the following three PIDs, you using the affected hardware:
   • UCSC-FI-6454
   • UCS-FI-64108
   • UCS-FI-6536
5. Look in the Bundle Version column. If the release listed is 4.2(2d) or 4.2(3b), you running the affected software.

Revision History

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.