Field Notice: FN74042 - Some Cisco ASR 9000 Series Aggregation Services Router Products May Experience Kernel Panic, Silent Reloading, or Hanging - BIOS/Firmware Upgrade Recommended
Available Languages
Notice
THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Products Affected
| Affected Product Name | Description | Comments |
|---|---|---|
| A99-12X100GE | ASR 9900 12-port 100GE Line Card | |
| A9K-24X10GE-1G-CM | ASR 9000 24-port 10GE & 1GE dual rate Consumption Model LC | |
| A9K-24X10GE-1G-SE | ASR 9000 24-port 10GE & 1GE dual rate SE LC | |
| A9K-24X10GE-1G-TR | ASR 9000 24-port 10GE & 1GE dual rate -TR LC | |
| A9K-48X10GE-1G-CM | ASR 9000 48-port 10G &1G dual rate Consumption model LC | |
| A9K-48X10GE-1G-SE | ASR 9000 48-port 10GE & 1GE dual rate SE LC | |
| A9K-48X10GE-1G-TR | ASR 9000 48-port 10GE & 1GE dual rate -TR LC | |
| A9K-4X100GE | ASR 9000 4-port 100GE LAN Line Card | |
| A9K-RSP880-LT-SE | ASR 9000 Route Switch Processor 880-LT for Service Edge | |
| A9K-RSP880-LT-TR | ASR 9000 Route Switch Processor 880-LT for Packet Trans | |
| ASR-9901 | ASR 9901 Compact Chassis, 2RU | ASR-9901-RP ASR-9901-LC |
| Affected Software Product | Affected Release | Affected Release Number | Comments |
|---|---|---|---|
| IOS XR Software | 7 | 7.8.1, 7.8.2, 7.9.1, 7.9.2 |
Defect Information
| Defect ID | Headline |
| CSCwf51196 | XR-VM Kernel Panic/Silent Reload/Hang due to Intel Microcode Errata BDE38 |
Problem Description
The Cisco ASR 9000 Series Aggregation Services Router products that are listed in the Products Affected section may experience kernel panic, may experience a silent reload with or without kernel dump being stored on the device, or may become unresponsive or hang if they are running one of the following releases of Cisco IOS XR Software:
- 7.8.1
- 7.8.2
- 7.9.1
- 7.9.2
Background
Cisco IOS XR Software releases 7.8.1, 7.8.2, 7.9.1, and 7.9.2 load the default BIOS instead of the latest BIOS version, exposing the CPU to BDE38 Errata. As a result, devices might experience kernel panic, may experience a silent reload with or without kernel dump, and may hang or become unresponsive under normal working conditions.
Following table shows the affected BIOS version for each of the affected product identifiers (PIDs):
| PID | Impacted Primary-BIOS Version | Non-Impacted Primary-BIOS Version with SMU |
|---|---|---|
| A9K-48X10GE-1G-SE/TR/CM A9K-24x10GE-1G-SE/TR/CM |
18.34 | 18.33 |
| A99-12x100GE | 9.34 | 9.33 |
| A9K-4X100GE | 9.34 | 9.33 |
| ASR-9901-RP | 22.27 | 22.26 |
| ASR-9901-LC | 23.23 | 23.22 |
| A9K-RSP880-LT-SE A9K-RSP880-LT-TR |
17.41 | 17.4 |
Problem Symptom
On the affected products that are experiencing the issue reported in this field notice, kernel core dump files (for example, default-sdr--1.20230716-005550.core.host.0_RSP0.kdump) may be created on the hard disk. If the RP on a Cisco ASR9901 router hangs, the router will be unresponsive until it is power-cycled. If the line card or RP hangs, the card will be in SW_INACTIVE state, as shown in the following example:
RP/0/RSP0/CPU0:xrg-403-asr9910#sh platform
Mon Aug 7 07:39:58.386 UTC
Node Type State Config state
--------------------------------------------------------------------------------
0/RSP0/CPU0 A99-RSP-SE(Active) IOS XR RUN NSHUT
0/RSP1/CPU0 A99-RSP-SE(Standby) IOS XR RUN NSHUT
0/FT0 ASR-9910-FAN OPERATIONAL NSHUT
0/FT1 ASR-9910-FAN OPERATIONAL NSHUT
0/0 A9K-24X10GE-1G-SE SW_INACTIVE NSHUT ...
Workaround/Solution
An affected system that experiences kernel panic or a silent reload will recover on its own. If an affected device hangs or is unresponsive, then a power cycle of the affected device, route switch processor (RSP), or line card is required.
Cisco recommends installing the SMUs for the products that are listed in the Products Affected section if they are running an affected software release. After SMU activation, a forced Field Programmable Device (FPD) upgrade is required for the affected device, RSP, or line card.
Upgrade the BIOS of an RSP and a line card with the force keyword, as shown in the following example:
#upgrade hw-module location 0/RSP0 fpd Primary-BIOS force
#upgrade hw-module location 0/0 fpd Primary-BIOS force
admin hw-module location 0/RSP0 reload noprompt
admin hw-module location 0/0 reload noprompt
After SMU activation and FPD force upgrade, use the CLI commands that are provided in the Problem Symptom section to validate the CPU microcode revision.
How to Identify Affected Products
To determine the FPD version, use the show hw-module fpd command. The following example shows the output from an affected ASR 9901 platform:
RP/0/RSP0/CPU0:ASR-9901-A#show hw-module fpd
Mon Aug 7 07:43:35.678 UTC
Auto-upgrade:Enabled
FPD Versions
==============
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------------
0/RSP0 ASR-9901-RP 1.0 CBC CURRENT 54.11 54.11
0/RSP0 ASR-9901-RP 1.0 Drax-FPGA CURRENT 0.38 0.38
0/RSP0 ASR-9901-RP 1.0 IPU-FPGA CURRENT 2.05 2.05
0/RSP0 ASR-9901-RP 1.0 IPU-FSBL CURRENT 1.104 1.104
0/RSP0 ASR-9901-RP 1.0 IPU-Linux CURRENT 1.104 1.104
0/RSP0 ASR-9901-RP 1.0 Primary-BIOS CURRENT 22.27 22.27
0/RSP0 ASR-9901-RP 1.0 SSDa-MICRON N/A 7.05 7.05
0/RSP0 ASR-9901-RP 1.0 SSDb-MICRON N/A 7.05 7.05
0/0 ASR-9901-LC 1.0 CBC CURRENT 55.07 55.07
0/0 ASR-9901-LC 1.0 Gamora-FPGA CURRENT 0.36 0.36
0/0 ASR-9901-LC 1.0 IPU-FPGA CURRENT 1.10 1.10
0/0 ASR-9901-LC 1.0 IPU-FSBL CURRENT 1.104 1.104
0/0 ASR-9901-LC 1.0 IPU-Linux CURRENT 1.104 1.104
0/0 ASR-9901-LC 1.0 Primary-BIOS CURRENT 23.23 23.23
0/0 ASR-9901-LC 1.0 SSDa-MICRON N/A 7.05 7.05
0/PT0 A9K-1600W-AC 0.0 PM0-PO-PriMCU CURRENT 17.137 17.137
0/PT0 A9K-1600W-AC 0.0 PM1-PO-PriMCU CURRENT 17.137 17.137
RP/0/RSP0/CPU0:ASR-9901-A#
Revision History
| Version | Description | Section | Date |
| 1.0 | Initial Release | — | 2023-NOV-06 |
For More Information
For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:
Receive Email Notification About New Field Notices
To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)