Field Notice: FN74314 - Firmware Issue in One of the SSD Models Used in Cisco ASR 9000 Series Products Causing Boot Failures During Platform Migration - Workaround Provided

Cisco has identified a firmware issue in one of the SSD models used in multiple Cisco ASR 9000 Series Route Processors (RPs), Route-Switch Processors (RSPs), and Lightspeed Plus-based line cards. This issue causes boot failures during migration activities.

This problem occurs in SSDs that are running firmware version FW1403. Customers have reported boot failures during migration that are caused by I/O timeouts that disrupt normal drive operations. 

Impacted SSD drives may be unable to boot and may display I/O errors and messages such as No install boot device found. The firmware enters an assert state, resulting in the system not recognizing the drive.

Impacted SSDs may repeatedly display errors related to disk-read operations that prevent normal boot-up.

In case of failure, during card boot-up, logs may appear as follows:

. . .
Starting S.M.A.R.T. daemon: smartd.
/etc/rc.d/init.d/pd-functions: line 75: /misc/scratch/pcie/kmsg.log: No such file or directory
Mon Jul 7 03:30:56 UTC 2025: PASSING control to /etc/init.d/pxe_install.sh start
Logging started at Mon Jul 7 03:30:56 UTC 2025...
[ 73.949948] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 73.956387] ata1.00: irq_stat 0x40000001
[ 73.960302] ata1.00: failed command: READ DMA
[ 73.964656] ata1.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 27 dma 4096 in
[ 73.964656] res 51/04:08:00:00:00/00:00:00:00:00/e0 Emask 0x1 (device error)
[ 73.979710] ata1.00: status: { DRDY ERR }
[ 73.983710] ata1.00: error: { ABRT }
[ 73.999905] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 74.006326] ata1.00: irq_stat 0x40000001
[ 74.010240] ata1.00: failed command: READ DMA
. . .

To help prevent potential failures, Cisco recommends following the procedure outlined below, which includes two main activities:

• Deploy and run the tool to analyze the installed SSD.
• Deploy the latest SSD firmware.

These activities should be performed only if no boot failure has been reported.

Note: If an SSD is already affected by this issue and is exhibiting symptoms such as boot failure or buffer allocation issues, please engage the Cisco Technical Assistance Center (TAC) for recovery assistance.

1. Deploy the SSD Analysis Tool Using SMU CSCws13902

Before proceeding with this step, ensure that the CSCws13902 SMU is downloaded and installed on the currently deployed release. The SMU installation is expected to be hitless.

After the installation is complete, run the script as shown in the following example:

sysadmin-vm:0_RSP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ cd /opt/cisco/calvados/bin
[sysadmin-vm:0_RSP0:/opt/cisco/calvados/bin]$ python3 FW1403_analyzer.py 
Location     Device   Model                Firmware   Status
----------------------------------------------------------------------
0/LC0        sda      SRM28240GF1M2AC1C    FW1403     PASS
0/RSP1       sda      SRM28240GF1M2AC1C    FW1403     PASS
0/RSP1       sdb      SRM28240GF1M2AC1C    FW1403     PASS
Results saved to: /var/log/ssd_analysis_results.json

At any point, one of two possible outcomes (PASS or FAIL) will be displayed in the Status column:

• Proceed to the next step only if all entries in the table show a PASS status.
• If any entry displays FAIL, contact the Cisco TAC for further assistance.

Note: The script will identify only cards with the SSD model SRM28240GF1M2AC1C in the output.

2. Deploy the Latest SSD Firmware Using SMU CSCws14032

This step involves downloading and installing the CSCws14032 SMU on the currently deployed release. Note that installing this SMU is expected to trigger a full chassis reload.

The SMU installation will make the new firmware available, and during the device reload, the updated SSD firmware will be automatically deployed to the SSD.

After the SMU installation, run the previously mentioned script for verification as shown in the following example:

sysadmin-vm:0_RSP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ cd /opt/cisco/calvados/bin
[sysadmin-vm:0_RSP0:/opt/cisco/calvados/bin]$ python3 FW1403_analyzer.py 
Location     Device   Model                Firmware   Status
----------------------------------------------------------------------
0/LC0        sda      SRM28240GF1M2AC1C    FW1601     PASS
0/RSP1       sda      SRM28240GF1M2AC1C    FW1601     PASS
0/RSP1       sdb      SRM28240GF1M2AC1C    FW1601     PASS
Results saved to: /var/log/ssd_analysis_results.json

A successful execution will display all entries with firmware FW1601 and a status of PASS.

To determine if an RP, RSP, or LC is affected, follow the steps below:

1. Refer to the Serial Number Validation section of this Field Notice. If an RP, RSP, or LC is identified as affected, proceed to verify the firmware version.
2. Before any failure occurs, check the device model and firmware version using the following command:

admin show smart-monitor location 0/rsp0 | inc "Location|Device Model|Firmware Version|SmartCtl info for"
Location: 0/RSP0
======== SmartCtl info for sda ========
Device Model : SRM28240GF1M2AC1C     <<<<< SSD
Firmware Version : FW1403            <<<<< Affected Firmware version
======== SmartCtl info for sdb ========
Device Model: SRM28240GF1M2AC1C      <<<<< SSD 
Firmware Version : FW1403            <<<<< Affected Firmware version
======== SmartCtl info for sdc ========

• In general, new card installation should be performed only after both mandatory SMUs have been deployed on the node.

• The new firmware introduces additional parameters in the show smart-monitor output. These can be disregarded from an operational standpoint, as high values do not indicate a failure.

admin show smart-monitor location <RP/LC> | utility egrep -e "^199"
199 UDMA_CRC_Error_Count    -O--CK   100   100   000    -    51539607552
199 UDMA_CRC_Error_Count    -O--CK   100   100   000    -    14349485735936

• If the analyzer is run on a device that does not contain any of the specified SSDs, the script output will appear as follows:

sysadmin-vm:0_RSP0# run chvrf 0 bash -l
[sysadmin-vm:0_RSP0:~]$ cd /opt/cisco/calvados/bin
[sysadmin-vm:0_RSP0:/opt/cisco/calvados/bin]$ python3 FW1403_analyzer.py 
No card found in chassis with target SSDs, no need of further analysis or upgrade on chassis required. exiting...