Field Notice: FN74415 - Firmware Issue in Cisco ASR9000 Series Routers May Cause Filesystem to Be Remounted to Read-Only Mode, Resulting in Kernel Panic - Software Upgrade Recommended

An issue that was identified in a specific solid-state drive (SSD) firmware may cause the SSD to be remounted in read-only (RO) mode. This leads to non-deterministic behavior of the Cisco IOS XR Software system, including kernel panic on the route processor.  

The issue occurs on some of the Cisco ASR9000 Series Routers that use SSDs with firmware releases D0MU077 and earlier. These SSDs use ext4 filesystems mounted in read-write (RW) mode initially. When the Linux kernel detects a filesystem journal abort on an ext4 filesystem, it automatically remounts it in RO mode without notifying Cisco IOS XR Software or SysAdmin. 

When the kernel detects a filesystem journaling error, it does one of the two following actions:

1. If the error is detected on a route processor card and the Cisco IOS XR Software release is 7.11.21 or later, kernel panic is raised, resulting in immediate card reset. The following message will be observed on the console just before the card reset:

[ 5009.233968] Kernel panic - not syncing: EXT4-fs panic from previous error

2. In all other cases, the Linux kernel remounts the filesystem in RO mode. A debug-level syslog message is reported indicating read-only filesystem: 

LC/0/4/CPU0:2025 Jan 10 13:59:26.907 : syslog_dev[113]: syslog_infra_hm[359] PID-31576: start_backing_thread:bind: Read-only file system 
RP/0/RSP1/CPU0:2026 Jan 13 05:27:24.862 : syslog_dev[111]: ntpd[435] PID-30056: start_backing_thread:bind: Read-only file system 
RP/0/RSP1/CPU0:2026 Jan 13 06:10:36.491 : syslog_dev[111]: fs_server[412] PID-2681: start_backing_thread:bind: Read-only file system

Filesystems Remounted in RO Mode

If a filesystem was remounted in RO mode in the SysAdmin VM of the active route processor, access to the SysAdmin VM may not be possible, resulting in the following sample output:

RP/0/RSP0/CPU0:xrg-402-asr9906#admin
start_backing_thread:bind: Read-only file system

To verify whether a file system is mounted in RO mode, inspect the /proc/mounts in Linux shell. It is possible to limit the check to important filesystems only. On any Cisco IOS XR Software release and platform combination, the following command sequence should produce consistent output:

RP/0/RSP0/CPU0:asr9906#run
[xr-vm_node0_RSP0_CPU0:~]$df | awk 'NR>1 {print $6}' | xargs -I % grep " % .*ro," /proc/mounts
 [xr-vm_node0_RSP0_CPU0:~]$exit
 
RP/0/RSP0/CPU0:asr9906#admin 
sysadmin-vm:0_RSP0# run  
[sysadmin-vm:0_RSP0:~]$df | awk 'NR>1 {print $6}' | xargs -I % grep " % .*ro," /proc/mounts
[sysadmin-vm:0_RSP0:~]$

Solution

To resolve this issue, Cisco recommends upgrading to Cisco IOS XR Software Release 25.02.21, 26.1.2, or later. Affected units can be recovered using the following manual recovery procedure. Hardware replacement is not necessary to resolve this condition.

1. To recover from the SSD RO condition, reload the affected module or box. If the card does not automatically reload, manually reload it using the following command in admin mode: 

hw-module location <location> reload noprompt

Example: 

sysadmin-vm:0_RP0# hw-module location 0/0 reload noprompt

2. After recovery, install the SMU that integrates the updated Micron 5100 SSD firmware release D0MU078, as shown in the following table:

To determine the affected SSD model and firmware release, run the following command: 

sysadmin-vm:0_RSP1# show smart-monitor location all | i "Model|Firmware|Location"

Location : 0/1
Model Family:     Micron 5100 Pro / 52x0 / 5300 SSDs <--- Affected SSD model
Device Model:     Micron_5100_MTFDDAV240TCB
Firmware Version: D0MU075                            <--- If the current firmware release is 77 or earlier, apply the solution