Field Notice: FN74414 - A Firmware Issue in Some Cisco NCS 5500 Series Routers May Cause Filesystem to Be Remounted to Read-Only Mode, Resulting in Kernel Panic - Software Upgrade Recommended

An issue that was identified in a specific solid-state drive (SSD) firmware may cause the SSD to be remounted in read-only (RO) mode. This leads to non-deterministic behavior of the Cisco IOS XR Software system, including kernel panic on the route processor.

This issue occurs on Cisco NCS 5500 Series Routers that use SSDs with firmware releases D0MU071, D0MU075, and D0MU077. These SSDs use ext4 filesystems that are mounted in read-write (RW) mode initially. When the kernel detects a filesystem journal abort on an ext4 filesystem, it automatically remounts it in RO mode without notifying Cisco IOS XR Software or SysAdmin. 

When the kernel detects a filesystem journaling error, it does one of the two following actions:

1. If the error is detected on a route processor card and the Cisco IOS XR Software release is 7.10.1 or later, kernel panic is raised, resulting in immediate card reset. The following message will be observed on the console just before the card reset:

[ 5009.233968] Kernel panic - not syncing: EXT4-fs panic from previous error

2. In all other cases, the Linux kernel remounts the filesystem in RO mode. A debug-level syslog message is reported indicating read-only filesystem: 

LC/0/2/CPU0: syslog_dev[111]: resmon[211] PID-20148: start_backing_thread:bind: Read-only file system 

LC/0/2/CPU0: syslog_dev[111]: resmon[211] PID-20150: start_backing_thread:bind: Read-only file system 

LC/0/2/CPU0: syslog_dev[111]: resmon[211] PID-20152: start_backing_thread:bind: Read-only file system 

LC/0/2/CPU0: syslog_dev[111]: resmon[211] PID-20151: start_backing_thread:bind: Read-only file system

 

Filesystems That Are Remounted in RO Mode

To verify whether a file system is mounted in a RO mode inspect the /proc/mounts in Linux shell. It is possible to limit the check to important filesystems only. On any Cisco IOS XR platform and release combination, the following command sequence should produce consistent output:

RP/0/RP0/CPU0:ios#run

[xr-vm_node0_RP0_CPU0:~]$df | awk 'NR>1 {print $6}' | xargs -I % grep " % .*ro,” /proc/mounts

[xr-vm_node0_RP0_CPU0:~]$

 

sysadmin-vm:0_RP0# run

[sysadmin-vm:0_RP0:~]$df | awk 'NR>1 {print $6}' | xargs -I % grep " % .*ro," /proc/mounts

[sysadmin-vm:0_RP0:~]$

Solution

To resolve this issue, Cisco recommends upgrading to Cisco IOS XR Software Release 26.1.2 or later. Affected units can be recovered using the following manual recovery procedure. Hardware replacement is not necessary to resolve this condition.

1. To recover from the SSD RO condition, reload the affected module or box. If the card does not automatically reload, manually reload it using the following command in admin mode: 

hw-module location <location> reload noprompt

Example: 

sysadmin-vm:0_RP0# hw-module location 0/0 reload noprompt

2. After recovery, install the SMU that integrates the updated Micron 5100 SSD firmware release D0MU078, as shown in the following table:

To determine the affected SSD model and firmware release, run the following command: 

RP/0/RP0/CPU0:R3#admin show smart-monitor location 0/RP0
 Tue Feb 3 07:31:27.930 CET 

************************************************************

Location : 0/RP0

************************************************************ 

======== SmartCtl info for sda ======== 
smartctl 7.1 2021-03-08 r5212 [x86_64-linux-5.4.142-yocto-standard] (local build) 
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org 
=== START OF INFORMATION SECTION ===

<snip>

Device Model: Micron_5100_MTFDDAV******    <--- Affected SSD model
Serial Number: MSAXXXXXX 

<snip>

Firmware Version: D0MU077                  <--- If the current firmware release is 77 or earlier, apply the solution