Contents

Introduction

Background

Prerequisites

Requirements

Components Used

Symptom

Solution

Related Information

Nexus 7000 Supervisor 2/2E Compact Flash Failure Recovery

Introduction

This article is an extension to the document “Nexus 7000 Supervisor 2/2E Compact Flash Failure Recovery” that addresses all possible failure scenarios. A possibility where flash recovery tool fails to run, this document may come handy. It is recommended to have console access to the device to perform the changes. Also, it is strongly recommended to not make any changes under the Linux kernel, which is not mentioned in the document, as this may have an impact on the switch operations. Cisco TAC supervision is advisable.

Background

As explained in the other document, each N7K supervisor 2/2E is equipped with 2 eUSB flash devices in RAID1 configuration, one primary and one mirror. Together they provide non-volatile repositories for boot images, startup configuration and persistent application data. In a situation where the Raid fails for a supervisor in the chassis, we run the flash recovery tool, to fix the same. In almost all cases, we resort to reloading/failing over the supervisor, if the flash recovery tool fails to run. There is a possibility to fix this without a reload/failover in certain scenario.

Prerequisites

Requirements

Cisco recommends that you have knowledge ofCisco Nexus OS, storage or flash disk recovery methods and Linux level debugging.

Components Used

Nexus 7000 series switches

Symptom

Raid failure is observed on a supervisor and while trying to recover the flash for the affected supervisors, following error appears when running the flash recovery tool,

Switches would run into Raid failure state with error code - 0xe1

ERROR: Cannot perform recovery. /dev/sdb has incorrect partition info.

ERROR: Disk /dev/sdb needs to be manually inspected for errors.

INFO: No recovery was attempted on module 5. All flashes left intact.

INFO: A detailed copy of the this log was saved as volatile:flash_repair_log_mod5.tgz.

Solution

Load the debug plugin on the switch, to login to the linux shell,

Switch# load bootflash:n7000-s2-debug-sh.6.1.4a.gbin

Please be careful, while running the commands here.

Once we get the linux prompt, look for the affected partition as per the error message. In our case it is /dev/sdb. It could be some other partitions too.

Linux(debug)# ls -l /dev/sd?

brw-r----- 1 root root 8, 0 Aug 28 2015 sda

brw-rw-r-- 1 root disk 8, 32 Dec 18 2013 sdc

brw-rw-r-- 1 root disk 8, 48 Dec 18 2013 sdd

brw-rw-r-- 1 root disk 8, 64 Dec 18 2013 sde

brw-rw-r-- 1 root disk 8, 80 Dec 18 2013 sdf

brw-rw-r-- 1 root disk 8, 96 Dec 18 2013 sdg

brw-rw-r-- 1 root disk 8, 112 Dec 18 2013 sdh

brw-rw-r-- 1 root disk 8, 128 Dec 18 2013 sdi

brw-rw-r-- 1 root disk 8, 144 Dec 18 2013 sdj

brw-rw-r-- 1 root disk 8, 160 Dec 18 2013 sdk

brw-rw-r-- 1 root disk 8, 176 Dec 18 2013 sdl

brw-rw-r-- 1 root disk 8, 192 Dec 18 2013 sdm

The partition is found to be missing, leading to error, while running the recovery tool. Create the missing partition manually, with same permission as other blocks.

Linux(debug)# mknod -m 664 /dev/sdb b 8 16

Now, we can see the sdb partition under /dev,

Linux(debug)# ls -l /dev/sd?

brw-r----- 1 root root 8, 0 Aug 28 2015 sda

brw-rw-r-- 1 root root 8, 16 May 26 07:31 sdb

brw-rw-r-- 1 root disk 8, 32 Dec 18 2013 sdc

brw-rw-r-- 1 root disk 8, 48 Dec 18 2013 sdd

brw-rw-r-- 1 root disk 8, 64 Dec 18 2013 sde

brw-rw-r-- 1 root disk 8, 80 Dec 18 2013 sdf

brw-rw-r-- 1 root disk 8, 96 Dec 18 2013 sdg

brw-rw-r-- 1 root disk 8, 112 Dec 18 2013 sdh

brw-rw-r-- 1 root disk 8, 128 Dec 18 2013 sdi

brw-rw-r-- 1 root disk 8, 144 Dec 18 2013 sdj

brw-rw-r-- 1 root disk 8, 160 Dec 18 2013 sdk

brw-rw-r-- 1 root disk 8, 176 Dec 18 2013 sdl

brw-rw-r-- 1 root disk 8, 192 Dec 18 2013 sdm

Exit from the linux shell and run the flash recovery tool again.

This time without any error messages and the Raid failure on the primary flash was recovered (0xf0). Confirmed the same using the command,

"slot x show system internal raid | i i cmos|block | head line 5"

It should run fine without such errors and should be able to recover the affected Supervisor from the Raid failure state. In case, recovery tool continues to fail to run, it could be due to another reason, or an actual corruption with the partition, and we may have to resort to a reload/failover.

Related Information

Nexus 7000 Supervisor 2/2E Compact Flash Failure Recovery