Contents

Introduction

Background Information

Prerequisites

Requirements

Components Used

Upgrade Procedure

Caveats

Related Information

Introduction

This document describes the procedure to upgrade the N7K-SUP2/E bootflash firmware on both the active and standby supervisor to permanently resolve the firmware issue addressed in the software defect CSCus22805 : N7K-SUP2/E: eUSB Flash Failure or Unable to Save Configuration and CSCuv18883 N77-SUP2E eUSB Compact Flash Failure or Unable to Save Configuration.

Note: This procedure has been extended to cover a wider range of bootflash vendor parts starting in NX-OS 6.2(20) and tracked by CSCvf36683 - N7K-SUP2/E: eUSB Flash Failure or Unable to Save Configuration.

Background Information

This procedure upgrades the bootflash firmware on both supervisors in a Nexus 7000 chassis while remaining on your current software version.  As an alternative, you may upgrade to an NX-OS software release which contains the firmware fix for CSCus22805.

Another option is to use the flash recovery tool (available for download) and can automatically fix any single flash errors when present. This is not a permanent fix. This procedure highlighted in this document upgrades the bootflash firmware permanently so that the onboard embedded flash devices are no longer susceptible to the RAID failure.

Each N7K supervisor 2/2E is equipped with 2 eUSB flash devices in RAID1 configuration, one primary and one mirror. Together they provide non-volatile repositories for boot images, startup configuration, and persistent application data.

What can happen is over a period of months or years in service, one of these devices may be disconnected from the USB bus, causing the RAID software to drop the device from the configuration. The device can still function normally with 1/2 devices. However, when the second device drops out of the array, the bootflash is remounted as read-only, meaning we cannot save configuration or files to the bootflash, or allow the standby to sync to the active in the event it is reloaded.

There is no operational impact on systems running in a dual flash failure state, however a reload of the affected supervisor is needed to recover from this state. Furthermore, any changes to running configuration will not be reflected in startup and would be lost in the event of a power outage.

Note:  CSCus22805 affects Nexus 7000 Supervisor 2 (N7K-SUP2) and Supervisor 2E (N7K-SUP2E) modules that run NX-OS releases from Release 6.1(1) to 6.2(12). NX-OS 6.2(14) is still vulnerable but contains automated workaround/recovery.

CSCuv18883 affects Nexus Supervisor 2E (N77-SUP2E) modules that run NX-OS releases from Release 6.1(1) to 6.2(14).

CSCvf36683 affects Nexus 7000 Supervisor 2 (N7K-SUP2) and Supervisor 2E (N7K-SUP2E) / (N77-SUP2E) modules that run NX-OS releases from Release 6.1(1) to 6.2(18), 7.2(1)D1(1) to 7.2(2)D1(2), 7.3(0)D1(1) to 7.3(2)D1(2), 8.0(1), 8.1(1), 8.2(1)

To summarize, for the 6.2 maintenance release,  6.2(20) has a fix for all 3 software defects mentioned in the above note.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

- Cisco Nexus 7000 Series Switch, Release 6.2(10)

- N7K SUP2

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Upgrade Procedure

1. Console into both the active and standby supervisors.

2. Copy NX-OS 7.2(1)D1(1) onto the bootflash of both the active and standby supervisors.

3. Set the kickstart and system boot variables on the standby to NX-OS 7.2(1)D1(1).

N7K-1(config)# boot kickstart bootflash:n7000-s2-kickstart.7.2.1.D1.1.bin sup-2

Performing image verification and compatibility check, please wait....

Note: system and kickstart bootvars are pointing to incompatible images

N7K-1(config)# boot system bootflash:n7000-s2-dk9.7.2.1.D1.1.bin sup-2

Performing image verification and compatibility check, please wait....

4. Save the configuration for all VDCs and confirm the boot variable on next reload is set to NX-OS 7.2(1)D1(1) on the standby supervisor.

N7K-1# copy running-config startup-config vdc-all

[########################################] 100%

Copy complete.

N7K-1# show mod

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

1    0      Supervisor Module-2                 N7K-SUP2           active *

2    0      Supervisor Module-2                 N7K-SUP2           ha-standby

4    48     1/10 Gbps Ethernet Module           N7K-F248XP-25E     ok

N7K-1# show boot

Current Boot Variables:

sup-1

kickstart variable = bootflash:/n7000-s2-kickstart.6.2.10.bin

system variable = bootflash:/n7000-s2-dk9.6.2.10.bin

sup-2

kickstart variable = bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin

system variable = bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin

No module boot variable set

Boot Variables on next reload:

sup-1

kickstart variable = bootflash:/n7000-s2-kickstart.6.2.10.bin

system variable = bootflash:/n7000-s2-dk9.6.2.10.bin

sup-2

kickstart variable = bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin

system variable = bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin

No module boot variable set

5. Reload the standby supervisor.

N7K-1# reload module 2

This command will reboot standby supervisor module. (y/n)?  [n] y

6. The standby supervisor boots 7.2(1)D1(1) kickstart image and upgrades the bootflash firmware. Once the standby supervisor attempts to boot 7.2(1)D1(1) system image, it will detect the version mismatch and reload itself with the previous image from the active supervisor.

NX7k SUP BIOS version ( 2.12 ) : Build - 05/29/2013 11:58:20

PM FPGA Version : 0x00000022

Power sequence microcode revision - 0x00000009 : card type - 10156EEA0

Booting Spi Flash : Primary

CPU Signature - 0x000106e4: Version - 0x000106e0

CPU - 1 : Cores - 4 : HTEn - 1 : HT - 2 : Features - 0xbfebfbff

FSB Clk - 532 Mhz :  Freq - 2154 Mhz - 2128 Mhz

MicroCode Version : 0x00000002

Memory - 12288 MB : Frequency - 1067 MHZ

Loading Bootloader: Done

IO FPGA Version   : 0x1000c

PLX Version       : 861910b5

Bios digital signature verification - Passed

USB bootflash status : [1-1:1-1]

Reset Reason Registers: 0x0 0x8

Filesystem type is ext2fs, partition type 0x83

GNU GRUB  version 0.97

Autobooting bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin bootflash:/n7000-s2-dk

9.7.2.1.D1.1.bin...

Filesystem type is ext2fs, partition type 0x83

Booting kickstart image: bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin....

...............................................................................

.............................................

Kickstart digital signature verification Successful

Image verification OK

ˇ

INIT: version 2boot device node /dev/sdc

Bootflash firmware upgraded successfully

boot device node /dev/sdc

boot mirror device node /dev/sdb

Bootflash mirror firmware upgraded successfully

boot mirror device node /dev/sdb

obfl device node /dev/sda

OBFL firmware upgraded successfully

obfl device node /dev/sda

Checking obfl filesystem.r

Checking all filesystems..r.r.r.retval=[1]

r done.

Starting mcelog daemon

rCreating logflash directories

Loading system software

/bootflash//n7000-s2-dk9.7.2.1.D1.1.bin read done

System image digital signature verification successful.

Uncompressing system image: bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin Wed Mar 2 15:56:21 UTC 2016

blogger: nothing to do.

C

..done Wed Mar 2 15:56:27 UTC 2016

INIT: Entering runlevel: 3

7. Once the standby supervisor is back into an ha-standby state, change the boot variables back to the correct NX-OS version and perform a supervisor switchover to failover the supervisors.

N7K-1# show module

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

1    0      Supervisor Module-2                 N7K-SUP2           active *

2    0      Supervisor Module-2                 N7K-SUP2           ha-standby

4    48     1/10 Gbps Ethernet Module           N7K-F248XP-25E     ok

N7K-1(config)# boot kickstart bootflash:n7000-s2-dk9.6.2.10.bin sup-2

Performing image verification and compatibility check, please wait....

Note: system and kickstart bootvars are pointing to incompatible images

N7K-1(config)# boot system bootflash:n7000-s2-dk9.6.2.10.bin sup-2

Performing image verification and compatibility check, please wait....

N7K-1# copy running-config startup-config vdc-all

[########################################] 100%

Copy complete.

N7K-1# system switchover

8. Confirm the previous active supervisor boots into an ha-standby state and set the boot variables on the standby to NX-OS 7.2(1)D1(1).

N7K-1# show module

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

1    0      Supervisor Module-2                 N7K-SUP2           ha-standby

2    0      Supervisor Module-2                 N7K-SUP2           active *

4    48     1/10 Gbps Ethernet Module           N7K-F248XP-25E     ok

N7K-1(config)# boot kickstart bootflash:n7000-s2-kickstart.7.2.1.D1.1.bin sup-1

Performing image verification and compatibility check, please wait....

Note: system and kickstart bootvars are pointing to incompatible images

N7K-1(config)# boot system bootflash:n7000-s2-dk9.7.2.1.D1.1.bin sup-1

Performing image verification and compatibility check, please wait....

9. Save the configuration for all VDCs and confirm the boot variable on next reload is set to NX-OS 7.2(1)D1(1) for the standby supervisor.

N7K-1# copy running-config startup-config vdc-all

[########################################] 100%

Copy complete.

N7K-1# show boot

Current Boot Variables:

sup-1

kickstart variable = bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin

system variable = bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin

sup-2

kickstart variable = bootflash:/n7000-s2-kickstart.6.2.10.bin

system variable = bootflash:/n7000-s2-dk9.6.2.10.bin

No module boot variable set

Boot Variables on next reload:

sup-1

kickstart variable = bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin

system variable = bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin

sup-2

kickstart variable = bootflash:/n7000-s2-kickstart.6.2.10.bin

system variable = bootflash:/n7000-s2-dk9.6.2.10.bin

No module boot variable set

9. Reload the standby supervisor.

N7K-1# reload module 1

This command will reboot standby supervisor module. (y/n)?  [n] y

10. The standby supervisor will boot 7.2(1)D1(1) kickstart image and upgrade the bootflash firmware. Once the standby supervisor attempts to boot 7.2(1)D1(1) system image it will detect the version mismatch and reload itself with the previous image from the active supervisor.

NX7k SUP BIOS version ( 2.12 ) : Build - 05/29/2013 11:58:20

PM FPGA Version : 0x00000022

Power sequence microcode revision - 0x00000009 : card type - 10156EEA0

Booting Spi Flash : Primary

CPU Signature - 0x000106e4: Version - 0x000106e0

CPU - 1 : Cores - 4 : HTEn - 1 : HT - 2 : Features - 0xbfebfbff

FSB Clk - 532 Mhz :  Freq - 2154 Mhz - 2128 Mhz

MicroCode Version : 0x00000002

Memory - 12288 MB : Frequency - 1067 MHZ

Loading Bootloader: Done

IO FPGA Version   : 0x1000c

PLX Version       : 861910b5

Bios digital signature verification – Passed

USB bootflash status : [1-1:1-1]

Reset Reason Registers: 0x0 0x8

Filesystem type is ext2fs, partition type 0x83

GNU GRUB  version 0.97

Autobooting bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin...

Filesystem type is ext2fs, partition type 0x83

Booting kickstart image: bootflash:/n7000-s2-kickstart.7.2.1.D1.1.bin....

...............................................................................

.............................................

Kickstart digital signature verification Successful

Image verification OK

ˇ

INIT: version 2boot device node /dev/sdc

Bootflash firmware upgraded successfully

boot device node /dev/sdc

boot mirror device node /dev/sdb

Bootflash mirror firmware upgraded successfully

boot mirror device node /dev/sdb

obfl device node /dev/sda

OBFL firmware upgraded successfully

obfl device node /dev/sda

Checking obfl filesystem.r

Checking all filesystems..r.r.r.retval=[1]

r done.

Starting mcelog daemon

rCreating logflash directories

Loading system software

/bootflash//n7000-s2-dk9.7.2.1.D1.1.bin read done

System image digital signature verification successful.

Uncompressing system image: bootflash:/n7000-s2-dk9.7.2.1.D1.1.bin Wed Mar 2 15:56:21 UTC 2016

blogger: nothing to do.

C

11. When the standby supervisor comes back into ha-standby state, you will want to change the boot variable back to NX-OS 6.2(10) and save the configuration.

N7K-1# show module

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

1    0      Supervisor Module-2                 N7K-SUP2           ha-standby

2    0      Supervisor Module-2                 N7K-SUP2           active *

4    48     1/10 Gbps Ethernet Module           N7K-F248XP-25E     ok

N7K-1(config)# boot kickstart bootflash:n7000-s2-dk9.6.2.10.bin sup-2

Performing image verification and compatibility check, please wait....

Note: system and kickstart bootvars are pointing to incompatible images

N7K-1(config)# boot system bootflash:n7000-s2-dk9.6.2.10.bin sup-2

Performing image verification and compatibility check, please wait....

N7K-1# copy running-config startup-config vdc-all

[########################################] 100%

Copy complete.

12. Once you complete the above steps on one member of the vPC complex, you will want to validate services and continue with carrying out the same procedure on the second member of the vPC pair.

Caveats

The procedure requires a supervisor switchover which is non disruptive. However, it is advisable to perform these steps at the time of a change window, in order to avoid any surprises.

Related Information

- Field Notice: FN - 63975 - Nexus 7000 Supervisor 2 and 2E Embedded Flash Write Error

- Nexus 7000 Supervisor 2 Flash Recovery Tool. See Flash_Recovery_Tool_ReadMe file in the tarfile for more details.

- Technical Support & Documentation - Cisco Systems