Recover Catalyst 9000 Switches from Upgrade Failures

Introduction

This document describes the common failures that occur when Catalyst 9000 series devices are upgraded and provides steps to recovery.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• How to upgrade Catalyst 9000 series switches.

Components Used

This document is based on this hardware version:

• Catalyst 9000 series switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Catalyst 9000 switches utilize Cisco IOS® XE software and operate in two modes, Install mode and Bundle mode.

Install mode is the newer and recommended mode to run. Install mode uses a package-provision file named packages.conf in order to boot the switch. In addition, there are a number of .pkg files that accompany packages.conf files in the flash drive. It is not recommended to alter the packages.conf and .pkg files.

Bundle mode uses the monolithic Cisco IOS image (.bin file) to boot the switch. The Bundle mode consumes more memory than the Install mode because the packages are extracted from the bundle and copied to RAM. You can run the Catalyst 9000 switches in either modes.

This is a partial sample output of some commands to verify the mode of operation. In INSTALL Mode:

Switch#show version | be Mode 
Switch Ports Model                 SW Version        SW Image        Mode
------ ----- -----                 ----------        ----------      ----
* 2    60    C9500-12Q             16.12.4           CAT9K_IOSXE     INSTALL <-------

Switch#show run | inc boot system 
boot system bootflash:packages.conf <-------

Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;    <-------
--snip--

This is a partial sample output of some commands to verify the mode of operation. In BUNDLE Mode:

Switch#show version | be Mode 
Switch Ports Model                  SW Version        SW Image          Mode
------ ----- -----                  ----------        ----------        ----
*    2 60    C9500-12Q              16.12.4           CAT9K_IOSXE       BUNDLE  <-------

Switch#show run | inc boot system 
boot system bootflash:cat9k_iosxe.16.12.04.SPA.bin <------- BUNDLE mode .bin image

Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = bootflash:cat9k_iosxe.16.12.04.SPA.bin; <------- switch set to boot from .bin image
--snip-- 

Failure Scenarios

Switch Boots up on Previous Version After Reload in Install Mode

This can happen if the switch boot statement still points to a previous version instead of the new version. To verify this, you can do these procedure:

Step 1. Verify the boot variable with the command  show boot .

Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = yes
Boot Mode = DEVICE
iPXE Timeout = 0

Step 2. Verify what version the current .conf file belongs to with the command  more flash:packages.conf | include pkg .

Switch#more flash:packages.conf | in pkg
boot rp 0 0 rp_boot cat9k-rpboot.17.03.02a.SPA.pkg
iso rp 0 0 rp_base cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 0 0 rp_daemons cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 0 0 rp_iosd cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 0 0 rp_security cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 0 0 rp_wlc cat9k-wlc.17.03.02a.SPA.pkg
iso rp 0 0 rp_webui cat9k-webui.17.03.02a.SPA.pkg
iso rp 0 0 srdriver cat9k-srdriver.17.03.02a.SPA.pkg
iso rp 0 0 guestshell cat9k-guestshell.17.03.02a.SPA.pkg
boot rp 1 0 rp_boot cat9k-rpboot.17.03.02a.SPA.pkg
iso rp 1 0 rp_base cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 1 0 rp_daemons cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 1 0 rp_iosd cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 1 0 rp_security cat9k-rpbase.17.03.02a.SPA.pkg
iso rp 1 0 rp_wlc cat9k-wlc.17.03.02a.SPA.pkg
iso rp 1 0 rp_webui cat9k-webui.17.03.02a.SPA.pkg
iso rp 1 0 srdriver cat9k-srdriver.17.03.02a.SPA.pkg
iso rp 1 0 guestshell cat9k-guestshell.17.03.02a.SPA.pkg

Step 3. If the file points to a previous version, check flash for correct .conf file with the commands dir flash: | include conf   and  more flash:<filename>.conf | include pkg .

Switch#dir flash: | in conf
81132 -rw- 7773 Jan 5 2021 14:59:34 +00:00 packages.conf

Step 4. Set the boot variable to correct .conf file and reboot the switch.

Switch(config)#boot system flash:packages.conf
Switch#copy run start
Switch#reload

Switch in ROMMON Mode After Upgrade/Reload in Install Mode 

Step 1. Console into the switch.

Step 2. Check the flash for the .conf file.

Switch:dir flash:

Step 3. Identify the correct .conf file.

Switch:more flash:packages.conf

Step 4. Boot the switch with correct .conf file

Switch:boot flash:packages.conf

Tip: In case you are unable to find the .conf file that corresponds to the correct software version boot up the switch with .conf file that corresponds to the previous software version. 

In the scenario where the device gets stuck in ROMMON with no valid image, you have the option to copy the file via USB onto the switch or via TFTP:

• OPTION #1: you can boot right from a USB stick with the this command:

boot usbflash0:cat9k_iosxe.16.12.01.SPA.bin

Tip: The USB Type A port supports USB flash drives with capacities from 128 MB to 8 GB (USB devices with port densities of 128 MB, 256 MB, 1 GB, 4 GB, and 8 GB are supported) formatted with the FAT file system

• OPTION #2:copy from a TFTP server is more elaborate. You either need to setup your laptop as a TFTP server and connect an ethernet cable to the management port of the device, or you need to connect the management port of the device to a network port of a device that is operative. You then configure the device with your image as a TFTP server.

tftp-server flash:cat9k_iosxe.16.12.01.SPA.bin

• Use a Router/Switch as the TFTP server is preferable so you do not have to worry about windows firewalls or computer account privilege issues. Once you have the TFTP server setup, you need to configure this in ROMMON.

DEFAULT_GATEWAY=172.16.0.1
IP_ADDRESS=172.16.0.10
IP_SUBNET_MASK=255.255.255.0
TFTP_SERVER=172.16.0.100

• Test your configuration by ping to the TFTP server.

ping 172.16.0.100

• Once the connectivity is established, boot right to the file on TFTP.

boot tftp://172.16.0.100/cat9k_iosxe.16.12.01.SPA.bin

Convert from Bundle to Install Mode

Perform the next procedure to convert the mode of operation from Bundle mode to Install mode:

This example shows of how to use the legacy request platform software package expand command:

Switch#request platform software package expand file flash:image_name.bin

Note: In the manual extraction process, the conf file created could be named as bin image file_name.conf. Identify the correct conf file and set the boot variable.

Convert from Install to Bundle Mode

Corrupt Bin File

To avoid this, verify the md5 checksum value before you perform the software install operation. In case the md5 checksum value does not match, copy the bin file again to the flash memory.

Switch#verify /md5 flash:<file name>.bin

Insufficient Space in Flash

To avoid this, perform a software clean operation before software install. The command syntax for a software clean is dependent on the version of code the switch is currently on.

For 16.6.3 and earlier:

Switch#request platform software package clean

For 16.6.4 and later:

Switch#install remove inactive

Stack Failure Scenarios (V-Mismatch)

If you get a V-mismatch, this means the software installed on the switch is different than the active as shown in this command output:

switch#show switch 
Switch/Stack Mac Address : 08ec.f5e1.8f80 - Local Mac Address
Mac persistency wait time: Indefinite
                                              H/W   Current
Switch#    Role    Mac Address     Priority Version  State 
-------------------------------------------------------------------------------------
*1        Active   08ec.f5e1.8f80     10     V01     Ready 
 2        Member   701f.5300.fa00     15     V01     V-Mismatch   <--- Indicates a version mismatch 

If you are in install mode, you have the option to resolve this with auto-upgrade command. The purpose of the auto-upgrade feature is to allow a switch to be upgraded to a compatible software image, so that the switch can join the switch stack. When a new switch attempts to join a switch stack, the active switch performs the compatibility check. Each stack member sends the results of the compatibility checks to the active switch, which uses the results to determine whether the switch can join the switch stack. If the software on the new switch is incompatible with the switch stack, the new switch enters version-mismatch (VM) mode. If the auto-upgrade feature is enabled on the current switch stack, the active switch automatically upgrades the new switch with the same software image installed on a compatible stack member. Auto-upgrade starts a few minutes after the mismatched software is detected.

You can enable auto-upgrade with the  software auto-upgrade enable   global configuration command on the new switch.

Note: Auto-upgrade is disabled by default. The auto-upgrade feature is not available in bundle mode. The switch stack must operate in install mode.

If you are in bundle mode, please do this procedure:

Step 1. Remove the mismatched switch from the stack. 

Step 2. Upgrade it to the correct version.

Step 4. Set the boot variable to the correct .bin file and save the configuration.

Step 5. Power off the switch, connect the stack cables and power on the switch.

Switch Boots Into ROMMON Due to Stack 1+1 Variable

You can identify this condition if the device displays these logs: :

switch:boot
attempting to boot from [flash:packages.conf]
Located file packages.conf
##########################################################################################################################
Validate packages: SHA-1 hash:
calculated 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA
expected 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA
Both links down, not waiting for other switches
Switch number is 2
Chassis 2 reloading, reason - Active/standby selection failed in 1+1 Mode                  <<<<<<
Feb 19 20:06:55.572 FP0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested 
Feb 19 20:06:57

Initializing Hardware...

Recovery:

Step 1. Unset the Stack 1+1 variable. 

Switch:unset STACK_1_1

Step 2. Once Stack 1+1 bootloader variable is unset, issue the   boot  command so that switch can boot from the boot variable set on the switch.

switch:boot 
##########################################################################################################################
Validate packages: SHA-1 hash:
calculated 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA
expected 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA

Both links down, not waiting for other switches
Switch number is 

Failure to Copy/Extract Bin Image to One of the Member Switches at the Time of the Upgrade

Sometimes errors can occur when images are copied to a member switch(es):

Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[2]: % Failed to copy file flash:cat9k_iosxe.17.03.03.SPA.bin from active switch 1 to switch 2, 
operation aborted 

Recovery:

Step 1. Check the flash on the member on which copy fails. Verify if it has sufficient space available or if the flash is corrupted.

Step 2. If it does not have sufficient space then delete files to ensure sufficient space is available.  

Step 3. If space is available and still it does not copy then format the flash drive of the member switch.     

Switch#format flash-1:       

Step 4. Once sufficient space is available or flash is recovered, perform the upgrade  

Step 5. In case even after a format the same error is observed for the same switch then troubleshoot for flash issues on the switch.

Note: For further release-specific instructions, please refer to release notes of the version to which you plan to upgrade.

Restrictions for Install Mode Versus Bundle Mode

The Install mode and the Bundle mode of operation have the next restrictions:

• Boot from a USB driver or TFTP in Install mode is not supported.

• Boot the switch in Bundle mode consumes more memory than the Install mode because the packages are extracted from the bundle and copied to the RAM.

• Auto-upgrade feature is disabled by default and is not available in the Bundle mode.

Legacy Request Platform Commands

This section explains about use of the  request platform  commands and is supported on the 9500 and 9300.

• This method is NOT recommended for Cisco Catalyst 9000 switches. Please use one of the methods mentioned previously.
• These commands are deprecated as of 16.10.1 and it is recommended that you use the install commands.
• The three most important commands are listed here (There are many more options but it is preferred to use the  install   commands instead of these). 

Expand

• This command takes whatever .bin file you specify and extract the .pkg files.

Switch#request platform software package expand switch all file flash:cat9k_iosxe.16.09.02.SPA.bin

• If the device operates in bundle mode. First run the expand command then change your boot statement to packages.conf and reload to get to install mode

Install 

• If you are already in install mode, use this command to move to another version.

request platform software package install switch all file flash:test auto-copy new

Clean

• Does the same thing as "install remove inactive".

request platform software package clean

ROMMON Improvements

The next 3 ROMMON enhancements are available from Cisco IOS XE version 16.12.X

• show romvar  —  New CLI command to dump ROMMON variables (equivalent to the "set" command while in ROMMON)
• The ability to display files from ROMMON mode
• show bootlog —  ROMMON upgrade logs are captured and make them available from both Cisco IOS and ROMMON mode

Note:  By default the feature is enabled. To disable the feature set   ROMMON_BOOT_LOG_DISABLE=1 at the ROMMON prompt

C9200L#show romvar
----------------------------------
ROMMON variables for Active Switch
----------------------------------
AUTOBOOT_COUNT="0"
AUTOBOOT_STATE="0"
BAUD="9600"
BOOT="tftp://chinmoha/pol.bin;"
BOOT64="flash:quake.itb.ssa.181009"
BOOT_LOADER_UPGRADE_DISABLE="yes"
BOOT_PARAM="ip=172.16.0.230::172.16.0.1:255.255.0.0:vore:eth0:on DEBUG_CONF= REAL_MGMTE_DEV="
CFG_MODEL_NUM="C9200L-48T-4X-E"
CLEI_CODE_NUMBER="INM6H00ARA"
DEFAULT_GATEWAY="172.16.0.1"
DEFAULT_ROUTER="172.16.0.1"
ENABLE_BREAK="yes"
IP_ADDRESS="172.16.0.230"
IP_MASK="255.255.0.0"
IP_SUBNET_MASK="255.255.0.0"
LICENSE_BOOT_LEVEL="network-essentials,all:C9200L-48;"
MAC_ADDR="70:B3:17:2E:1C:80"
MANUAL_BOOT="yes"
MODEL_NUM="C9200L-48T-4X"
MODEL_REVISION_NUM="29"
MOTHERBOARD_ASSEMBLY_NUM="73-19242-03"
MOTHERBOARD_REVISION_NUM="04"
MOTHERBOARD_SERIAL_NUM="JAE22370MLD"
ROMMON_AUTOBOOT_ATTEMPT="3"
ROMMON_BOARDID="0x315"
ROMMON_BOARDREV="0x4"
ROMMON_BOOT_LOG="1"             ---> boot log is enabled (default)
STANDALONE="1"
SWITCH_NUMBER="1"
SYSTEM_SERIAL_NUM="JAE22370MLD"
TAN_NUM="68-101384-01"
TEMPLATE="advanced"
TFTP_SERVER="172.16.0.25"
USB_DB_INFO="73-18785-03A0JAE22410UH0"
VERSION_ID="PPC"
ABNORMAL_RESET_COUNT="0"
BSI="0"
RANDOM_NUM="966540990"

C9200L#show bootlog switch active r0
=========================================================

//// ROMMMON Boot up log start ////

System Bootstrap, Version 99.2, DEVELOPMENT SOFTWARE
Compiled Wed 12/12/2018 14:56:53 by chinmoha
Copyright (c) 2018 by Cisco Systems, Inc.

Current ROMMON image : Primary
C9200L-48T-4X platform with 2097152 Kbytes of main memory


switch: boot: attempting to boot from [tftp://chinmoha/pol_try.bin]
Filename     : /chinmoha/pol_try.bin
IpAddress    : 172.16.0.230
TftpServer   : 172.16.0.25
TftpBlkSize  : 1468
*
File Size    : 349275061

//// ROMMMON Boot up log end ////

==========================================================
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.155 (xelinux@xe-linux-bld1) (gcc version 5.3.0 (GCC) ) #1 SMP Thu Dec 13 00:46:18 PST 2018
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] Cisco Package: start=0xa2000000
[    0.000000] Cisco Package: size=0x12b02000
[    0.000000] Cisco Package at 0xffffffc022000000 (313532416 bytes)
[    0.000000] crashkernel reserved: 0x00000000fde00000 - 0x00000000ffe00000 (32 MB)
[    0.000000] cma: Reserved 128 MiB at 0x00000000f0000000
[    0.000000] On node 0 totalpages: 524229
[    0.000000]   DMA zone: 8192 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 524229 pages, LIFO batch:31
[    0.000000] PERCPU: Embedded 15 pages/cpu @ffffffc07ffa3000 s24472 r8192 d28776 u61440
[    0.000000] pcpu-alloc: s24472 r8192 d28776 u61440 alloc=15*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 516037
--snip--

ROMMON - Allow File Transfer to/from USB/TFTP While in ROMMON Mode

These commands are useful to transfer to/from USB/TFTP  while in ROMMON mode. 

Important: These commands are available from Cisco IOS XE 17.1.1

switch: copy ?
usage:
	copy <path-to-source-file> <path-to-destination-file>
switch: 
switch: dir
usage:
	dir <path-to-directory>
supported filesystem(s):
         flash: ro flash
     bootflash: ro system partition
     usbflash0: rw front-panel USB
     usbflash1: rw back-panel USB
switch: dir flash:

Size       Attributes Name
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
—snip—
616          -rw-     vlan.dat
4096         drw-     tech_support
835079148    -rw-     cat9k_iosxe.17.01.01.SPA.bin
910498192    -rw-     cat9k_iosxe.BLD_POLARIS_DEV_LATEST_20191212_030414_2.SSA.bin
9307         -rw-     cat9k_iosxe.BLD_POLARIS_DEV_LATEST_20180819_200912_2.SS.conf
—snip—
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch: copy flash:vlan.dat usbflash0:
Using Destination file path: usbflash0:vlan.dat
Reading :        616
Writing :        616
switch: 
switch: dir usbflash0:                               

Size       Attributes Name
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
—snip—
880878448    -rw-     cat9k_iosxe.sh_tech_cts
835079148    -rw-     cat9k_iosxe.17.01.01.SPA.bin
4096         -rw-     ._cat9k_iosxe.17.01.01.SPA.bin
910498192    -rw-     cat9k_iosxe.BLD_POLARIS_DEV_LATEST_20191212_030414_2.SSA.bin
616          -rw-     vlan.dat
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Related Information