Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Failure Scenarios

Switch Boots up on Previous Version After Reload in Install Mode

Switch in ROMMON Mode After Upgrade/Reload in Install Mode

Convert from Bundle to Install Mode

Convert from Install to Bundle Mode

Corrupt Bin File

Insufficient Space in Flash

Stack Failure Scenarios (V-Mismatch)

Switch Boots Into ROMMON Due to Stack 1+1 Variable

Failure to Copy/Extract Bin Image to One of the Member Switches at the Time of the Upgrade

Restrictions for Install Mode Versus Bundle Mode

Legacy Request Platform Commands

ROMMON Improvements

ROMMON - Allow File Transfer to/from USB/TFTP While in ROMMON Mode

Related Information

Introduction

This document describes the common failures that occur when Catalyst 9000 series devices are upgraded and provides steps to recovery.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- How to upgrade Catalyst 9000 series switches.

Components Used

This document is based on this hardware version:

- Catalyst 9000 series switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Catalyst 9000 switches utilize Cisco IOS® XE software and operate in two modes, Install mode and Bundle mode.

Install mode is the newer and recommended mode to run. Install mode uses a package-provision file named packages.conf in order to boot the switch. In addition, there are a number of .pkg files that accompany packages.conf files in the flash drive. It is not recommended to alter the packages.conf and .pkg files.

Bundle mode uses the monolithic Cisco IOS image (.bin file) to boot the switch. The Bundle mode consumes more memory than the Install mode because the packages are extracted from the bundle and copied to RAM. You can run the Catalyst 9000 switches in either modes.

This is a partial sample output of some commands to verify the mode of operation. In INSTALL Mode:

Switch#show version | be Mode

Switch Ports Model                 SW Version        SW Image        Mode

------ ----- -----                 ----------        ----------      ----

* 2    60    C9500-12Q             16.12.4           CAT9K_IOSXE     INSTALL <-------

Switch#show run | inc boot system

boot system bootflash:packages.conf <-------

Switch#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:packages.conf;    <-------

--snip--

This is a partial sample output of some commands to verify the mode of operation. In BUNDLE Mode:

Switch#show version | be Mode

Switch Ports Model                  SW Version        SW Image          Mode

------ ----- -----                  ----------        ----------        ----

*    2 60    C9500-12Q              16.12.4           CAT9K_IOSXE       BUNDLE  <-------

Switch#show run | inc boot system

boot system bootflash:cat9k_iosxe.16.12.04.SPA.bin <------- BUNDLE mode .bin image

Switch#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = bootflash:cat9k_iosxe.16.12.04.SPA.bin; <------- switch set to boot from .bin image

--snip--

Failure Scenarios

Switch Boots up on Previous Version After Reload in Install Mode

This can happen if the switch boot statement still points to a previous version instead of the new version. To verify this, you can do these procedure:

Step 1. Verify the boot variable with the command  show boot .

Switch#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:packages.conf;

Boot Variables on next reload:

BOOT variable = flash:packages.conf;

Manual Boot = no

Enable Break = yes

Boot Mode = DEVICE

iPXE Timeout = 0

Step 2. Verify what version the current .conf file belongs to with the command  more flash:packages.conf | include pkg .

Switch#more flash:packages.conf | in pkg

boot rp 0 0 rp_boot cat9k-rpboot.17.03.02a.SPA.pkg

iso rp 0 0 rp_base cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 0 0 rp_daemons cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 0 0 rp_iosd cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 0 0 rp_security cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 0 0 rp_wlc cat9k-wlc.17.03.02a.SPA.pkg

iso rp 0 0 rp_webui cat9k-webui.17.03.02a.SPA.pkg

iso rp 0 0 srdriver cat9k-srdriver.17.03.02a.SPA.pkg

iso rp 0 0 guestshell cat9k-guestshell.17.03.02a.SPA.pkg

boot rp 1 0 rp_boot cat9k-rpboot.17.03.02a.SPA.pkg

iso rp 1 0 rp_base cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 1 0 rp_daemons cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 1 0 rp_iosd cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 1 0 rp_security cat9k-rpbase.17.03.02a.SPA.pkg

iso rp 1 0 rp_wlc cat9k-wlc.17.03.02a.SPA.pkg

iso rp 1 0 rp_webui cat9k-webui.17.03.02a.SPA.pkg

iso rp 1 0 srdriver cat9k-srdriver.17.03.02a.SPA.pkg

iso rp 1 0 guestshell cat9k-guestshell.17.03.02a.SPA.pkg

Step 3. If the file points to a previous version, check flash for correct .conf file with the commands dir flash: | include conf   and  more flash:<filename>.conf | include pkg .

Switch#dir flash: | in conf

81132 -rw- 7773 Jan 5 2021 14:59:34 +00:00 packages.conf

Step 4. Set the boot variable to correct .conf file and reboot the switch.

Switch(config)#boot system flash:packages.conf

Switch#copy run start

Switch#reload

Switch in ROMMON Mode After Upgrade/Reload in Install Mode

Step 1. Console into the switch.

Step 2. Check the flash for the .conf file.

Switch:dir flash:

Step 3. Identify the correct .conf file.

Switch:more flash:packages.conf

Step 4. Boot the switch with correct .conf file

Switch:boot flash:packages.conf

Tip: In case you are unable to find the .conf file that corresponds to the correct software version boot up the switch with .conf file that corresponds to the previous software version.

In the scenario where the device gets stuck in ROMMON with no valid image, you have the option to copy the file via USB onto the switch or via TFTP:

- OPTION #1: you can boot right from a USB stick with the this command:

boot usbflash0:cat9k_iosxe.16.12.01.SPA.bin

Tip: The USB Type A port supports USB flash drives with capacities from 128 MB to 8 GB (USB devices with port densities of 128 MB, 256 MB, 1 GB, 4 GB, and 8 GB are supported) formatted with the FAT file system

- OPTION #2:copy from a TFTP server is more elaborate. You either need to setup your laptop as a TFTP server and connect an ethernet cable to the management port of the device, or you need to connect the management port of the device to a network port of a device that is operative. You then configure the device with your image as a TFTP server.

tftp-server flash:cat9k_iosxe.16.12.01.SPA.bin

- Use a Router/Switch as the TFTP server is preferable so you do not have to worry about windows firewalls or computer account privilege issues. Once you have the TFTP server setup, you need to configure this in ROMMON.

DEFAULT_GATEWAY=172.16.0.1

IP_ADDRESS=172.16.0.10

IP_SUBNET_MASK=255.255.255.0

TFTP_SERVER=172.16.0.100

- Test your configuration by ping to the TFTP server.

ping 172.16.0.100

- Once the connectivity is established, boot right to the file on TFTP.

boot tftp://172.16.0.100/cat9k_iosxe.16.12.01.SPA.bin

Convert from Bundle to Install Mode

Perform the next procedure to convert the mode of operation from Bundle mode to Install mode:

Procedure

Command or Action

Purpose

Step 1

enable

Example:

switch#enable

Enables privileged EXEC mode.

- Enter your password if prompted.

Step 2

show version

Example:

switch#show version

Verifies the mode of operation

Step 3

show boot

Example:

switch#show boot

Displays the contents of the BOOT variable, the name of the configuration file pointed to by the CONFIG_FILE variable, the contents of the BOOTLDR variable, and the configuration register

Step 4

no boot system

Example:

switch(config)#no boot system

Removes the startup system image specification.

Note: Ensure that the .bin file of the Cisco IOS version to be used is loaded before you apply the no boot system command.

Step 5

boot system switch all flash

Example:

switch(config)#boot system switch all flash:packages.conf

Updates the boot variable on all the switches in the stack.

Step 6

exit

Example:

switch(config)#exit

Exits the configuration mode to the next highest mode in the hierarchy.

Step 7

write memory

Example:

switch#write memory

Copies the file from the source to the destination.

Step 8

install add file flash

Example:

switch#install add file flash:cat9k_iosxe.16.12.04.SPA.bin activate commit

or

switch#install add file flash:cat9k_iosxe.16.12.04.SPA.bin activate

switch#install commit

Expands individual Cisco IOS XE software packages and the provision file from a specified bundle to a specific destination directory.

Note: if you do not want to complete the upgrade in one step, it is possible to break it down into each individual step

Step 9

reload

Example:

This operation can require a reload of the system.  Do you want to proceed [y/n]

It asks you to confirm if you want to reload before the device reloads

Step 10

show version

Example:

switch#show version

Verifies the mode of operation

This example shows of how to use the legacy request platform software package expand command:

Switch#request platform software package expand file flash:image_name.bin

Note: In the manual extraction process, the conf file created could be named as bin image file_name.conf. Identify the correct conf file and set the boot variable.

Convert from Install to Bundle Mode

Perform the next procedure to convert the mode of operation from Install mode to Bundle mode:

Procedure

Command or Action

Purpose

Step 1

enable

Example:

switch#enable

Enables privileged EXEC mode.

- Enter your password if prompted.

Step 2

show version

Example:

switch#show version

Verifies the mode of operation

Step 3

show boot

Example:

switch#show boot

Displays the contents of the BOOT variable, the name of the configuration file pointed to by the CONFIG_FILE variable, the contents of the BOOTLDR variable, and the configuration register

Step 4

no boot system

Example:

switch(config)#no boot system

Removes the startup system image specification.

Note: Ensure that the .bin file of the Cisco IOS version to be used is loaded before you apply the no boot system command.

Step 5

boot system switch all flash

Example:

switch(config)#boot system switch all flash:cat9k_iosxe.16.12.04.SPA.bin

Updates the boot variable on all the switches in the stack.

Step 6

exit

Example:

switch(config)#exit

Exits the configuration mode to the next highest mode in the hierarchy.

Step 7

write memory

Example:

switch#write memory

Copies the file from the source to the destination.

Step 8

Step 9

reload

Example:

switch#reload

It asks you to confirm if you want to reload before the device reloads

Step 10

show version

Example:

switch#show version

Verifies the mode of operation

Corrupt Bin File

To avoid this, verify the md5 checksum value before you perform the software install operation. In case the md5 checksum value does not match, copy the bin file again to the flash memory.

Switch#verify /md5 flash:<file name>.bin

Insufficient Space in Flash

To avoid this, perform a software clean operation before software install. The command syntax for a software clean is dependent on the version of code the switch is currently on.

For 16.6.3 and earlier:

Switch#request platform software package clean

For 16.6.4 and later:

Switch#install remove inactive

Stack Failure Scenarios (V-Mismatch)

If you get a V-mismatch, this means the software installed on the switch is different than the active as shown in this command output:

switch#show switch

Switch/Stack Mac Address : 08ec.f5e1.8f80 - Local Mac Address

Mac persistency wait time: Indefinite

H/W   Current

Switch#    Role    Mac Address     Priority Version  State

-------------------------------------------------------------------------------------

*1        Active   08ec.f5e1.8f80     10     V01     Ready

2        Member   701f.5300.fa00     15     V01     V-Mismatch   <--- Indicates a version mismatch

If you are in install mode, you have the option to resolve this with auto-upgrade command. The purpose of the auto-upgrade feature is to allow a switch to be upgraded to a compatible software image, so that the switch can join the switch stack. When a new switch attempts to join a switch stack, the active switch performs the compatibility check. Each stack member sends the results of the compatibility checks to the active switch, which uses the results to determine whether the switch can join the switch stack. If the software on the new switch is incompatible with the switch stack, the new switch enters version-mismatch (VM) mode. If the auto-upgrade feature is enabled on the current switch stack, the active switch automatically upgrades the new switch with the same software image installed on a compatible stack member. Auto-upgrade starts a few minutes after the mismatched software is detected.

You can enable auto-upgrade with the  software auto-upgrade enable   global configuration command on the new switch.

Note: Auto-upgrade is disabled by default. The auto-upgrade feature is not available in bundle mode. The switch stack must operate in install mode.

If you are in bundle mode, please do this procedure:

Step 1. Remove the mismatched switch from the stack.

Step 2. Upgrade it to the correct version.

Step 4. Set the boot variable to the correct .bin file and save the configuration.

Step 5. Power off the switch, connect the stack cables and power on the switch.

Switch Boots Into ROMMON Due to Stack 1+1 Variable

You can identify this condition if the device displays these logs: :

switch:boot

attempting to boot from [flash:packages.conf]

Located file packages.conf

##########################################################################################################################

Validate packages: SHA-1 hash:

calculated 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA

expected 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA

Both links down, not waiting for other switches

Switch number is 2

Chassis 2 reloading, reason - Active/standby selection failed in 1+1 Mode                  <<<<<<

Feb 19 20:06:55.572 FP0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested

Feb 19 20:06:57

Initializing Hardware...

Recovery:

Step 1. Unset the Stack 1+1 variable.

Switch:unset STACK_1_1

Step 2. Once Stack 1+1 bootloader variable is unset, issue the   boot  command so that switch can boot from the boot variable set on the switch.

switch:boot

##########################################################################################################################

Validate packages: SHA-1 hash:

calculated 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA

expected 550C9730:667B2788:DD6F6B06:D0FFA819:01A315DA

Both links down, not waiting for other switches

Switch number is

Failure to Copy/Extract Bin Image to One of the Member Switches at the Time of the Upgrade

Sometimes errors can occur when images are copied to a member switch(es):

Preparing install operation ...

[1]: Copying software from active switch 1 to switch 2

[2]: % Failed to copy file flash:cat9k_iosxe.17.03.03.SPA.bin from active switch 1 to switch 2,

operation aborted

Recovery:

Step 1. Check the flash on the member on which copy fails. Verify if it has sufficient space available or if the flash is corrupted.

Step 2. If it does not have sufficient space then delete files to ensure sufficient space is available.

Step 3. If space is available and still it does not copy then format the flash drive of the member switch.

Switch#format flash-1:

Step 4. Once sufficient space is available or flash is recovered, perform the upgrade

Step 5. In case even after a format the same error is observed for the same switch then troubleshoot for flash issues on the switch.

Note: For further release-specific instructions, please refer to release notes of the version to which you plan to upgrade.

Restrictions for Install Mode Versus Bundle Mode

The Install mode and the Bundle mode of operation have the next restrictions:

- Boot from a USB driver or TFTP in Install mode is not supported.

- Boot the switch in Bundle mode consumes more memory than the Install mode because the packages are extracted from the bundle and copied to the RAM.

- Auto-upgrade feature is disabled by default and is not available in the Bundle mode.

Legacy Request Platform Commands

This section explains about use of the  request platform  commands and is supported on the 9500 and 9300.

- This method is NOT recommended for Cisco Catalyst 9000 switches. Please use one of the methods mentioned previously.

- These commands are deprecated as of 16.10.1 and it is recommended that you use the install commands.

- The three most important commands are listed here (There are many more options but it is preferred to use the  install   commands instead of these).

Expand

- This command takes whatever .bin file you specify and extract the .pkg files.

Switch#request platform software package expand switch all file flash:cat9k_iosxe.16.09.02.SPA.bin

- If the device operates in bundle mode. First run the expand command then change your boot statement to packages.conf and reload to get to install mode

Install

- If you are already in install mode, use this command to move to another version.

request platform software package install switch all file flash:test auto-copy new

Clean

- Does the same thing as "install remove inactive".

request platform software package clean

ROMMON Improvements

The next 3 ROMMON enhancements are available from Cisco IOS XE version 16.12.X

- show romvar  —  New CLI command to dump ROMMON variables (equivalent to the "set" command while in ROMMON)

- The ability to display files from ROMMON mode

- show bootlog —  ROMMON upgrade logs are captured and make them available from both Cisco IOS and ROMMON mode

Note:  By default the feature is enabled. To disable the feature set   ROMMON_BOOT_LOG_DISABLE=1 at the ROMMON prompt

C9200L#show romvar

----------------------------------

ROMMON variables for Active Switch

----------------------------------

AUTOBOOT_COUNT="0"

AUTOBOOT_STATE="0"

BAUD="9600"

BOOT="tftp://chinmoha/pol.bin;"

BOOT64="flash:quake.itb.ssa.181009"

BOOT_LOADER_UPGRADE_DISABLE="yes"

BOOT_PARAM="ip=172.16.0.230::172.16.0.1:255.255.0.0:vore:eth0:on DEBUG_CONF= REAL_MGMTE_DEV="

CFG_MODEL_NUM="C9200L-48T-4X-E"

CLEI_CODE_NUMBER="INM6H00ARA"

DEFAULT_GATEWAY="172.16.0.1"

DEFAULT_ROUTER="172.16.0.1"

ENABLE_BREAK="yes"

IP_ADDRESS="172.16.0.230"

IP_MASK="255.255.0.0"

IP_SUBNET_MASK="255.255.0.0"

LICENSE_BOOT_LEVEL="network-essentials,all:C9200L-48;"

MAC_ADDR="70:B3:17:2E:1C:80"

MANUAL_BOOT="yes"

MODEL_NUM="C9200L-48T-4X"

MODEL_REVISION_NUM="29"

MOTHERBOARD_ASSEMBLY_NUM="73-19242-03"

MOTHERBOARD_REVISION_NUM="04"

MOTHERBOARD_SERIAL_NUM="JAE22370MLD"

ROMMON_AUTOBOOT_ATTEMPT="3"

ROMMON_BOARDID="0x315"

ROMMON_BOARDREV="0x4"

ROMMON_BOOT_LOG="1"             ---> boot log is enabled (default)

STANDALONE="1"

SWITCH_NUMBER="1"

SYSTEM_SERIAL_NUM="JAE22370MLD"

TAN_NUM="68-101384-01"

TEMPLATE="advanced"

TFTP_SERVER="172.16.0.25"

USB_DB_INFO="73-18785-03A0JAE22410UH0"

VERSION_ID="PPC"

ABNORMAL_RESET_COUNT="0"

BSI="0"

RANDOM_NUM="966540990"

C9200L#show bootlog switch active r0

=========================================================

//// ROMMMON Boot up log start ////

System Bootstrap, Version 99.2, DEVELOPMENT SOFTWARE

Compiled Wed 12/12/2018 14:56:53 by chinmoha

Copyright (c) 2018 by Cisco Systems, Inc.

Current ROMMON image : Primary

C9200L-48T-4X platform with 2097152 Kbytes of main memory

switch: boot: attempting to boot from [tftp://chinmoha/pol_try.bin]

Filename     : /chinmoha/pol_try.bin

IpAddress    : 172.16.0.230

TftpServer   : 172.16.0.25

TftpBlkSize  : 1468

*

File Size    : 349275061

//// ROMMMON Boot up log end ////

==========================================================

[    0.000000] Booting Linux on physical CPU 0x0

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 4.4.155 (xelinux@xe-linux-bld1) (gcc version 5.3.0 (GCC) ) #1 SMP Thu Dec 13 00:46:18 PST 2018

[    0.000000] Boot CPU: AArch64 Processor [410fd034]

[    0.000000] Cisco Package: start=0xa2000000

[    0.000000] Cisco Package: size=0x12b02000

[    0.000000] Cisco Package at 0xffffffc022000000 (313532416 bytes)

[    0.000000] crashkernel reserved: 0x00000000fde00000 - 0x00000000ffe00000 (32 MB)

[    0.000000] cma: Reserved 128 MiB at 0x00000000f0000000

[    0.000000] On node 0 totalpages: 524229

[    0.000000]   DMA zone: 8192 pages used for memmap

[    0.000000]   DMA zone: 0 pages reserved

[    0.000000]   DMA zone: 524229 pages, LIFO batch:31

[    0.000000] PERCPU: Embedded 15 pages/cpu @ffffffc07ffa3000 s24472 r8192 d28776 u61440

[    0.000000] pcpu-alloc: s24472 r8192 d28776 u61440 alloc=15*4096

[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3

[    0.000000] Detected VIPT I-cache on CPU0

[    0.000000] CPU features: enabling workaround for ARM erratum 845719

[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 516037

--snip--

ROMMON - Allow File Transfer to/from USB/TFTP While in ROMMON Mode

These commands are useful to transfer to/from USB/TFTP  while in ROMMON mode.

Important: These commands are available from Cisco IOS XE 17.1.1

switch: copy ?

usage:

copy <path-to-source-file> <path-to-destination-file>

switch:

switch: dir

usage:

dir <path-to-directory>

supported filesystem(s):

flash: ro flash

bootflash: ro system partition

usbflash0: rw front-panel USB

usbflash1: rw back-panel USB

switch: dir flash:

Size       Attributes Name

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

—snip—

616          -rw-     vlan.dat

4096         drw-     tech_support

835079148    -rw-     cat9k_iosxe.17.01.01.SPA.bin

910498192    -rw-     cat9k_iosxe.BLD_POLARIS_DEV_LATEST_20191212_030414_2.SSA.bin

9307         -rw-     cat9k_iosxe.BLD_POLARIS_DEV_LATEST_20180819_200912_2.SS.conf

—snip—

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switch: copy flash:vlan.dat usbflash0:

Using Destination file path: usbflash0:vlan.dat

Reading :        616

Writing :        616

switch:

switch: dir usbflash0:

Size       Attributes Name

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

—snip—

880878448    -rw-     cat9k_iosxe.sh_tech_cts

835079148    -rw-     cat9k_iosxe.17.01.01.SPA.bin

4096         -rw-     ._cat9k_iosxe.17.01.01.SPA.bin

910498192    -rw-     cat9k_iosxe.BLD_POLARIS_DEV_LATEST_20191212_030414_2.SSA.bin

616          -rw-     vlan.dat

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Related Information

- Catalyst 9000 Switches Booting to Switch: Prompt Due to Stack 1+1 Variable

- Upgrade Guide for Cisco IOS XE Catalyst 9000 Switches

- In-Service Software Upgrade (ISSU) on Catalyst 3850, Catalyst 9000 series switches

- Install Versus Bundle Mode

- Cisco Technical Support & Downloads