Troubleshoot Rommon on Catalyst 9000 Series Switches

Introduction

This document describes how to boot Catalyst 9000 Series switches out of the bootloader prompt (rommon) and how to recover a password.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• Catalyst 9200
• Catalyst 9300
• Catalyst 9400
• Catalyst 9500
• Catalyst 9600

Note: Consult the appropriate configuration guide for the commands that are used in order to enable these features on other Cisco platforms.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

In some instances, a Catalyst 9000 switch boots in rommon, a bootloader prompt that becomes available either when the switch cannot load a full Cisco IOS® software image, or when you have manually interrupted the normal boot process to perform actions like password recovery.

The switch: prompt indicates that the device is in rommon / bootloader mode. The bootloader provides a limited set of actions to administer the device. To see the list of available actions, issue the ? command at the switch: prompt.

switch: ?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
alias               Set and display aliases command
boot                Load and boot an executable image
cat                 Concatenate (type) file(s)
copy                Copy a file
date                Show or Set system date
delete              delete file(s)
dir                 List files in directories
dns-lookup          Send DNS standard query packets
emergency-install   Initiate Disaster Recovery
help                Present list of available commands
history             Monitor command history
md5                 Compute MD5 checksum of a file
mkdir               Create directory(ies)
meminfo             Main memory information
net-show            Display current network configuration
ping                Send ICMP ECHO_REQUEST packets to a network host
rename              Rename a file/directory
reset               Reset the system
rmdir               delete directory(ies)
set                 Set or display environment variables
unalias             Unset an alias
unset               Unset one or more environment variables
version             Display boot loader version
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Options to Boot a Switch Out of the Bootloader

Three options exist:

1. Boot from files that are present in the flash file system.
2. Direct boot via TFTP or USB.
3. Emergency-install (this action erases the flash file system).

Option 1: Boot from Files Present in the Flash File System

This section describes steps to take when you have complete, bootable files already present in the flash file system of the switch which has booted to the bootloader / rommon prompt. If the switch does not have files stored internally, or if you are unsure about the state of those files, use Option 2 or Option 3 instead of this option.

Step 1. Run the dir flash: command.

Step 2. Identify one of two file types to boot the switch. The two types are:

• A bootable binary image file (denoted by a.binfile extension) for Catalyst 9000 Series Switches. This file can be seen as: cat9k_iosxe_<majorversion.minorversion.releasenumber>.SPA.bin

  Note: Catalyst 9200 Series switches do not use the same universal image as current 9300, 9400, 9500, and 9600 Series switches. A Catalyst 9200 Series switch binary image has a similar file name convention to: cat9k_lite_iosxe...SPA.bin

• A bootable configuration (.conf) file that specifies packages that were previously extracted from a relevant bundle file. To boot this file type, you must also have specific package (pkg) files in flash.

Note: This guide does not cover the operational differences between these two boot methods, known as bundle mode and install mode. In a switch stack, boot modes must match across stack members. For applicable platforms, a stackwise virtual pair must run in install mode. Run the command show version from the exec prompt to determine the current boot mode of an operational switch stack.

Example of a .bin file which can be used to boot the switch into bundle mode:

switch: dir flash:
<snip> 
cat9k_iosxe.16.12.04.SPA.bin

Example of a .conf file with packages in flash. (The next step describes how to verify that these are the correct packages.)

switch: dir flash:
<snip>
14464008 -rw- cat9k-cc_srdriver.16.12.04.SPA.pkg
89134084 -rw- cat9k-espbase.16.12.04.SPA.pkg
1684484 -rw- cat9k-guestshell.16.12.04.SPA.pkg
531063808 -rw- cat9k-rpbase.16.12.04.SPA.pkg
41799673 -rw- cat9k-rpboot.16.12.04.SPA.pkg
31478788 -rw- cat9k-sipbase.16.12.04.SPA.pkg
60392448 -rw- cat9k-sipspa.16.12.04.SPA.pkg
23217156 -rw- cat9k-srdriver.16.12.04.SPA.pkg
7560 -rw- packages.conf                       <-- .conf file calls a group of packages 
12907520 -rw- cat9k-webui.16.12.04.SPA.pkg
9216 -rw- cat9k-wlc.16.12.04.SPA.pkg

Step 2a. If you choose to boot a .conf file, you must have the correct associated packages in flash. A failed boot of this type can indicate package corruption or an incorrect "conf file for the packages in flash.

To verify if you have a conf file that matches the packages contained in flash, issue cat flash:<filename>.conf and replace <filename> with the relevant package configuration file name. In this case, the file is called packages.

Note: The default name for a package configuration .conf file is  packages.conf. Some upgrade procedures can result in different filenames.

Text output of packages.conf, which indicates what .pkg (package) files are needed in the flash file system for the switch to boot properly:

switch: cat flash:packages.conf

#! /usr/binos/bin/packages_conf.sh

sha1sum: fb7ea5ea75a0cbf14ce81cecf110e5a6d526df86

# sha1sum above - used to verify that this file is not corrupted.

#
# package.conf: provisioned software file for build 2020-07-09_21.53
#

# NOTE: Editing this file by hand is not recommended. 

<snip>
# This is for CAT9k
boot rp 0 0 rp_boot cat9k-rpboot.16.12.04.SPA.pkg  <-- package files are for version 16.12.4
iso rp 0 0 rp_base cat9k-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_daemons cat9k-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_iosd cat9k-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_security cat9k-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_wlc cat9k-wlc.16.12.04.SPA.pkg
iso rp 0 0 rp_webui cat9k-webui.16.12.04.SPA.pkg
iso rp 0 0 srdriver cat9k-srdriver.16.12.04.SPA.pkg
iso rp 0 0 guestshell cat9k-guestshell.16.12.04.SPA.pkg

<snip>

Step 3. Issue either the boot flash:<filename>.bin command or the boot flash:<filename>.conf command with the proper <filename>.

switch: boot flash:packages.conf
boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf
##################################################################...<snip>

Option 2: Direct Boot from USB / TFTP

This section describes the procedure to boot a Catalyst 9000 Series Switch from the bootloader / rommon prompt with a USB flash drive or TFTP server. In this method, the switch can only be booted into bundle mode. The switch does not copy the files you boot to the flash file system, nor is the option available. The switch administrator must copy the relevant files to flash after the switch is booted. If you need install mode, either convert the switch after successful boot in bundle mode, or use the steps described in Option 3.

Step 1. Download Catalyst 9000 Series Switch software from cisco.com (example: cat9k_iosxe.16.12.4.bin). Make note of the provided Message Digest 5 (MD5) hash for later use.

Step 2. Transfer the downloaded image to a USB flash drive or TFTP server.

Step 3a. (USB only) Plug the USB into the switch. Run the command dir usbflash0: and confirm that you see the correct file.

switch: dir usbflash0:

Size           Attributes  Name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
805827585        -rw-      cat9k_iosxe.16.12.04.SPA.bin
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Step 3b. (TFTP only) To use TFTP, you must set variables so that the switch can communicate on the local IP network which connects to the TFTP server.

Use set to set an address:

switch: set IP_ADDRESS 192.168.1.2

Use set to set a subnet mask:

switch: set IP_SUBNET_MASK 255.255.255.0

Use set to set a default gateway:

switch: set DEFAULT_GATEWAY 192.168.1.1

Use ping to test connectivity to the TFTP server:

switch: ping 192.168.1.10
Pinging 192.168.1.10, 4 time(s), with packet-size 16
service type : 0
total length : 9216 bytes
identification : 56580
fragmentation : 0
time to live : 254
protocol : 1
source : 192.168.1.2
destination : 192.168.1.10
<snip> 

Step 4. Use boot to boot the image from TFTP orusbflash0:

switch: boot tftp://192.168.1.10/cat9k_iosxe.16.12.04.SPA.bin
boot: attempting to boot from [tftp://192.168.1.10/cat9k_iosxe.16.12.04.SPA.bin]
h/w (environment):
 mac       : aa:bb:cc:dd:ee:ff
n/w (environment):
 ip        : 192.168.1.2
 mask      : 255.255.255.0
 gateway   : 192.168.1.1
h/w:
 interface : eth0 (Ethernet)
 mac       : aa:bb:cc:dd:ee:ff
n/w (ip v4):
 ip        : 192.168.1.2
 mask      : 255.255.255.0
 route(s)  : 0.0.0.0 -> 192.168.1.0/255.255.255.0
n/w (ip v6):
 ip(s)     : FE80::1234:5678:9123:4567/64
           : 2001:111:2222:333:4444:5555:6666:7777/64
 route(s)  : :: -> 2001:111:2222:333::/64
           : :: -> FE80::/64
           : FE80::999:8888:7777:6666 -> ::/
tftp v4:
 server    : 192.168.1.10
 file      : cat9k_iosxe.16.12.04.SPA.bin
 blocksize : 1460
!!!!!!!!!!!!!!!!!!!!!!
<snip>

Option 3: Copy and Expand with Emergency-Install

The emergency-install procedure allows you to boot the switch in install mode when the desired packages are not contained in the flash file system, or when you do not want to boot in bundle mode first, and then convert to install mode. With the emergency-install method, the switch boots with the proper package files, package configuration file, and boot variable on first boot.

Caution: These steps completely erase the flash file system. Any previous configuration or saved files are erased.

Note: Catalyst 9200 Series Switches do not support emergency-install.

Step 1. Have a binary image (.bin) file accessible via TFTP or USB. For more information, review steps 1 to 3 of Option 2: Direct Boot from USB / TFTP in this guide.

Step 2. Verify that a recovery file is available on the flash recovery partition with thedir sda9: command.

The recovery file works in conjunction with the software file that you download from cisco.com to perform the copy-and-expand process.

switch: dir sda9:

Size Attributes Name
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
21656489 -rw- cat9k-recovery.SSA.bin  <-- Recovery Image
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Step 3. Initiate the emergency install procedure with the emergency-install <filepath> command. This command formats the flash file system and expands the image into the component packages. Allow some time for the process to complete.

switch: emergency-install tftp://192.168.1.10/cat9k_iosxe.16.12.04.SPA.bin
WARNING: The system partition (bootflash:) can be erased during the system recovery install process.
Are you sure you want to proceed? [y] y/n [n]: y
Starting system recovery (tftp://192.168.1.10/cat9k_iosxe.16.12.04.SPA.bin) ...
boot: attempting to boot from [sda9:cat9k-recovery.SSA.bin]
boot: reading file cat9k-recovery.SSA.bin
############################

<snip>
Downloading bundle tftp://192.168.1.10/cat9k_iosxe.16.12.04.SPA.bin...
curl_vrf=2
% Total    % Received % Xferd Average  Speed   Time    Time     Time     Current
                              Dload    Upload  Total   Spent    Left     Speed
100  768M  100 768M   0     0 5522k         0  0:02:22 0:02:22  --:--:-- 7646k
<snip>

Preparing flash....
Flash filesystem unmounted successfully /dev/sda3
Syncing device....
Emergency Install successful... Rebooting
can reboot now

Step 4. The switch can return to the bootloader / rommon (switch:) prompt. Run the boot flash:packages.conf command .

switch: boot flash:packages.conf
boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf
#
#############################

Recover a Password-Protected Switch

To recover a password-protected switch, you can ignore the startup configuration. Adjust a variable in the bootloader prompt to bypass the startup-config, which contains a password.

Note: On Catalyst 9400 Switches with High Availability (Two Supervisors), when performing password recovery, you must remove the secondary supervisor before powering on. Otherwise, the primary can load the existing configuration from the secondary supervisor. After the password has been configured as desired, you can insert the secondary supervisor, and it can pull the current configuration from the primary supervisor.

At the bootloader (switch:) prompt, run the SWITCH_IGNORE_STARTUP_CFG=1 command.

switch: SWITCH_IGNORE_STARTUP_CFG=1

Use boot to boot the switch via a method described in the section titled Options to Boot a Switch Out of the Bootloader.

After the switch has booted, you can use the unconfigured switch to recover your startup configuration from the flash file system, via the copy:startup-config <filepath://> command. After you have the switch configured as desired, issue the no system ignore startupconfig switch all command and the write memory command from the exec prompt to allow the switch to load the startup configuration on future bootup.

Caution: If you do not issue no system ignore startupconfig switch all and write memory, the switch boots with no configuration on future reloads.

Bypass password recovery lockout mechanism

A switch can show an error message when you attempt to interrupt the boot process and access the bootloader.

The message indicates that password recovery is disabled.

The password-recovery mechanism has been triggered, but
is currently disabled. Access to the boot loader prompt
through the password-recovery mechanism is disallowed at
this point. However, if you agree to let the system be
reset back to the default system configuration, access
to the boot loader prompt can still be allowed.

Would you like to reset the system back to the default configuration (y/n)?

Response y resets the switch to default configuration and allows access to the bootloader / rommon prompt.

Response n boots the switch with its current boot statement and startup configuration.

Related Information

Technical Support and Documentation - Cisco Systems