Contents

Introduction

Prerequisites

Requirements

Components Used

Install Versus Bundle Mode

Install Mode

Bundle Mode

Verify the Mode

Upgrade

Flash Cleanup

Before Flash Cleanup

After Flash Cleanup

Auto-Upgrade Feature for Catalyst 3850 Series Switches

Configure

Enable Auto-Upgrade Feature

Verify

Recover from a 3850 Series Switch Boot Failure

Standard Recovery Methods

USB Recovery

Corrupt File Recovery

Emergency Recovery

Introduction

This document describes how to upgrade Cisco Catalyst 3850 Series Switches and provides recovery techniques for software or boot failures.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- TFTP

- FTP

- Experience with Cisco IOS® XE software upgrades

Components Used

The information in this document is based on the Cisco Catalyst 3850 Series Switch that runs Cisco IOS XE Versions 03.03.00 and later. The examples in this document use a stacked solution; however, the same commands can be run on a standalone switch.

Note: In order to download Cisco IOS XE images from the Cisco website, you must have a valid Cisco Connection Online (CCO) account with entitled credentials. Cisco does not offer a free TFTP/FTP solution. Install and configure the TFTP/FTP before you begin.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Install Versus Bundle Mode

The Cisco Catalyst 3850 Series Switch has two modes of operation: INSTALL and BUNDLE.

There are minor differences between the two modes. Review the configuration guide for more details.

Cisco recommends that the INSTALL mode is used during operation because it allows for a more complete set of features and requires fewer resources upon boot. This document provides a brief overview of each mode for reference.

Install Mode

This is the default mode for the switch. The INSTALL mode uses a package-provisioning file named packages.conf in order to boot the switch. In addition, there are a number of .pkg files in the flash. Cisco recommends that you do not alter these files unless directed by a Cisco Technical Assistance Center (TAC) engineer.

Bundle Mode

If you are comfortable with the use of traditional monolithic Cisco IOS images in order to boot the switch, then the BUNDLE mode is likely familiar.

The BUNDLE mode consumes more memory than the INSTALL mode because the packages are extracted from the Bundle and copied to the RAM.

Verify the Mode

In order to verify the mode, enter the show version command:

3850-stack#show version

Cisco IOS Software, Cisco IOS-XE Software, Catalyst L3 Switch Software

(CAT3K_CAA-UNIVERSALK9-M), Version 03.03.00SE RELEASE SOFTWARE (fc1)

Switch Ports Model           SW Version     SW Image              Mode

------------------           ----------     ----------            ----

1 32    WS-C3850-24P    03.03.00SE     cat3k_caa-universalk9 INSTALL

*    2 56    WS-C3850-48T    03.03.00SE     cat3k_caa-universalk9 INSTALL

Upgrade

In order to begin the upgrade process, download the Cisco IOS® XE .bin file from the Cisco web site and place it in the flash of your active switch. The process that is used in order to copy the file to the switch is not covered in this document.

Note: It is important to note that this process shows the commands needed to upgrade a 3850 switch running older 3.x.x versions. To upgrade 3850 switches running 16.x.x versions, use the command request platform software package install switch all file <file> or install add file <file> activate commit. Please refer to the Release Notes for Cisco Catalyst 3850 Series Switches for additional details.

When you copy the .bin file to a single switch, the install process replicates the file to the other switches in the stack. Once the file is present, enter this command:

3850-stack#software install file flash:cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin switch 1-2

Note: There are many options available after each command; however, in this example a basic upgrade is run.

When the upgrade process begins, the switch pushes the .bin file to the stack member peers.

Preparing install operation ...

[2]: Copying software from active switch 1 to switch 2

After all of the members receive the .bin file, it is automatically expanded to the flash.

[1 2]: Starting install operation

[1 2]: Expanding bundle flash:

cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin

[1 2]: Copying package files

[1 2]: Package files copied

[1 2]: Finished expanding bundle flash:

cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin

[1 2]: Verifying and copying expanded package files to flash:

[1 2]: Verified and copied expanded package files to flash:

[1 2]: Starting compatibility checks

[1 2]: Finished compatibility checks

[1 2]: Starting application pre-installation processing

[1 2]: Finished application pre-installation processing

Next, the switch lists a summary of the files that are marked for both removal and addition to the packages.conf pointer file.

[1]: Old files list:

Removed cat3k_caa-base.SPA.03.03.00SE.pkg

Removed cat3k_caa-drivers.SPA.03.03.00SE.pkg

Removed cat3k_caa-infra.SPA.03.03.00SE.pkg

Removed cat3k_caa-iosd-universalk9.SPA.150-1.EZ.pkg

Removed cat3k_caa-platform.SPA.03.03.00SE.pkg

Removed cat3k_caa-wcm.SPA.10.1.100.0.pkg

[2]: Old files list:

Removed cat3k_caa-base.SPA.03.03.00SE.pkg

Removed cat3k_caa-drivers.SPA.03.03.00SE.pkg

Removed cat3k_caa-infra.SPA.03.03.00SE.pkg

Removed cat3k_caa-iosd-universalk9.SPA.150-1.EZ.pkg

Removed cat3k_caa-platform.SPA.03.03.00SE.pkg

Removed cat3k_caa-wcm.SPA.10.1.100.0.pkg

[1]: New files list:

Added cat3k_caa-base.SPA.03.03.01SE.pkg

Added cat3k_caa-drivers.SPA.03.03.01SE.pkg

Added cat3k_caa-infra.SPA.03.03.01SE.pkg

Added cat3k_caa-iosd-universalk9.SPA.150-1.EZ1.pkg

Added cat3k_caa-platform.SPA.03.03.01SE.pkg

Added cat3k_caa-wcm.SPA.10.1.110.0.pkg

[2]: New files list:

Added cat3k_caa-base.SPA.03.03.01SE.pkg

Added cat3k_caa-drivers.SPA.03.03.01SE.pkg

Added cat3k_caa-infra.SPA.03.03.01SE.pkg

Added cat3k_caa-iosd-universalk9.SPA.150-1.EZ1.pkg

Added cat3k_caa-platform.SPA.03.03.01SE.pkg

Added cat3k_caa-wcm.SPA.10.1.110.0.pkg

Lastly, the packges.conf file is updated and committed.

[1 2]: Creating pending provisioning file

[1 2]: Finished installing software.  New software will load on reboot.

[1 2]: Committing provisioning file

[1 2]: Do you want to proceed with reload? [yes/no]: yes

Verify that the update process is properly completed upon reload.

3850-stack#show ver | i INSTALL

1 32    WS-C3850-24P     03.03.01SE      cat3k_caa-universalk9 INSTALL

*    2 56    WS-C3850-48T     03.03.01SE      cat3k_caa-universalk9 INSTALL

Flash Cleanup

Residual files remain in the flash from previous versions. To clean up the residual files, enter the software clean command instead of a manual deletion of the files. This purges the files that the switch no longer needs.

Note: This command also deletes the .bin file that is used in order to install the new Cisco IOS software. It is important to remember that once it is extracted, you no longer need it.

The next two sections provide examples of how the flash appears before and after the software clean command is used.

Before Flash Cleanup

3850-stack#show flash

-#- --length-- ---------date/time----------- -----------------------path-----------------------

2    2097152 Feb 16 2014 11:38:46.0 +00:00 nvram_config

4  257016048 Jan 28 2014 17:22:12.0 +00:00 cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

5       4096 Jan 28 2014 17:25:50.0 +00:00 mnt

6       4096 Jan 28 2014 17:25:50.0 +00:00 mnt/images

7       4096 Jan 28 2014 17:25:52.0 +00:00 mnt/images/ap.bak

8         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap1g2.md5

9   11591680 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap1g2

10         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g1.md5

11   10444800 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g1

12         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g2.md5

13   13568000 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g2

14         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/c1140.md5

15   10291200 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/c1140

16         11 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/version.info

17       1214 Jan 28 2014 17:25:10.0 +00:00 packages.conf.00-

18   79112096 Jan 28 2014 17:25:06.0 +00:00 cat3k_caa-base.SPA.03.03.00SE.pkg

19    6474428 Jan 28 2014 17:25:06.0 +00:00 cat3k_caa-drivers.SPA.03.03.00SE.pkg

20   34501468 Jan 28 2014 17:25:06.0 +00:00 cat3k_caa-infra.SPA.03.03.00SE.pkg

21       1248 Feb 16 2014 11:27:51.0 +00:00 packages.conf

22   34763952 Jan 28 2014 17:25:06.0 +00:00 cat3k_caa-iosd-universalk9.SPA.150-1.EZ.pkg

23        796 Feb 19 2014 11:43:13.0 +00:00 vlan.dat

24   24992476 Jan 28 2014 17:25:06.0 +00:00 cat3k_caa-platform.SPA.03.03.00SE.pkg

25   77167308 Jan 28 2014 17:25:06.0 +00:00 cat3k_caa-wcm.SPA.10.1.100.0.pkg

26       1224 Jan 28 2014 16:39:58.0 +00:00 packages.conf.01-

27       6571 Dec 20 2013 08:56:32.0 +00:00 BLANK_CONFIG.cfg

28  257193048 Feb 16 2014 11:19:44.0 +00:00 cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin

30   79113792 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-base.SPA.03.03.01SE.pkg

31   74409080 Jan 28 2014 16:39:54.0 +00:00 cat3k_caa-base.SPA.03.02.01.SE.pkg

32    2775728 Jan 28 2014 16:39:54.0 +00:00 cat3k_caa-drivers.SPA.03.02.01.SE.pkg

33    6476476 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-drivers.SPA.03.03.01SE.pkg

34   32478052 Jan 28 2014 16:39:54.0 +00:00 cat3k_caa-infra.SPA.03.02.01.SE.pkg

35   30389028 Jan 28 2014 16:39:54.0 +00:00 cat3k_caa-iosd-universalk9.SPA.150-1.EX1.pkg

36   18313952 Jan 28 2014 16:39:54.0 +00:00 cat3k_caa-platform.SPA.03.02.01.SE.pkg

37   63402700 Jan 28 2014 16:39:54.0 +00:00 cat3k_caa-wcm.SPA.10.0.101.0.pkg

38   34503664 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-infra.SPA.03.03.01SE.pkg

39   34788684 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-iosd-universalk9.SPA.150-1.EZ1.pkg

40   25009040 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-platform.SPA.03.03.01SE.pkg

41   77296448 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-wcm.SPA.10.1.110.0.pkg

237428736 bytes available (1302147072 bytes used)

After Flash Cleanup

3850-stack#software clean

Preparing clean operation ...

[1 2]: Cleaning up unnecessary package files

[1 2]: No path specified, will use booted path flash:packages.conf

[1 2]: Cleaning flash:

[1]: Preparing packages list to delete ...

In use files, will not delete:

cat3k_caa-base.SPA.03.03.01SE.pkg

cat3k_caa-drivers.SPA.03.03.01SE.pkg

cat3k_caa-infra.SPA.03.03.01SE.pkg

cat3k_caa-iosd-universalk9.SPA.150-1.EZ1.pkg

cat3k_caa-platform.SPA.03.03.01SE.pkg

cat3k_caa-wcm.SPA.10.1.110.0.pkg

packages.conf

[2]: Preparing packages list to delete ...

In use files, will not delete:

cat3k_caa-base.SPA.03.03.01SE.pkg

cat3k_caa-drivers.SPA.03.03.01SE.pkg

cat3k_caa-infra.SPA.03.03.01SE.pkg

cat3k_caa-iosd-universalk9.SPA.150-1.EZ1.pkg

cat3k_caa-platform.SPA.03.03.01SE.pkg

cat3k_caa-wcm.SPA.10.1.110.0.pkg

packages.conf

[1]: Files that will be deleted:

cat3k_caa-base.SPA.03.02.01.SE.pkg

cat3k_caa-base.SPA.03.03.00SE.pkg

cat3k_caa-drivers.SPA.03.02.01.SE.pkg

cat3k_caa-drivers.SPA.03.03.00SE.pkg

cat3k_caa-infra.SPA.03.02.01.SE.pkg

cat3k_caa-infra.SPA.03.03.00SE.pkg

cat3k_caa-iosd-universalk9.SPA.150-1.EX1.pkg

cat3k_caa-iosd-universalk9.SPA.150-1.EZ.pkg

cat3k_caa-platform.SPA.03.02.01.SE.pkg

cat3k_caa-platform.SPA.03.03.00SE.pkg

cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin

cat3k_caa-wcm.SPA.10.0.101.0.pkg

cat3k_caa-wcm.SPA.10.1.100.0.pkg

packages.conf.00-

packages.conf.01-

[2]: Files that will be deleted:

cat3k_caa-base.SPA.03.02.01.SE.pkg

cat3k_caa-base.SPA.03.03.00SE.pkg

cat3k_caa-drivers.SPA.03.02.01.SE.pkg

cat3k_caa-drivers.SPA.03.03.00SE.pkg

cat3k_caa-infra.SPA.03.02.01.SE.pkg

cat3k_caa-infra.SPA.03.03.00SE.pkg

cat3k_caa-iosd-universalk9.SPA.150-1.EX1.pkg

cat3k_caa-iosd-universalk9.SPA.150-1.EZ.pkg

cat3k_caa-platform.SPA.03.02.01.SE.pkg

cat3k_caa-platform.SPA.03.03.00SE.pkg

cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin

cat3k_caa-wcm.SPA.10.0.101.0.pkg

cat3k_caa-wcm.SPA.10.1.100.0.pkg

packages.conf.00-

packages.conf.01-

[1 2]: Do you want to proceed with the deletion? [yes/no]: yes

[1 2]: Clean up completed

Here is the output from the show flashcommand after the flash cleanup:

3850-stack#show flash

-#- --length-- ---------date/time----------- -------------------path--------------------

2    2097152 Feb 16 2014 11:38:46.0 +00:00 nvram_config

4       4096 Jan 28 2014 17:25:50.0 +00:00 mnt

5       4096 Jan 28 2014 17:25:50.0 +00:00 mnt/images

6       4096 Jan 28 2014 17:25:52.0 +00:00 mnt/images/ap.bak

7         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap1g2.md5

8   11591680 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap1g2

9         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g1.md5

10   10444800 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g1

11         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g2.md5

12   13568000 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/ap3g2

13         40 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/c1140.md5

14   10291200 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/c1140

15         11 Oct 03 2013 05:02:21.0 +00:00 mnt/images/ap.bak/version.info

16       1248 Feb 16 2014 11:27:51.0 +00:00 packages.conf

17        796 Feb 19 2014 11:43:13.0 +00:00 vlan.dat

18       6571 Dec 20 2013 08:56:32.0 +00:00 BLANK_CONFIG.cfg

20   79113792 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-base.SPA.03.03.01SE.pkg

21    6476476 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-drivers.SPA.03.03.01SE.pkg

22   34503664 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-infra.SPA.03.03.01SE.pkg

23   34788684 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-iosd-universalk9.SPA.150-1.EZ1.pkg

24   25009040 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-platform.SPA.03.03.01SE.pkg

25   77296448 Feb 16 2014 11:27:46.0 +00:00 cat3k_caa-wcm.SPA.10.1.110.0.pkg

1231515648 bytes available (308060160 bytes used)

Auto-Upgrade Feature for Catalyst 3850 Series Switches

One scenario in which a new switch is introduced into a current stack of Catalyst 3850 Series Switches is when a new switch is purchased in order to expand the number of usable ports in the stack. In order to successfully add a new switch to a stack, you must ensure that the same software version is run on the new switch. Prior to Cisco IOS XE Version 3.3.1, the only way to ensure that the versions match is to stage the new switch prior to introduction into the stack.

The Catalyst 3850 Series Switches include a feature called Auto-Upgrade. The goal of this feature is to ensure that a newly added switch is automatically provisioned by the stack members with the correct Cisco IOS XE version.

Note: Auto-Upgrade is disabled by default and is not available in BUNDLE mode.

In order to use the Auto-Upgrade feature, add the software auto-upgrade enable command into the configuration of the current stack. This ensures that any newly added stack members are automatically upgraded.

Configure

Once the switch is stacked and booted, there is an indication that there is a mismatch in versions and the new member does not fully join the stack.

To watch the SYSLOG as the switch attempts to join, notice that the Auto-Advise feature alerts that the newly added switch runs a different software version and mode.

Note: For this example, the new switch runs Cisco IOS XE Version 3.2.2 in BUNDLE mode.

%STACKMGR-1-STACK_LINK_CHANGE: STANDBY: 1 stack-mgr:

Stack port 2 on switch 1 is up  (3850-Stack-1)

%STACKMGR-1-STACK_LINK_CHANGE: 2 stack-mgr:

Stack port 1 on switch 2 is up

%STACKMGR-6-SWITCH_ADDED: 2 stack-mgr:

Switch 3 has been added to the stack.

%STACKMGR-6-SWITCH_ADDED: STANDBY:1 stack-mgr:

Switch 3 has been added to the stack.  (3850-Stack-1)

%INSTALLER-6-AUTO_ADVISE_SW_INITIATED: 2 installer:

Auto advise initiated for switch 3

%INSTALLER-6-AUTO_ADVISE_SW: 2 installer:

Switch 3 running bundled software has been added

%INSTALLER-6-AUTO_ADVISE_SW: 2 installer:

to the stack that is running installed software.

%INSTALLER-6-AUTO_ADVISE_SW: 2 installer:

The 'software auto-upgrade' command can be used to

%INSTALLER-6-AUTO_ADVISE_SW: 2 installer:

convert switch 3 to the installed running mode by

%INSTALLER-6-AUTO_ADVISE_SW: 2 installer:

installing its running software.

Once the newly joined member is fully booted, a mismatch is detected:

3850-Stack#show switch

Switch/Stack Mac Address : 0c27.24cf.ab80 - Local Mac Address

Mac persistency wait time: Indefinite

H/W   Current

Switch#   Role    Mac Address     Priority Version  State

------------------------------------------------------------

*1       Active   0c27.24cf.ab80     14      B0      Ready

2       Standby  f41f.c238.a800     13      B0      Ready

3       Member   b4e9.b0d3.6600     12      0       V-Mismatch

Enable Auto-Upgrade Feature

In Global Configuration mode, enter the software auto-upgrade enable command. This enables the feature for any new switches that join the stack.

3850-Stack(config)#software auto-upgrade enable

3850-Stack(config)#end

Reload the newly added switch only; a full stack reload is not necessary. In this case, the newly added switch is switch 3, so the reload slot 3 command is entered.

Tip: The slot mentioned in these commands designates the switch in the stack (slot 1 = switch 1).

3850-Stack#reload slot 3

Proceed with reload? [confirm]

%STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr:

Received reload request for switch 3, reason Reload Slot Command

%STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr:

Stack port 2 on switch 1 is down

%STACKMGR-6-SWITCH_REMOVED: 1 stack-mgr:

Switch 3 has been removed from the stack.

%STACKMGR-1-STACK_LINK_CHANGE: STANDBY:

2 stack-mgr:  Stack port 1 on switch 2 is down  (3850-Stack-2)

Starting SWITCH-DELETE sequence, switch 3

SWITCH-DELETE sequence complete, switch 3

%STACKMGR-6-SWITCH_REMOVED: STANDBY:2 stack-mgr:

Switch 3 has been removed from the stack.  (3850-Stack-2)

Starting SWITCH-DELETE sequence, switch 3 (3850-Stack-2)

SWITCH-DELETE sequence complete, switch 3 (3850-Stack-2)

The switch reloads in the background momentarily. Then, you see this:

%STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr:

Stack port 2 on switch 1 is up

3850-Stack#

%STACKMGR-1-STACK_LINK_CHANGE: STANDBY:2 stack-mgr:

Stack port 1 on switch 2 is up  (3850-Stack-2)

3850-Stack#

%STACKMGR-6-SWITCH_ADDED: 1 stack-mgr:

Switch 3 has been added to the stack.

%STACKMGR-6-SWITCH_ADDED: STANDBY:2 stack-mgr:

Switch 3 has been added to the stack.  (3850-Stack-2)

The conversion from BUNDLE to INSTALL mode occurs, followed by a reload:

%INSTALLER-6-AUTO_UPGRADE_SW_INITIATED: 1 installer:

Auto upgrade initiated for switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Converting switch 3 to installed mode by

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

installing its running software

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Setting the boot var on switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Finished installing the running software on switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Reloading switch 3 to boot in installed mode

%STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr:

Received reload request for switch 3, reason Auto upgrade

%STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr:

Stack port 2 on switch 1 is down

%STACKMGR-6-SWITCH_REMOVED: 1 stack-mgr:

Switch 3 has been r

3850-Stack#emoved from the stack.

%STACKMGR-1-STACK_LINK_CHANGE: STANDBY:2 stack-mgr:

Stack port 1 on switch 2 is down  (3850-Stack-2)

Starting SWITCH-DELETE sequence, switch 3

SWITCH-DELETE sequence complete, switch 3

%STACKMGR-6-SWITCH_REMOVED: STANDBY:2 stack-mgr:

Switch 3 has been removed from the stack.  (3850-Stack-2)

3850-Stack#

Starting SWITCH-DELETE sequence, switch 3 (3850-Stack-2)

SWITCH-DELETE sequence complete, switch 3 (3850-Stack-2)

After the reboot, the upgrade continues:

%INSTALLER-6-AUTO_UPGRADE_SW_INITIATED: 1 installer:

Auto upgrade initiated for switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Searching stack for software to upgrade switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Found donor switch 1 to auto upgrade switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Upgrading switch 3 with software from switch 1

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Finished installing software on switch 3

%INSTALLER-6-AUTO_UPGRADE_SW: 1 installer:

Reloading switch 3 to complete the auto upgrade

%STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr:

Received reload request for switch 3, reason Auto upgrade

%STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr:

Stack port 2 on switch 1 is down

%STACKMGR-6-SWITCH_REMOVED: 1 stack-mgr:

Switch 3 has been removed from the stack.

%STACKMGR-1-STACK_LINK_CHANGE: STANDBY:2 stack-mgr:

Stack port

3850-Stack#t 1 on switch 2 is down  (3850-Stack-2)

Starting SWITCH-DELETE sequence, switch 3

SWITCH-DELETE sequence complete, switch 3

%STACKMGR-6-SWITCH_REMOVED: STANDBY:2 stack-mgr:

Switch 3 has been removed from the stack.  (3850-Stack-2)

Another reload is performed automatically. Once the switch boots up, it successfully joins the stack with the correct Cisco IOS XE version and software mode.

%STACKMGR-6-SWITCH_ADDED: 1 stack-mgr:

Switch 3 has been added to the stack.

%STACKMGR-6-SWITCH_ADDED: STANDBY:2 stack-mgr:

Switch 3 has been added to the stack.  (3850-Stack-2)

%STACKMGR-6-SWITCH_READY: STANDBY:2 stack-mgr:

Switch 3 is ready.  (3850-Stack-2)

%STACKMGR-6-SWITCH_READY: 1 stack-mgr:  Switch 3 is ready.

Starting SWITCH-ADD sequence, switch 3

%NGWC_USB_CONSOLE-6-CONFIG_ENABLE: Switch 3:

Console media-type changed to default

Starting SWITCH-ADD sequence, switch 3 (3850-Stack-2)

SWITCH-ADD sequence complete, switch 3 (3850-Stack-2)

SWITCH-ADD sequence complete, switch 3

Verify

Use the show switch and show version commands in order to verify that the upgrade process is completed properly:

3850-Stack#show switch

Switch/Stack Mac Address : 0c27.24cf.ab80 - Local Mac Address

Mac persistency wait time: Indefinite

H/W   Current

Switch#   Role    Mac Address     Priority Version  State

------------------------------------------------------------

*1       Active   0c27.24cf.ab80     14     B0      Ready

2       Standby  f41f.c238.a800     13     B0      Ready

3       Member   b4e9.b0d3.6600     12     B0      Ready

3850-Stack#show version

Switch Ports Model          SW Version    SW Image              Mode

------ ----- -----          ----------    ----------            ----

*    1 56    WS-C3850-48P   03.03.01SE    cat3k_caa-universalk9 INSTALL

2 56    WS-C3850-48P   03.03.01SE    cat3k_caa-universalk9 INSTALL

3 56    WS-C3850-48P   03.03.01SE    cat3k_caa-universalk9 INSTALL

Recover from a 3850 Series Switch Boot Failure

This section describes possible recovery methods for a 3850 Series Switch boot failure, such as a corrupt boot image, a corrupt packages.conf file, or missed files.

Note: Ensure that you have knowledge of the two possible boot modes, INSTALL and BUNDLE, before you continue.

Standard Recovery Methods

This section describes the two standard methods that are used in order to recover from a Catalyst 3850 Series Switch boot failure.

USB Recovery

The 3850 Series Switches have a USB port on the front that is used for console access. This USB port is also used with flash drives for image backup and recovery.

If stuck at the switch: prompt with a corrupt image or .conf file, boot to a file that is stored on the USB drive or copy an image from the USB to internal flash. Complete these steps in order to recover from the boot failure:

1. Verify that the flash drive is recognized and the .bin file exists:

switch: dir usbflash0:

Directory of usbflash0:/

74  -rw-  223734376  cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

2. Boot to the USB image:

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

Note: This process boots the switch into BUNDLE mode.

Tip: You can also copy the .bin file from usbflash0: to flash:, and point the boot statement towards internal flash.

Corrupt File Recovery

There are instances when the packages.conf calls files no longer exist in flash. You can manually boot an image from the switch: prompt file; however, upon reload it calls the packages.conf file again and fails to boot. If this occurs, Cisco recommends to back up the current packages.conf file and rename it or delete it. This process is mandatory, as the next step fails if a .conf file already exists.

Once the .bin file is extracted, a new packages.conf file is created. Complete these steps in order to recover from a corrupt packages.conf file:

1. Once booted (in BUNDLE mode), verify the files in the flash:

Switch#dir flash:

Directory of flash:/

15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf

2. Copy or rename the current packages.conf file:

Switch#cp flash:packages.conf flash:packages.conf.badop

Destination filename [packages.conf.bad]?

Copy in progress...C

1243 bytes copied in 0.140 secs (8879 bytes/sec)

Switch#dir flash:

Directory of flash:/

15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf

15502  -rw-        1243   Aug 1 2013 11:53:51 +00:00  packages.conf.bad

Switch#del flash:packages.conf

Delete filename [packages.conf]?

Delete flash:/packages.conf? [confirm]

3. Expand the bundle in order to create a new packages.conf file:

Switch#software expand running switch 1 to flash:

Preparing expand operation ...

[1]: Expanding the running bundle

[1]: Copying package files

[1]: Package files copied

[1]: Finished expanding the running bundle

4. Verify the boot:

Switch#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable does not exist

Boot Variables on next reload:

BOOT variable = flash:packages.conf;

Manual Boot = no

Enable Break = no

5. Reload the switch:

Switch#reload

Reload command is being issued on Active unit, this will reload the whole stack

Proceed with reload? [confirm]

Emergency Recovery

If the previous recovery methods fail, the 3850 Series Switches have a trap door method to use in order to recover the system. A terminal must be connected to the management port of the switch that runs a TFTP server. Download a valid image file from CCO and store it in the root of the TFTP server.

It is likely that the switch is stuck at the switch: prompt. However, if you are in a boot loop, use the Mode button on the front of the switch in order to break the cycle: hold the button for approximately ten seconds, and the switch breaks the cycle and stops at the switch: prompt.

Complete these steps in order to perform an emergency recovery:

1. Set the switch IP address:

switch:  set IP_ADDR 192.0.2.123/255.255.255.0

2. Set the default gateway:

switch: set DEFAULT_ROUTER 192.0.2.1

3. Ping the terminal that contains the TFTP server in order to test the connectivity:

switch: ping 192.0.2.1

ping 192.0.2.1 with 32 bytes of data ...

Host 192.0.2.1 is alive.

4. Verify that the emergency files exist in the switch file system:

switch: dir sda9:

Directory of sda9:/

2  drwx  1024       .

2  drwx  1024       ..

11  -rwx  18958824   cat3k_caa-recovery.bin

36903936 bytes available (20866048 bytes used)

5. Run the emergency install feature:

switch: emergency-install tftp://192.0.2.1/cat3k_caa-universalk9.

SPA.03.03.00.SE.150-1.EZ.bin

The bootflash will be erased during install operation, continue (y/n)?Y

Starting emergency recovery (tftp://192.0.2.1/cat3k_caa-universalk9.

SPA.03.02.02.SE.150-1.EX2.bin)...

Reading full image into memory......................done

Nova Bundle Image

--------------------------------------

Kernel Address    : 0x6042f5d8

Kernel Size       : 0x317ccc/3243212

Initramfs Address : 0x607472a4

Initramfs Size    : 0xdc6546/14443846

Compression Format: .mzip

Bootable image at @ ram:0x6042f5d8

Bootable image segment 0 address range [0x81100000, 0x81b80000]

is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@@@@@@@@@@@@@@@@@@@

File "sda9:cat3k_caa-recovery.bin" uncompressed and installed,

entry point: 0x811060f0

Loading Linux kernel with entry point 0x811060f0 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Initiating Emergency Installation of bundle tftp://192.0.2.1/

cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

Downloading bundle tftp://192.0.2.1/ cat3k_caa-universalk9.

SPA.03.03.00.SE.150-1.EZ.bin...

Validating bundle tftp://192.0.2.1/ cat3k_caa-universalk9.

SPA.03.03.00.SE.150-1.EZ.bin...

Installing bundle tftp://192.0.2.1/ cat3k_caa-universalk9.

SPA.03.03.00.SE.150-1.EZ.bin...

Verifying bundle tftp://192.0.2.1/ cat3k_caa-universalk9.

SPA.03.03.00.SE.150-1.EZ.bin...

Package cat3k_caa-base.SPA.03.03.00.SE.pkg is Digitally Signed

Package cat3k_caa-drivers.SPA.03.03.00.SE.pkg is Digitally Signed

Package cat3k_caa-infra.SPA.03.03.00.SE.pkg is Digitally Signed

Package cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg is Digitally Signed

Package cat3k_caa-platform.SPA.03.03.00.SE.pkg is Digitally Signed

Package cat3k_caa-wcm.SPA.10.0.111.0.pkg is Digitally Signed

Preparing flash...

Syncing device...

Emergency Install successful... Rebooting

Restarting system.