Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Replace a Member of C9300 or C9200 Stack

Verify Pre-Replacement

Replace

Verify Post-Replacement

Replace a Redundant Supervisor of C9400 Standalone Chassis

Verify Pre-Replacement

Replace

Verify Post-Replacement

Replace a Supervisor of C9400 Dual-Sup StackWise-Virtual

Verify Pre-Replacement

Replace

Verify Post-Replacement

Replace a Member of the C9500 StackWise-Virtual

Verify Pre-Replacement

Replace

Verify Post-Replacement

Replace a Redundant Supervisor of C9600 Dual-Sup Standalone Chassis

Verify Pre-Replacement

Replace

Verify Post-Replacement

Replace a Supervisor of C9600 Dual-Sup StackWise-Virtual

Verify Pre-Replacement

Replace

Verify Post-Replacement

Replace a Supervisor of C9600 Quad-Sup StackWise-Virtual

Replace and Verify

Introduction

This document describes how to replace a supervisor module or a stack member of Catalyst 9K switches in HA (High Availability) setup.

Prerequisites

Requirements

Cisco recommends that you are familiar with stacking, stackwise-virtual (SVL), and "bundle" Vs "install" boot mode related concepts on Catalyst 9K switches.

Components Used

The information in this document is based on these software and hardware versions:

- C9200

- C9300

- C9400

- C9500

- C9600

Note: Consult the appropriate configuration guide for the commands that are used in order to enable these features on other Cisco platforms.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document includes the processes to replace a member of stackable switch types:

- C9200/C9300 stack member

- C9500 using SVL

- Supervisor of a C9400/C9600 chassis in their various mode of operations (standalone, dual-sup, SVL and quad-sup SVL).

Replace a Member of C9300 or C9200 Stack

In this example, you  replace a member of a C9300 stack. (The switch is in this example uses switch 2 in "Install" boot mode).

Note: The same process can be used for the replacement of a C9200 stack member.

Verify Pre-Replacement

Check the current stack state and prepare for the swap. Ensure that boot variable on switch is set pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.

cat9K#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:packages.conf;

Boot Variables on next reload:

BOOT variable = flash:packages.conf;

Manual Boot = no

Enable Break = yes

Boot Mode = DEVICE

iPXE Timeout = 0

Note: If the switch is in "Install" boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode.

C9300#show run all | in software auto

no software auto-upgrade source url

software auto-upgrade enable

Note:If the stack is in "Bundle" boot mode, you need a copy of the IOS-XE .bin file that is in use on active on an USB stick or local TFTP server which can be accessed from new switch/member, through it's out-of-band (OOB) management port

Check that the stack is physically connected in full-ring, i.e if you power off the switch member in question, then existing stack would not split to cause stack-merge. Once verified, move to next steps.

Switch#sh switch neighbors

Switch #  Port 1  Port 2

--------  ------  ------

1           2       3

2           3       2

3           1       2

Note: If active switch member needs to be replaced, perform a failover to the standby switch in the stack, and wait for it to take over the active role. Skip this step, if you are going to replace any other member of the stack.

C9300#redundancy force-switchover

System configuration has been modified. Save? [yes/no]: yes

Building configuration...

Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]

Replace

Power off the member switch that needs to be replaced, disconnect power-stack and data-stack cables from it. Replace the member with new one at powered down state, reconnect the data-stack cables and power it on.

Note: IF the new unit is not running same software version as existing stack, you need to match that. For example, existing stack is running 17.3.1 and new unit is running 16.9.3.

If your stack is in "Bundle" boot mode, break into the ROMMON of new switch, while it is booting up. With help of an USB stick or OOB TFTP access, manually boot the new switch with same software version as existing stack.

Preparing to autoboot. [Press Ctrl-C to interrupt] 3 (interrupted)

rommon 1 >

rommon 2 > boot usbflash0:cat9k_iosxe.17.03.01.SPA.bin

If the stack is in "Install" boot mode, automatic software upgrade must be kicked in by current active of the stack, as soon it detects an incompatible software version or boot mode on new member switch. Typically, no manual intervention is needed at this stage.

Note: During the software auto-upgrade process, if there is need of microcode_update, the process can take several minutes. Please be patient and monitor the process closely.

Logs from Stack Active

Sep 13 07:20:21.261 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 2 has been added to the stack.

Sep 13 07:20:22.268 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 2 has been added to the stack.

Sep 13 07:20:22.546 UTC: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Switch 1 R0/0: issu_stack: Incompatible software detected.

** snip **

Sep 13 07:47:37.443 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Switch 1 R0/0: auto_upgrade_trigger: Auto upgrade initiated for switch 2.

Sep 13 07:47:37.496 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Switch 1 R0/0: auto_upgrade_trigger: Searching stack for software to upgrade switch 2.

Sep 13 07:47:37.519 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Switch 1 R0/0: auto_upgrade_trigger: Found donor switch 1 to auto upgrade switch 2.

Sep 13 07:47:37.538 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Switch 1 R0/0: auto_upgrade_trigger: Upgrading switch 2 with software from switch 1.

Sep 13 07:47:46.769 UTC: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: Switch 1 R0/0: auto_upgrade_trigger: Initiating SMU autoupgrade for switch 2

Sep 13 07:47:47.272 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: Switch 1 R0/0: auto_upgrade_trigger: Finished installing software on switch 2.

** snip **

Sep 13 07:57:18.981 UTC: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

Sep 13 07:57:18.981 UTC: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

Sep 13 07:57:49.863 UTC: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

Sep 13 07:57:50.865 UTC: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Verify Post-Replacement

Check the state of the switches once SSO is completed. At this time, you can reconnect the stack-power cables, if applicable.

C9300#show switch

Switch/Stack Mac Address : 70d3.79be.6c80 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role    Mac Address    Priority Version  State

-------------------------------------------------------------------------------------

*1     Active   70d3.79be.6c80   1       V01     Ready

2     Standby  70d3.7984.8580   2       V01     Ready

!

C9300# show module

Switch Ports  Model     Serial No. MAC address     Hw Ver. Sw Ver.

------ ----- --------- ----------- -------------- ------- --------

1      41    C9300-24U FCW2125L0BH 70d3.79be.6c80   V01    17.03.01

2      41    C9300-24U FCW2125L03W 70d3.7984.8580   V01    17.03.01

C9300#show redundancy

Redundant System Information :

------------------------------

Available system uptime = 58 minutes

Switchovers system experienced = 0

Standby failures = 0

Last switchover reason = none

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = slot 1

Current Software state = ACTIVE

Uptime in current state = 58 minutes

Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Fri 07-Aug-20 21:32 by mcpre

BOOT = flash:packages.conf;flash:;

CONFIG_FILE =

Configuration register = 0x102

Peer Processor Information :

----------------------------

Standby Location = slot 2

Current Software state = STANDBY HOT

Uptime in current state = 4 minutes

Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Fri 07-Aug-20 21:32 by mcpre

BOOT = flash:packages.conf;flash:;

CONFIG_FILE =

Configuration register = 0x102

Replace a Redundant Supervisor of C9400 Standalone Chassis

In this example, you replace the Active supervisor of a C9404 chassis. (The switch is in this example uses in slot 3 in "Install" boot mode.)

Verify Pre-Replacement

Check that the boot variable on switch is set pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.

Note: If the switch is in "Install" boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode.

C9400#show run all | in software auto

no software auto-upgrade source url

software auto-upgrade enable

Note: If your active supervisor is running in "Bundle" boot mode, please keep a copy of running software file (.bin file that you are running on active) in an USB stick or local TFTP server which can be accessed from new supervisor, through it's out-of-band (OOB) management port.

If active supervisor needs to be replaced (like in this example of ours), perform a failover to standby supervisor and wait for it to take over active's role. Skip this step if you are going to replace standby supervisor.

C9400#redundancy force-switchover

System configuration has been modified. Save? [yes/no]: yes

Building configuration...

Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]

Replace

Remove faulty supervisor from chassis and insert the new one, with a console cable plugged into it.

Note: Initially both supervisors might not be on same software version and you need to match them. For example, active supervisor could be running 16.9.5 and new/standby 16.9.4.

If your active supervisor is running in "Bundle" boot mode, then break into the ROMMON of new supervisor, while it is booting up. With help of an USB stick or OOB TFTP access, manually boot the supervisor with same software version as your active supervisor.

Preparing to autoboot. [Press Ctrl-C to interrupt] 3 (interrupted)

rommon 1 >

rommon 2 > boot usbflash0:cat9k_iosxe.16.09.05.SPA.bin

If your active supervisor is running in "Install" boot mode, automatic software upgrade must be kicked in by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor. Typically, no manual intervention is needed at this stage.

*Jun 16 19:50:15.122: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 3/0

*Jun 16 19:50:42.374: %SPA_OIR-6-ONLINECARD: SPA (C9400-SUP-1) online in subslot 3/0

C9400#

*Jun 16 19:50:43.376: 3 0 0:Ignore this incremental sync, session not ready

C9400#

*Jun 16 19:52:10.003: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F1

C9400#

*Jun 16 19:51:16.469: %IOSXE-3-PLATFORM: R1/0: kernel: dplr_intrpt: Entered dplr_intrpt_module_init dplr_intrpt 1

*Jun 16 19:51:16.472: %IOSXE-3-PLATFORM: R1/0: kernel: chr_mmap: Allocating DMA Reserve Pool ...

*Jun 16 19:52:27.950: %IOSXE_OIR-6-ONLINECARD: Card (rp) online in slot R1

*Jun 16 19:52:28.727: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: R0/0: auto_upgrade_client: Auto upgrade initiated for RP 1.

*Jun 16 19:52:28.748: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: R0/0: auto_upgrade_client: Searching stack for software to upgrade RP 1.

*Jun 16 19:52:28.760: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: R0/0: auto_upgrade_client: Found donor RP 0 to auto upgrade RP 1.

*Jun 16 19:52:28.773: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: R0/0: auto_upgrade_client: Upgrading RP 1 with software from RP 0.

*Jun 16 19:52:39.655: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

*Jun 16 19:52:39.655: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

*Jun 16 19:52:39.642: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: R0/0: auto_upgrade_client: Initiating SMU autoupgrade for RP 1

*Jun 16 19:52:40.832: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: R0/0: auto_upgrade_client: Finished installing software on RP 1.

*Jun 16 19:52:40.847: %AUTO_UPGRADE-5-AUTO_UPGRADE_RELOAD: R0/0: auto_upgrade_client: Reloading RP 1 to complete the auto upgrade.

*Jun 16 19:52:41.622: %IOSXE_OIR-6-OFFLINECARD: Card (rp) offline in slot R1

** snip **

*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

** snip **

*Jun 16 19:57:33.582: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Jun 16 19:57:34.623: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Verify Post-Replacement

Check the state of the supervisors once SSO is completed

C9400#show module

Chassis Type: C9404R

Mod Ports                           Card Type                     Model                Serial No.

---+---------------------+-------------------------------+-----------------------+--------------------

2         10                     Supervisor 1 Module           C9400-SUP-1           JAE22100647

3         10                     Supervisor 1 Module           C9400-SUP-1

Mod                    MAC addresses              Hw         Fw              Sw         Status

---+----------------------------------------+----------+--------------+--------------+------------

2         A8B4.56BF.316C to A8B4.56BF.3175       1.0       16.12.1r       16.09.05        ok

3                                                                                                                                           ok             <--- Started Syncing

Mod   Redundancy Role            Operating Redundancy Mode           Configured Redundancy Mode

---+----------------------+---------------------------------------+------------------------------------------------

2          Active                         sso                                  sso

3          Standby                        sso                                  sso

Chassis MAC address range: 44 addresses from a8b4.56bf.3140 to a8b4.56bf.316b

C9400#show redundancy

Redundant System Information :

------------------------------

Available system uptime = 10 minutes

Switchovers system experienced = 0

Standby failures = 0

Last switchover reason = none

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = slot 2

Current Software state = ACTIVE

Uptime in current state = 10 minutes

Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.5, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Thu 22-Aug-19 18:14 by mcpre

BOOT = bootflash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Peer Processor Information :

----------------------------

Standby Location = slot 3

Current Software state = STANDBY HOT

Uptime in current state = 0 minutes

Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.5, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Thu 22-Aug-19 18:14 by mcpre

BOOT = bootflash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Replace a Supervisor of C9400 Dual-Sup StackWise-Virtual

This example is for a C9400 stackwise-Virtual setup (one supervisor in each chassis), where chassis-1's supervisor (Active switch) has gone bad and needs to be replaced. The SVL is running in "Install" boot mode.

Verify Pre-Replacement

Check the current StackWise-Virtual related configurations and the state of supervisors. Ensure that boot variable on switch is set correctly pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.

9400-3#show stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1             1               TenGigabitEthernet1/5/0/1 <<< switch 1 needs to be replaced here

2             1               TenGigabitEthernet2/5/0/1

9400-3#show bootvar

BOOT variable = flash:packages.conf;

Configuration Register is 0x102

MANUAL_BOOT variable = no

BAUD variable = 9600

ENABLE_BREAK variable = yes

BOOTMODE variable does not exist

IPXE_TIMEOUT variable does not exist

CONFIG_FILE variable =

Note: If the switch is in "Install" boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode.

9400-3#show run all | in software auto

no software auto-upgrade source url

software auto-upgrade enable

Note: If your active supervisor is running in "Bundle" boot mode, please keep a copy of running software file (.bin file that you are running on active) in an USB stick or local TFTP server which can be accessed from new supervisor, through it's out-of-band (OOB) management port.

If active supervisor needs to be replaced (like in our example), perform a failover to standby supervisor and wait for standby to take over active's role. Skip this step if you are replacing standby supervisor.

9400-1#redundancy force-switchover

System configuration has been modified. Save? [yes/no]: yes

Building configuration...

Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]

Replace

Power off the chassis where supervisor needs to be replaced (In our example, it is chassis-1).

Remove the line cards from the backplane from respective chassis (where supervisor needs to be replaced). There is no need to take the line cards completely out of the chassis, as long as they are not attached to backplane, it is fine. That way, when new supervisor is inserted and being pre-staged, the remote switches of the connections (Multi-chassis etherchannel) do not put their local ports into err-disabled state (LACP and so on).

9400-3#show module

Chassis Type: C9410R

Switch Number 1

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

Mod  Redundancy Role      Operating Redundancy Mode    Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

Switch Number 2

Mod Ports Card Type                               Model           Serial No.

---+-----+--------------------------------------+--------------+--------------

1    48   48-Port UPOE w/ 24p mGig 24p RJ-45     C9400-LC-48UX   JAE2138067S

2    48   48-Port UPOE 10/100/1000 (RJ-45)       C9400-LC-48U    JAE2141091P

5   10    Supervisor 1 Module                    C9400-SUP-1     JAE2220082A

Mod          MAC addresses            Hw        Fw             Sw           Status

---+--------------------------------+----+------------+------------------+--------

1   707D.B9CF.6D1C to 707D.B9CF.6D4B 1.0    16.12.2r     16.12.03a           ok

2   6CB2.AE42.2704 to 6CB2.AE42.2733 1.0    16.12.2r     16.12.03a           ok

5   AC3A.675B.E26C to AC3A.675B.E275 1.0    16.12.2r     16.12.03a           ok

Mod  Redundancy Role        Operating Redundancy Mode  Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

5    Active                    non-redundant               sso

Insert the new supervisor into same slot where faulty supervisor was present and power it on. It must boot up in Stand-alone mode (Non-SVL), leave the Stackwise-virtual links disconnected for now.

- If your active supervisor is running in "Bundle" boot mode, then copy the software bin file (same as current active supervisor of SVL) to bootflash of new standby supervisor and change boot string accordingly.

- If your active supervisor is running in "Install" boot mode, then manual software upgrade is not needed. Software and boot mode of new supervisor must be upgraded automatically by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor.

Configure the new supervisor with Stackwise-virtual settings. (You must use same SVL domain number to match existing member.)

Switch#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Switch(config)#stackwise-virtual

Please reboot the switch for Stackwise Virtual configuration to take effect

Switch(config-stackwise-virtual)#domain 100

Switch(config-stackwise-virtual)#exit

Configure SVL and DAD ports. Use the same ports, that were used on the faulty supervisor.

9400-1(config)#interface tenGigabitEthernet 5/0/1

9400-1(config-if)#stackwise-virtual link 1

WARNING: All the extraneous configurations will be removed for TenGigabitEthernet5/0/1 on reboot

INFO: Upon reboot, the config will be part of running config but not part of start up config.

Check that SVL configuration is applied correctly to the new switch.

Switch#show stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Disabled

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

Stackwise Virtual Configuration After Reboot:

---------------------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1     1                       TenGigabitEthernet5/0/1

Check the SVL settings in ROMMON from IOSd CLI (available in versions16.12.x or higher)

9400-1#show romvar

ROMMON variables:

MAC_ADDR="70:0F:6A:DE:54:34"

SWITCH_NUMBER="1"

MODEL_NUM="C9400-SUP-1"

SYSTEM_SERIAL_NUM=""

MOTHERBOARD_SERIAL_NUM="JAE221703NQ"

TEMPLATE="access"

BAUD="9600"

LICENSE_BOOT_LEVEL="network-advantage+dna-advantage,all:MACALLAN-CHASSIS;"

MCP_STARTUP_TRACEFLAGS="00000000:00000000"

CALL_HOME_DEBUG="0000000000000"

D_STACK_DAD=""

CONFIG_FILE=""

BOOTLDR=""

SWITCH_IGNORE_STARTUP_CFG="0"

MANUAL_BOOT="no"

AUTOREBOOT_RESTORE="0"

ENABLE_BREAK="yes"

RET_2_RTS=""

AUTO_SWITCH_CONSOLE_DISABLE="0"

BOOT="flash:cat9k_iosxe.16.12.03a.SPA.bin;"

D_STACK_DISTR_STACK_LINK2=""

ABNORMAL_RESET_COUNT="1"

ROMMON_AUTOBOOT_ATTEMPT="3"

BSI="0"

RET_2_RCALTS=""

RANDOM_NUM="421133355"

D_STACK_DISTR_STACK_LINK1="Te5/0/1,"

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="100"

Save the configurations and power off chassis where new supervisor is placed into.

Connect StackWise-Virtual links between two chassis and prefer to leave the Dual-active detection link disconnected (if applicable).

Power on the chassis and monitor the boot process through console.

- If your SVL is running in "Bundle" boot mode, ensure that new supervisor is coming up with same software version as Active. If not, break into ROMMON again and boot it manually using the correct software version.

- If the SVL is running in "Install" boot mode, then "software auto-upgrade" must take care of pushing right software version and boot mode into new supervisor, without any manual intervention.

Active supervisor's log

*Sep 12 07:20:25.457: %ILPOWER-6-SET_ILPOWER: Set power allocated to POE to 4420 for slot 0

*Sep 12 07:20:30.621: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Chassis 2 R0/0: issu_stack: Incompatible software detected. Details: Active's subpackage boot mode does not match with member's super boot mode. Please boot switch 1 in subpackage mode.

*Sep 12 07:20:40.779: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.

*Sep 12 07:21:00.978: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade initiated for switch 1.

*Sep 12 07:21:01.031: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Chassis 2 R0/0: auto_upgrade_client: Searching stack for software to upgrade switch 1.

*Sep 12 07:21:01.053: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Chassis 2 R0/0: auto_upgrade_client: Found donor switch 2 to auto upgrade switch 1.

*Sep 12 07:21:01.074: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Chassis 2 R0/0: auto_upgrade_client: Upgrading switch 1 with software from switch 2.

Logs from new supervisor's console

Waiting for remote chassis to join

#########################################

Chassis number is 1

All chassis in the stack have been discovered. Accelerating discovery

Chassis 1 reloading, reason - System requested reload <<< reload is instructed by current active as part of software auto-upgrade

Sep 12 07:25:23.306: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload fru code

All the running configurations must be automatically synced from Active supervisor to the new one. Wait for these logs from active supervisor.

*Sep 12 07:33:39.803: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Sep 12 07:33:40.837: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

- Once SSO is completed, proceed with connecting the Dual-active detection (DAD) link and other network uplink ports on new supervisor, as applicable.

- Push the line cards back inside, to get them attached to backplane again

- Verify that all line cards have booted fine, passed online diagnostic tests and brought up their interfaces, including port-channel binding and so on.

Verify Post-Replacement

Check the StackWise Virtual related configurations and the states of the switch using these commands.

9400-3#sh redundancy

Redundant System Information :

------------------------------

Available system uptime = 1 hour, 31 minutes

Switchovers system experienced = 0

Standby failures = 0

Last switchover reason = none

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = Switch 2

Current Software state = ACTIVE

Uptime in current state = 31 minutes

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.3a, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Tue 28-Apr-20 09:37 by mcpre

BOOT = flash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Peer Processor Information :

----------------------------

Standby Location = Switch 1

Current Software state = STANDBY HOT

Uptime in current state = 4 minutes

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.3a, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Tue 28-Apr-20 09:37 by mcpre

BOOT = flash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

!

9400-3#sh stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1          1                   TenGigabitEthernet1/5/0/1

2          1                   TenGigabitEthernet2/5/0/1

9400-3#sh module

Chassis Type: C9410R

Switch Number 1

Mod Ports         Card Type                       Model          Serial No.

---+-----+--------------------------------------+--------------+--------------

1    48     48-Port UPOE w/ 24p mGig 24p RJ-45   C9400-LC-48UX  JAE22360153

2    48     48-Port UPOE w/ 24p mGig 24p RJ-45   C9400-LC-48UX  JAE215103V7

5    10     Supervisor 1 Module                  C9400-SUP-1    JAE221703NQ

Mod   MAC addresses                   Hw     Fw          Sw                Status

---+--------------------------------+----+------------+------------------+--------

1    00B7.71FA.D878 to 00B7.71FA.D8A7 1.0  16.12.2r     16.12.03a           ok

2    4C77.6DBF.4A94 to 4C77.6DBF.4AC3 1.0  16.12.2r     16.12.03a           ok

5    AC3A.675B.E9AC to AC3A.675B.E9B5 1.0  16.12.2r     16.12.03a           ok

Mod Redundancy Role       Operating Redundancy Mode   Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

5     Standby                  sso                          sso

**snip**

Replace a Member of the C9500 StackWise-Virtual

In this example, you are considering a C9500 Stackwise Virtual setup with Switch-1 (the Active switch) as a faulty switch that needs to be replaced. The SVL is running in INSTALL boot mode.

Verify Pre-Replacement

Check the current StackWise-Virtual related configurations and the state of the switches. Ensure that boot variable is set correctly, points to packages.conf, and config-register is set to 0x2102.

C9500-1#show stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1            1                TwentyFiveGigE1/0/1

TwentyFiveGigE1/0/2

2            1                TwentyFiveGigE2/0/1

TwentyFiveGigE2/0/2

C9500-1#show stackwise-virtual dual-active-detection

Dual-Active-Detection Configuration:

-------------------------------------

Switch Dad port

------ ------------

1       TwentyFiveGigE1/0/3

2       TwentyFiveGigE2/0/3    <<<<<<<<<< Ports configured for Dual-Active Detection (DAD)

Note : Configs of these DAD ports do not show up in running-config

!

interface TwentyFiveGigE 1/0/3

end

!

interface TwentyFiveGigE 2/0/3

end

C9500-1#show switch

Switch/Stack Mac Address : f4db.e619.0480 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role      Mac Address    Priority  Version  State

-------------------------------------------------------------------------------------

*1     Active    f4db.e619.0480    15       V02     Ready

2     Standby   f4db.e618.fa80     1       V02     Ready

C9500-1#show redundancy

Redundant System Information :

------------------------------

Available system uptime = 4 minutes

Switchovers system experienced = 0

Standby failures = 0

Last switchover reason = none

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = slot 1

Current Software state = ACTIVE

Uptime in current state = 4 minutes

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Tue 19-Nov-19 10:04 by mcpre

BOOT = flash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Peer Processor Information :

----------------------------

Standby Location = slot 2

Current Software state = STANDBY HOT

Uptime in current state = 1 minute

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Tue 19-Nov-19 10:04 by mcpre

BOOT = flash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Note: If the SVL is running in INSTALL boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode. (Skip this step, if SVL is running in Bundle boot mode).

C9500-1#show run all | in software auto

no software auto-upgrade source url

software auto-upgrade enable

If active switch needs to be replaced, perform a failover to Standby switch and wait for standby to take over active's role. (Skip this step if you are replacing Standby unit).

C9500-1#redundancy force-switchover

System configuration has been modified. Save? [yes/no]: yes

Building configuration...

Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]

Replace

Power off the Switch that needs to be replaced. Disconnect all the cables from that switch.

C9500-1#show switch

Switch/Stack Mac Address : f4db.e619.0480 - Foreign Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role      Mac Address    Priority Version  State

-------------------------------------------------------------------------------------

1     Member    0000.0000.0000    0        V02    Removed  << switch 1 is powered down

*2     Active    f4db.e618.fa80    1        V02    Ready

Power on the new switch. It must boot up in Stand-alone mode (Non-SVL). (Skip this step if the current Active SVL is running in INSTALL boot mode)

Check the software version on new unit. If it does not match with existing member of StackWise-Virtual unit, then pre-stage that to match the software version and license, with existing member of SVL. (You can load the correct software version through TFTP/FTP/SFTP or using an USB stick, and after matching the software version and license on new unit, proceed to next step.

Cisco IOS XE Software, Version 16.12.02

Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Tue 19-Nov-19 10:04 by mcpre

Note: If SVL is running in INSTALL boot mode and software-autoupgrade is enabled, then typically the existing active member of SVL must be able to match the code and boot mode of new unit automatically.

Configure StackWise Virtual on the new switch. You must use same SVL domain number to match existing member.

Switch#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Switch(config)#stackwise-virtual

Please reboot the switch for Stackwise Virtual configuration to take effect

Switch(config-stackwise-virtual)#domain 100

Switch(config-stackwise-virtual)#exit

Configure SVL and DAD ports. Usesame ports, that were used on the faulty switch.

Switch(config)#int range twe1/0/1-2

Switch(config-if-range)#stackwise-virtual link 1

WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/1 on reboot

WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/2 on reboot

Switch(config-if-range)#exit

Switch(config)#int twe1/0/3

Switch(config-if)#stackwise-virtual dual-active-detection

WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/3 on reboot.

Check that SVL configuration is applied correctly to the new switch.

Switch#show stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Disabled

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

Stackwise Virtual Configuration After Reboot:

---------------------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1     1                       TwentyFiveGigE1/0/1

TwentyFiveGigE1/0/2

Switch#show stackwise-virtual dual-active-detection

Dual-Active-Detection Configuration:

-------------------------------------

Switch Dad port

------ ------------

Distributed Stack DAD Configuration After Reboot:

------------------------------------------------

Switch Dad port

------ --------

1        TwentyFiveGigE1/0/3

Save the configurations and power off the new switch.

Connect StackWise-Virtual links between existing SVL member and the new unit. Prefer to leave the Dual-active detection link disconnected.

Power on the new unit. If there is a conflict in switch number, then new unit must be re-numbered automatically.

Chassis is reloading, reason: Configured Switch num conflicts with peer, Changing local switch number to 1 and reloading to take effect

Sep 10 22:41:50.738: %PMAN-3-PROCHOLDDOWN: R0/0: The process nif_mgr has been helddown (rc 69)

Note: If new unit is running an incompatible software or boot mode and existing SVL member is running INSTALL boot mode, then software auto-upgrade kicks in, to bring new unit on INSTALL boot mode, without manual intervention.

*Sep 10 22:47:05.996: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.

All the running configurations are automatically synced from Active switch to the New switch. No additional configurations are needed. Wait for these logs from active switch.

*Sep 11 01:02:28.974: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

C9500-1#

*Sep 11 01:02:30.009: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Proceed with connecting the Dual-active detection (DAD) link and other network ports. (Once SSO is completed)

Verify Post-Replacement

Check the StackWise Virtual related configurations and the states of the switch using these commands.

C9500-1#show stackwise-virtual

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link    Ports

------ ----------------------    ------

1              1              TwentyFiveGigE1/0/1

TwentyFiveGigE1/0/2

2              1              TwentyFiveGigE2/0/1

TwentyFiveGigE2/0/2

C9500-1#show redundancy

Redundant System Information :

------------------------------

Available system uptime = 14 minutes

Switchovers system experienced = 0

Standby failures = 0

Last switchover reason = none

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = slot 2

Current Software state = ACTIVE

Uptime in current state = 14 minutes

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Tue 19-Nov-19 10:04 by mcpre

BOOT = flash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Peer Processor Information :

----------------------------

Standby Location = slot 1

Current Software state = STANDBY HOT

Uptime in current state = 1 minute

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2019 by Cisco Systems, Inc.

Compiled Tue 19-Nov-19 10:04 by mcpre

BOOT = flash:packages.conf;

CONFIG_FILE =

Configuration register = 0x102

Replace a Redundant Supervisor of C9600 Dual-Sup Standalone Chassis

In this example, you are considering to replace the active supervisor at slot 3 of a C9606 chassis. (The switch is running in "Install" boot mode.)

Verify Pre-Replacement

Check that boot variable on switch is set correctly pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.

Note: If the switch is running in "Install" boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode.

C9600R-1#show run all | in software auto

no software auto-upgrade source url

software auto-upgrade enable

Note: If your active supervisor is running in "Bundle" boot mode, please keep a copy of running software file (.bin file that you are running on active) in an USB stick or local TFTP server which can be accessed from new supervisor, through it's out-of-band (OOB) management port.

Replace

If active supervisor needs to be replaced (like in this example of ours), perform a failover to standby supervisor and wait for it to take over active's role. (Skip this step if you are going to replace standby supervisor).

C9600R-1#redundancy force-switchover

System configuration has been modified. Save? [yes/no]: yes

Building configuration...

Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]

Remove faulty supervisor from chassis and insert the new one, with a console cable plugged into it.

Note: Initially both supervisors might not be on same software version and you need to match them. For example, active supervisor could be running 16.12.4 and new/standby 16.12.2.

If your active supervisor is running in "Bundle" boot mode, then break into the ROMMON of new supervisor, while it is booting up. With help of an USB stick or OOB TFTP access, manually boot the supervisor with same software version as your active supervisor. Later on, after new standby joins SSO, copy the running software to it's local bootflash.

Preparing to autoboot. [Press Ctrl-C to interrupt] 3 (interrupted)

rommon 1 >

rommon 2 > boot disk0:cat9k_iosxe.16.12.04.SPA.bin

If your active supervisor is running in "Install" boot mode, automatic software upgrade must be kicked in by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor. Typically, no manual intervention is needed at this stage.

*Sep 12 21:32:04.886: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

*Sep 12 21:32:04.886: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

*Sep 12 21:32:07.773: %REDUNDANCY-2-IPC: IOS versions do not match.

*Sep 12 21:32:07.823: %SMART_LIC-5-EVAL_START: Entering evaluation period

*Sep 12 21:32:28.980: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: R1/0: auto_upgrade_client: Initiating SMU autoupgrade for RP 0

*Sep 12 21:32:30.867: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: R1/0: auto_upgrade_client: Finished installing software on RP 0.

*Sep 12 21:32:30.908: %AUTO_UPGRADE-5-AUTO_UPGRADE_RELOAD: R1/0: auto_upgrade_client: Reloading RP 0 to complete the auto upgrade.

** snip **

*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

** snip **

*Sep 12 21:36:37.786: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

*Sep 12 21:36:37.786: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

**snip**

*Sep 12 21:39:24.085: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Sep 12 21:39:25.124: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Verify Post-Replacement

Check the state of the supervisors once SSO is completed

C9606R-1#show mod

Chassis Type: C9606R

Mod  Ports  Card Type                               Model        Serial No.

---+-----+--------------------------------------+--------------+--------------

1    24    24-Port 40GE/12-Port 100GE             C9600-LC-24C   CAT2313L2WQ

2    48    48-Port 10GE / 25GE                    C9600-LC-48YL  CAT2314L36W

3    0     Supervisor 1 Module                    C9600-SUP-1    CAT2310L5C1

4    0     Supervisor 1 Module                    C9600-SUP-1    CAT2311L4DQ

5    48    48-Port 10GE / 25GE                    C9600-LC-48YL  CAT2310L57N

Mod       MAC addresses               Hw      Fw        Sw               Status

---+--------------------------------+----+------------+------------------+--------

1   DC8C.37C9.AC00 to DC8C.37C9.AC7F  1.0  17.1.1[FC2]  16.12.04           ok

2   DC8C.37C9.FD00 to DC8C.37C9.FD7F  1.0  17.1.1[FC2]  16.12.04           ok

3   DC8C.3772.C780 to DC8C.3772.C7FF  1.0  17.1.1[FC2]  16.12.04           ok

4   DC8C.3772.E580 to DC8C.3772.E5FF  1.0  17.1.1[FC2]  16.12.04           ok

5   DC8C.3773.0280 to DC8C.3773.02FF  1.0  17.1.1[FC2]  16.12.04           ok

Mod  Redundancy Role      Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3     Standby                     sso                      sso

4     Active                      sso                      sso

Chassis MAC address range: 64 addresses from 6cb2.ae4a.9680 to 6cb2.ae4a.96bf

C9606R-1#show redundancy

Redundant System Information :

------------------------------

Available system uptime = 1 day, 11 hours, 32 minutes

Switchovers system experienced = 1

Standby failures = 1

Last switchover reason = user forced

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = slot 4

Current Software state = ACTIVE

Uptime in current state = 35 minutes

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Thu 09-Jul-20 21:49 by mcpre

BOOT =

CONFIG_FILE =

Peer Processor Information :

----------------------------

Standby Location = slot 3

Current Software state = STANDBY HOT

Uptime in current state = 3 minutes

Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Thu 09-Jul-20 21:49 by mcpre

BOOT =

CONFIG_FILE =

Replace a Supervisor of C9600 Dual-Sup StackWise-Virtual

In this example, you are considering a C9600 stackwise-Virtual setup (one supervisor in each chassis), where chassis-1's supervisor (Active switch) has gone bad and needs to be replaced. The SVL is running in "Install" boot mode.

Verify Pre-Replacement

Check the current StackWise-Virtual related configurations and the state of supervisors. Ensure that boot variable on switch is set correctly pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.

C9600_SVL#sh stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

2           1                 FortyGigabitEthernet2/1/0/1

FortyGigabitEthernet2/1/0/2

1           1                 FortyGigabitEthernet1/1/0/1  << supervisor of SW1 needs to be replaced

FortyGigabitEthernet1/1/0/2

C9600_SVL#show bootvar

BOOT variable = bootflash:packages.conf;

MANUAL_BOOT variable = no

BAUD variable = 9600

ENABLE_BREAK variable = yes

BOOTMODE variable does not exist

IPXE_TIMEOUT variable does not exist

CONFIG_FILE variable =

Note: If the SVL is in "Install" boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode.

C9600_SVL#show run all | in software auto

no software auto-upgrade source url

software auto-upgrade enable

If your active supervisor is running in "Bundle" boot mode, please keep a copy of running software file (.bin file that you are running on active) in an USB stick or local TFTP server which can be accessed from new supervisor, through it's out-of-band (OOB) management port.

- If active supervisor needs to be replaced (like in our example), perform a failover to standby supervisor and wait for standby to take over active's role. Skip this step if you are replacing standby supervisor.

C9600_SVL#redundancy force-switchover

System configuration has been modified. Save? [yes/no]: yes

Building configuration...

Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]

Replace

Power off the chassis where supervisor needs to be replaced. In our example, it is chassis-1.

Remove the line cards from the backplane from respective chassis (where supervisor needs to be replaced), except one where StackWise-Virtual links (SVL) are attached to. For those line cards, where SVLs are configured, remove all connections except the SVLs itself. That way, when new supervisor is inserted and being pre-staged, the remote switches of the connections (Multi-chassis etherchannel) do not put their local ports into err-disabled state (LACP and so on).

C9600_SVL#show module

Chassis Type: C9606R

Switch Number 1

Mod  Ports        Card Type                       Model           Serial No.

---+-----+--------------------------------------+--------------+--------------

Mod     MAC addresses                Hw      Fw            Sw           Status

---+--------------------------------+----+------------+------------------+--------

Mod  Redundancy Role    Operating Redundancy Mode  Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

Switch Number 2

Mod Ports           Card Type                       Model         Serial No.

---+-----+--------------------------------------+--------------+--------------

1    24    24-Port 40GE/12-Port 100GE              C9600-LC-24C   CAT2310L4DW

2    48    48-Port 10GE / 25GE                     C9600-LC-48YL  CAT2310L59S

3    0     Supervisor 1 Module                     C9600-SUP-1    CAT2340L40Q

5    24    24-Port 40GE/12-Port 100GE              C9600-LC-24C   CAT2313L2W1

Mod        MAC addresses              Hw    Fw             Sw              Status

---+--------------------------------+----+------------+------------------+--------

1   DC8C.379F.DB80 to DC8C.379F.DBFF  1.0  17.3.1r[FC2]  17.03.01           ok

2   DC8C.3772.FD80 to DC8C.3772.FDFF  1.0  17.3.1r[FC2]  17.03.01           ok

3   7C21.0E5D.0800 to 7C21.0E5D.087F  1.0  17.3.1r[FC2]  17.03.01           ok

5   DC8C.37A0.D180 to DC8C.37A0.D1FF  1.0  17.3.1r[FC2]  17.03.01           ok

Mod    Redundancy Role  Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3       Active               non-redundant                 sso

Chassis 2 MAC address range: 64 addresses from 2c4f.523b.bd00 to 2c4f.523b.bd3f

Insert the new supervisor into same slot where faulty supervisor was present and power it on. It must boot up in Stand-alone mode (Non-SVL), leave the Stackwise-virtual links and DAD links disconnected for now.

If your active supervisor is running in "Bundle" boot mode, then copy the software bin file (same as current active supervisor of SVL) to bootflash of new standby supervisor and change bootstring accordingly.

If your active supervisor is running in "Install" boot mode, then manual software upgrade is not needed. Software and boot mode of new supervisor must be upgraded automatically by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor.

Configure the new supervisor with Stackwise-virtual settings. (You must use same SVL domain number to match existing member).

Switch#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

Switch(config)#stackwise-virtual

Please reboot the switch for Stackwise Virtual configuration to take effect

Switch(config-stackwise-virtual)#domain 100

Switch(config-stackwise-virtual)#exit

Configure SVL and DAD ports. Usesame ports, that were used on the faulty supervisor.

Switch(config)#int range fortyGigabitEthernet 1/0/1 -2

Switch(config-if-range)#stackwise-virtual link 1

Switch(config)#int range twentyFiveGigE 2/0/25 -26

Switch(config-if-range)#stackwise-virtual dual-active-detection

Check that SVL configuration is applied correctly to the new switch.

Switch#show stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual Configuration After Reboot:

---------------------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link  Ports

------ ----------------------  ------

1          1                   FortyGigabitEthernet1/0/1

FortyGigabitEthernet1/0/2

Switch#show stackwise-virtual dual-active-detection

In dual-active recovery mode: No

Dual-Active-Detection Configuration:

-------------------------------------

Switch Dad port Status

------ ------------ ---------

Distributed Stack DAD Configuration After Reboot:

------------------------------------------------

Switch     Dad port            Status

------      --------          ------

1      TwentyFiveGigE2/0/25   down

TwentyFiveGigE2/0/26   down

If software version is 16.12.x or higher, you can check the SVL settings in ROMMON from IOSd CLI.

Switch#show romvar

ROMMON variables:

BOARDID="38"

ETHER_PORT="2"

PS1="rommon ! >"

MAC_ADDR="7C:21:0E:5D:04:00"

DOPPLER_E_WA="1"

RETRY="0"

MODEL_NUM="C9600-SUP-1"

SYSTEM_SERIAL_NUM="CAT2340L3Y5"

MOTHERBOARD_SERIAL_NUM="CAT2340L3Y5"

TEMPLATE="core"

BAUD="9600"

AUTO_SWITCH_CONSOLE_DISABLE="0"

PSEUDO_OIR_REMOVE_SET="1"

CALL_HOME_DEBUG="0000000000000"

ENABLE_BREAK="yes"

RET_2_RTS=""

CRASHINFO="bootflash:crashinfo_RP_00_00_20200225-024401-UTC"

MCP_STARTUP_TRACEFLAGS="00000000:00000000"

CONFIG_FILE=""

BOOTLDR=""

RECOVERY_RELOAD_DISABLE=""

SWITCH_PRIORITY="1"

SWITCH_NUMBER="1"

SWITCH_IGNORE_STARTUP_CFG="0"

D_STACK_DISTR_STACK_LINK2=""

MANUAL_BOOT="no"

AUTOREBOOT_RESTORE="0"

ABNORMAL_RESET_COUNT="0"

ROMMON_AUTOBOOT_ATTEMPT="3"

BSI="0"

RET_2_RCALTS=""

RANDOM_NUM="1430571596"

BOOT="bootflash:cat9k_iosxe.16.12.02.SPA.bin;"

D_STACK_DISTR_STACK_LINK1="Fo1/0/1,Fo1/0/2,"

D_STACK_DAD="Twe2/0/25,Twe2/0/26,"

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="100"

Save the configurations and power off chassis where new supervisor is placed into.

Connect StackWise-Virtual links between two chassis and prefer to leave the Dual-active detection link disconnected (if applicable).

Power on the chassis and monitor the boot process through console.

Note: If your SVL is running in "Bundle" boot mode, please ensure that new supervisor is coming up with same software version as Active. If not, break into ROMMON again and boot it manually using the correct software version.

Note: If the SVL is in "Install" boot mode, verify that software auto-upgrade is enabled. If not, then enable that by configuring "software auto-upgrade enable" from global configuration mode.

Active supervisor's log-

*Sep 13 00:59:49.367: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.

*Sep 13 00:59:51.988: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.

*Sep 13 00:59:52.135: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Chassis 2 R0/0: issu_stack: Incompatible software detected. Details: Active's subpackage boot mode does not match with member's super boot mode. Please boot switch 1 in subpackage mode.

*Sep 13 00:59:52.297: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.

*Sep 13 00:59:53.311: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade initiated for switch 1.

*Sep 13 00:59:53.368: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Chassis 2 R0/0: auto_upgrade_client: Searching stack for software to upgrade switch 1.

*Sep 13 00:59:53.397: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Chassis 2 R0/0: auto_upgrade_client: Found donor switch 2 to auto upgrade switch 1.

*Sep 13 00:59:53.423: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Chassis 2 R0/0: auto_upgrade_client: Upgrading switch 1 with software from switch 2.

Logs from new supervisor's console-

Waiting for remote chassis to join

##################################################################################

Chassis number is 1

All chassis in the stack have been discovered. Accelerating discovery

Chassis 1 reloading, reason - System requested reload <<< reload is instructed by current active as part of software auto-upgrade

All the running configurations are automatically synced from Active supervisor to the new one. Wait for these logs from active supervisor.

*Sep 13 01:14:18.552: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Sep 13 01:14:18.577: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Once SSO is completed, proceed with connecting the Dual-active detection (DAD) link as well.

- Push the line cards back inside (for chassis where supervisor is replaced), to get those attached to backplane again. Now, reconnect the cables.

- Verify that all line cards have booted fine, passed online diagnostic tests and brought up their interfaces, including port-channel binding and so on.

Verify Post-Replacement

Check the StackWise Virtual related configurations and the states of the switch using these commands.

C9600_SVL#show redundancy

Redundant System Information :

------------------------------

Available system uptime = 1 hour, 27 minutes

Switchovers system experienced = 0

Standby failures = 0

Last switchover reason = none

Hardware Mode = Duplex

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Maintenance Mode = Disabled

Communications = Up

Current Processor Information :

-------------------------------

Active Location = Switch 2

Current Software state = ACTIVE

Uptime in current state = 1 hour, 27 minutes

Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Fri 07-Aug-20 21:32 by mcpre

BOOT = bootflash:packages.conf;

CONFIG_FILE =

Peer Processor Information :

----------------------------

Standby Location = Switch 1

Current Software state = STANDBY HOT

Uptime in current state = 0 minutes

Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Fri 07-Aug-20 21:32 by mcpre

BOOT = bootflash:packages.conf;

CONFIG_FILE =

C9600_SVL#show stackwise-virtual

Stackwise Virtual Configuration:

--------------------------------

Stackwise Virtual : Enabled

Domain Number : 100

Switch Stackwise Virtual Link Ports

------ ---------------------- ------

1          1                   FortyGigabitEthernet1/1/0/1

FortyGigabitEthernet1/1/0/2

2          1                   FortyGigabitEthernet2/1/0/1

FortyGigabitEthernet2/1/0/2

C9600_SVL#show stackwise-virtual dual-active-detection

In dual-active recovery mode: No

Recovery Reload: Enabled

Dual-Active-Detection Configuration:

-------------------------------------

Switch   Dad port             Status

------  ------------          ---------

1      TwentyFiveGigE1/2/0/25  up

TwentyFiveGigE1/2/0/26  up

2      TwentyFiveGigE2/2/0/25  up

TwentyFiveGigE2/2/0/26  up

C9600_SVL#show module

Chassis Type: C9606R

Switch Number 1

Mod Ports       Card Type                             Model       Serial No.

---+-----+--------------------------------------+--------------+--------------

1   24     24-Port 40GE/12-Port 100GE             C9600-LC-24C   CAT2252L0PR

2   48     48-Port 10GE / 25GE                    C9600-LC-48YL  CAT2334L0BA

3   0      Supervisor 1 Module                    C9600-SUP-1    CAT2340L3Y5

5   48     48-Port 10GE / 25GE                    C9600-LC-48YL  CAT2337L509

Mod   MAC addresses                    Hw  Fw              Sw              Status

---+--------------------------------+----+------------+------------------+--------

1    70B3.175A.8100 to 70B3.175A.817F 1.0  17.3.1r[FC2]  17.03.01            ok

2    10B3.D652.9900 to 10B3.D652.997F 1.0  17.3.1r[FC2]  17.03.01            ok

3    7C21.0E5D.0400 to 7C21.0E5D.047F 1.0  17.3.1r[FC2]  17.03.01            ok

5    4C71.0D7C.8400 to 4C71.0D7C.847F 1.0  17.3.1r[FC2]  17.03.01            ok

Mod  Redundancy Role     Operating Redundancy Mode  Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3     Standby                sso                           sso

Switch Number 2

**snip**

Replace a Supervisor of C9600 Quad-Sup StackWise-Virtual

In this example, you are considering a C9600 Quad Sup stackwise-Virtual setup (two supervisors in each chassis), where a supervisor has gone bad and needs to be replaced. The SVL is running in "Install" boot mode.

Replace and Verify

Pulling out the faulty supervisor

- If the supervisor to be replaced isGlobal Active supervisor (Sw-1 Slot 3 in the image shown), perform a failover so that the Global Standby (Sw-2 Slot 3 in the previous picture) takes over as Active. Wait until there is a new Global Standby and SSO gets completed. (In this case, Sw-1 Slot 4 to become new Global Standby).

- If the supervisor to be replaced isGlobal Standby supervisor (Sw-2 Slot 3 in the image shown), pull the supervisor out. Wait until there is a new Global Standby and SSO gets completed. (In this case, Sw-2 Slot 4 to become new Global Standby)

- If the supervisor to be replaced isICS supervisor(Sw-1 Slot 4 or Sw-2 Slot 4 in the image shown), pull the supervisor out.

Inserting the new supervisor

- If the new supervisor is running on 17.x code, the steps are straight forward. Just insert the new supervisor. If ICS supervisors have 17.x image, they automatically boot and become part of Quad-sup. Even if it is running different 17.x code than that of what is running one on the production setup, Software Auto-upgrade to automatically take care of upgrading the ICS supervisor with the same 17.x code in INSTALL mode.

- If the new supervisor is running on 16.x code, or if you are unsure of the code it is running on, try to insert the supervisor on a spare chassis and get it upgraded to 17.x code. If there is no spare chassis to upgrade, these steps have to be taken.

- This step is very important. Insert the ICS supervisor and use Ctrl+C to break them to rommon.If you miss to break into ROMMON and supervisor boots up on 16.x code, this could take down the complete chassis where the supervisor was inserted

Look for any SVL related rommon variables. These variables start with D_STACK. Usually a new supervisor do not have these variables set.

D_STACK_DISTR_STACK_LINK2=""

D_STACK_DAD="Fo1/0/13,Fo1/0/15,"

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="255"

D_STACK_DISTR_STACK_LINK1="Fo1/0/10,Fo1/0/15,Fo1/0/16,Fo1/0/17,Fo1/0/3,Fo1/0/6,"

Unset all the previously shown variables

rommon 1 > unset D_STACK_DAD

rommon 1 > unset D_STACK_DISTR_STACK_LINK1

rommon 1 > unset D_STACK_DOMAIN_NUM

rommon 1 > unset D_STACK_MODE

Look for variable "SWITCH_NUMBER=1". If the switch number is 2, set the variable to 1. If it is already 1, move to the next step.

rommon 1 > SWITCH_NUMBER=1

Set to manual boot the supervisor.

rommon 1 > MANUAL_BOOT=YES

Manually boot the ICS supervisor in bundle mode using USB/TFTP on 17.x code. Do not change boot variable in rommon. Just boot it manually from rommon.

- The supervisor is reset as it detects an existing ICS in SVL mode, so it gets converted from stand-alone to stackwise virtual mode. It could again fall back into rommon, since auto-boot is disabled.

Unset manual boot to enable auto-boot.

rommon 1 > unset MANUAL_BOOT

Manually boot the ICS supervisor in bundle mode using USB/TFTP on 17.x code. Do not change boot variable in rommon. Just boot it manually from rommon. This step boots the ICS in BUNDLE mode.

Note: Software Auto-upgrade is meant to automatically take care of upgrading the ICS supervisor with 17.x code in INSTALL mode and reloads ICS sups to come up in RPR. If auto-upgrade is disabled, you can also run the command "install autoupgrade" from active supervisor.