Replace a Supervisor Module or Stack Member of Catalyst 9000 in HA
Available Languages
Contents
Introduction
This document describes how to replace a supervisor module or a stack member of Catalyst 9K switches in HA (High Availability) setup.
Prerequisites
Requirements
Cisco recommends that you are familiar with stacking, stackwise-virtual (SVL), and "bundle" Vs "install" boot mode related concepts on Catalyst 9K switches.
Components Used
The information in this document is based on these software and hardware versions:
- C9200
- C9300
- C9400
- C9500
- C9600
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
This document includes the processes to replace a member of stackable switch types:
- C9200/C9300 stack member
- C9500 using SVL
- Supervisor of a C9400/C9600 chassis in their various mode of operations (standalone, dual-sup, SVL and quad-sup SVL).
Replace a Member of C9300 or C9200 Stack
In this example, you replace a member of a C9300 stack. (The switch is in this example uses switch 2 in "Install" boot mode).
Verify Pre-Replacement
Check the current stack state and prepare for the swap. Ensure that boot variable on switch is set pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.
cat9K#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;
Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = yes
Boot Mode = DEVICE
iPXE Timeout = 0
C9300#show run all | in software auto
no software auto-upgrade source url
software auto-upgrade enable
Check that the stack is physically connected in full-ring, i.e if you power off the switch member in question, then existing stack would not split to cause stack-merge. Once verified, move to next steps.
Switch#sh switch neighbors
Switch # Port 1 Port 2
-------- ------ ------
1 2 3
2 3 2
3 1 2
C9300#redundancy force-switchover System configuration has been modified. Save? [yes/no]: yes Building configuration... Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]
Replace
Power off the member switch that needs to be replaced, disconnect power-stack and data-stack cables from it. Replace the member with new one at powered down state, reconnect the data-stack cables and power it on.
If your stack is in "Bundle" boot mode, break into the ROMMON of new switch, while it is booting up. With help of an USB stick or OOB TFTP access, manually boot the new switch with same software version as existing stack.
Preparing to autoboot. [Press Ctrl-C to interrupt] 3 (interrupted)
rommon 1 >
rommon 2 > boot usbflash0:cat9k_iosxe.17.03.01.SPA.bin
If the stack is in "Install" boot mode, automatic software upgrade must be kicked in by current active of the stack, as soon it detects an incompatible software version or boot mode on new member switch. Typically, no manual intervention is needed at this stage.
Logs from Stack Active
Sep 13 07:20:21.261 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 2 has been added to the stack.
Sep 13 07:20:22.268 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 2 has been added to the stack.
Sep 13 07:20:22.546 UTC: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Switch 1 R0/0: issu_stack: Incompatible software detected.
** snip **
Sep 13 07:47:37.443 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Switch 1 R0/0: auto_upgrade_trigger: Auto upgrade initiated for switch 2.
Sep 13 07:47:37.496 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Switch 1 R0/0: auto_upgrade_trigger: Searching stack for software to upgrade switch 2.
Sep 13 07:47:37.519 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Switch 1 R0/0: auto_upgrade_trigger: Found donor switch 1 to auto upgrade switch 2.
Sep 13 07:47:37.538 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Switch 1 R0/0: auto_upgrade_trigger: Upgrading switch 2 with software from switch 1.
Sep 13 07:47:46.769 UTC: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: Switch 1 R0/0: auto_upgrade_trigger: Initiating SMU autoupgrade for switch 2
Sep 13 07:47:47.272 UTC: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: Switch 1 R0/0: auto_upgrade_trigger: Finished installing software on switch 2.
** snip **
Sep 13 07:57:18.981 UTC: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
Sep 13 07:57:18.981 UTC: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
Sep 13 07:57:49.863 UTC: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Sep 13 07:57:50.865 UTC: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Verify Post-Replacement
Check the state of the switches once SSO is completed. At this time, you can reconnect the stack-power cables, if applicable.
C9300#show switch
Switch/Stack Mac Address : 70d3.79be.6c80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active 70d3.79be.6c80 1 V01 Ready
2 Standby 70d3.7984.8580 2 V01 Ready
!
C9300# show module
Switch Ports Model Serial No. MAC address Hw Ver. Sw Ver.
------ ----- --------- ----------- -------------- ------- --------
1 41 C9300-24U FCW2125L0BH 70d3.79be.6c80 V01 17.03.01
2 41 C9300-24U FCW2125L03W 70d3.7984.8580 V01 17.03.01
C9300#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 58 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 58 minutes
Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Fri 07-Aug-20 21:32 by mcpre
BOOT = flash:packages.conf;flash:;
CONFIG_FILE =
Configuration register = 0x102
Peer Processor Information :
----------------------------
Standby Location = slot 2
Current Software state = STANDBY HOT
Uptime in current state = 4 minutes
Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Fri 07-Aug-20 21:32 by mcpre
BOOT = flash:packages.conf;flash:;
CONFIG_FILE =
Configuration register = 0x102
Replace a Redundant Supervisor of C9400 Standalone Chassis
In this example, you replace the Active supervisor of a C9404 chassis. (The switch is in this example uses in slot 3 in "Install" boot mode.)
Verify Pre-Replacement
Check that the boot variable on switch is set pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.
C9400#show run all | in software auto
no software auto-upgrade source url
software auto-upgrade enable
If active supervisor needs to be replaced (like in this example of ours), perform a failover to standby supervisor and wait for it to take over active's role. Skip this step if you are going to replace standby supervisor.
C9400#redundancy force-switchover System configuration has been modified. Save? [yes/no]: yes Building configuration... Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]
Replace
Remove faulty supervisor from chassis and insert the new one, with a console cable plugged into it.
If your active supervisor is running in "Bundle" boot mode, then break into the ROMMON of new supervisor, while it is booting up. With help of an USB stick or OOB TFTP access, manually boot the supervisor with same software version as your active supervisor.
Preparing to autoboot. [Press Ctrl-C to interrupt] 3 (interrupted)
rommon 1 >
rommon 2 > boot usbflash0:cat9k_iosxe.16.09.05.SPA.bin
If your active supervisor is running in "Install" boot mode, automatic software upgrade must be kicked in by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor. Typically, no manual intervention is needed at this stage.
*Jun 16 19:50:15.122: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 3/0
*Jun 16 19:50:42.374: %SPA_OIR-6-ONLINECARD: SPA (C9400-SUP-1) online in subslot 3/0
C9400#
*Jun 16 19:50:43.376: 3 0 0:Ignore this incremental sync, session not ready
C9400#
*Jun 16 19:52:10.003: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F1
C9400#
*Jun 16 19:51:16.469: %IOSXE-3-PLATFORM: R1/0: kernel: dplr_intrpt: Entered dplr_intrpt_module_init dplr_intrpt 1
*Jun 16 19:51:16.472: %IOSXE-3-PLATFORM: R1/0: kernel: chr_mmap: Allocating DMA Reserve Pool ...
*Jun 16 19:52:27.950: %IOSXE_OIR-6-ONLINECARD: Card (rp) online in slot R1
*Jun 16 19:52:28.727: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: R0/0: auto_upgrade_client: Auto upgrade initiated for RP 1.
*Jun 16 19:52:28.748: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: R0/0: auto_upgrade_client: Searching stack for software to upgrade RP 1.
*Jun 16 19:52:28.760: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: R0/0: auto_upgrade_client: Found donor RP 0 to auto upgrade RP 1.
*Jun 16 19:52:28.773: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: R0/0: auto_upgrade_client: Upgrading RP 1 with software from RP 0.
*Jun 16 19:52:39.655: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*Jun 16 19:52:39.655: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
*Jun 16 19:52:39.642: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: R0/0: auto_upgrade_client: Initiating SMU autoupgrade for RP 1
*Jun 16 19:52:40.832: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: R0/0: auto_upgrade_client: Finished installing software on RP 1.
*Jun 16 19:52:40.847: %AUTO_UPGRADE-5-AUTO_UPGRADE_RELOAD: R0/0: auto_upgrade_client: Reloading RP 1 to complete the auto upgrade.
*Jun 16 19:52:41.622: %IOSXE_OIR-6-OFFLINECARD: Card (rp) offline in slot R1
** snip **
*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
** snip **
*Jun 16 19:57:33.582: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Jun 16 19:57:34.623: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Verify Post-Replacement
Check the state of the supervisors once SSO is completed
C9400#show module
Chassis Type: C9404R
Mod Ports Card Type Model Serial No.
---+---------------------+-------------------------------+-----------------------+--------------------
2 10 Supervisor 1 Module C9400-SUP-1 JAE22100647
3 10 Supervisor 1 Module C9400-SUP-1
Mod MAC addresses Hw Fw Sw Status
---+----------------------------------------+----------+--------------+--------------+------------
2 A8B4.56BF.316C to A8B4.56BF.3175 1.0 16.12.1r 16.09.05 ok
3 ok <--- Started Syncing
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+----------------------+---------------------------------------+------------------------------------------------
2 Active sso sso
3 Standby sso sso
Chassis MAC address range: 44 addresses from a8b4.56bf.3140 to a8b4.56bf.316b
C9400#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 10 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 2
Current Software state = ACTIVE
Uptime in current state = 10 minutes
Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 18:14 by mcpre
BOOT = bootflash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
Peer Processor Information :
----------------------------
Standby Location = slot 3
Current Software state = STANDBY HOT
Uptime in current state = 0 minutes
Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 18:14 by mcpre
BOOT = bootflash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
Replace a Supervisor of C9400 Dual-Sup StackWise-Virtual
This example is for a C9400 stackwise-Virtual setup (one supervisor in each chassis), where chassis-1's supervisor (Active switch) has gone bad and needs to be replaced. The SVL is running in "Install" boot mode.
Verify Pre-Replacement
Check the current StackWise-Virtual related configurations and the state of supervisors. Ensure that boot variable on switch is set correctly pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.
9400-3#show stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 TenGigabitEthernet1/5/0/1 <<< switch 1 needs to be replaced here
2 1 TenGigabitEthernet2/5/0/1
9400-3#show bootvar
BOOT variable = flash:packages.conf;
Configuration Register is 0x102
MANUAL_BOOT variable = no
BAUD variable = 9600
ENABLE_BREAK variable = yes
BOOTMODE variable does not exist
IPXE_TIMEOUT variable does not exist
CONFIG_FILE variable =
9400-3#show run all | in software auto
no software auto-upgrade source url
software auto-upgrade enable
If active supervisor needs to be replaced (like in our example), perform a failover to standby supervisor and wait for standby to take over active's role. Skip this step if you are replacing standby supervisor.
9400-1#redundancy force-switchover System configuration has been modified. Save? [yes/no]: yes Building configuration... Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]
Replace
Power off the chassis where supervisor needs to be replaced (In our example, it is chassis-1).
Remove the line cards from the backplane from respective chassis (where supervisor needs to be replaced). There is no need to take the line cards completely out of the chassis, as long as they are not attached to backplane, it is fine. That way, when new supervisor is inserted and being pre-staged, the remote switches of the connections (Multi-chassis etherchannel) do not put their local ports into err-disabled state (LACP and so on).
9400-3#show module
Chassis Type: C9410R
Switch Number 1
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
Switch Number 2
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 48 48-Port UPOE w/ 24p mGig 24p RJ-45 C9400-LC-48UX JAE2138067S
2 48 48-Port UPOE 10/100/1000 (RJ-45) C9400-LC-48U JAE2141091P
5 10 Supervisor 1 Module C9400-SUP-1 JAE2220082A
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 707D.B9CF.6D1C to 707D.B9CF.6D4B 1.0 16.12.2r 16.12.03a ok
2 6CB2.AE42.2704 to 6CB2.AE42.2733 1.0 16.12.2r 16.12.03a ok
5 AC3A.675B.E26C to AC3A.675B.E275 1.0 16.12.2r 16.12.03a ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
5 Active non-redundant sso
Insert the new supervisor into same slot where faulty supervisor was present and power it on. It must boot up in Stand-alone mode (Non-SVL), leave the Stackwise-virtual links disconnected for now.
- If your active supervisor is running in "Bundle" boot mode, then copy the software bin file (same as current active supervisor of SVL) to bootflash of new standby supervisor and change boot string accordingly.
- If your active supervisor is running in "Install" boot mode, then manual software upgrade is not needed. Software and boot mode of new supervisor must be upgraded automatically by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor.
Configure the new supervisor with Stackwise-virtual settings. (You must use same SVL domain number to match existing member.)
Switch#conf t Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#stackwise-virtual Please reboot the switch for Stackwise Virtual configuration to take effect
Switch(config-stackwise-virtual)#domain 100 Switch(config-stackwise-virtual)#exit
Configure SVL and DAD ports. Use the same ports, that were used on the faulty supervisor.
9400-1(config)#interface tenGigabitEthernet 5/0/1
9400-1(config-if)#stackwise-virtual link 1
WARNING: All the extraneous configurations will be removed for TenGigabitEthernet5/0/1 on reboot
INFO: Upon reboot, the config will be part of running config but not part of start up config.
Check that SVL configuration is applied correctly to the new switch.
Switch#show stackwise-virtual Stackwise Virtual Configuration: -------------------------------- Stackwise Virtual : Disabled Switch Stackwise Virtual Link Ports ------ ---------------------- ------ Stackwise Virtual Configuration After Reboot: --------------------------------------------- Stackwise Virtual : Enabled Domain Number : 100 Switch Stackwise Virtual Link Ports ------ ---------------------- ------ 1 1 TenGigabitEthernet5/0/1
Check the SVL settings in ROMMON from IOSd CLI (available in versions16.12.x or higher)
9400-1#show romvar
ROMMON variables:
MAC_ADDR="70:0F:6A:DE:54:34"
SWITCH_NUMBER="1"
MODEL_NUM="C9400-SUP-1"
SYSTEM_SERIAL_NUM=""
MOTHERBOARD_SERIAL_NUM="JAE221703NQ"
TEMPLATE="access"
BAUD="9600"
LICENSE_BOOT_LEVEL="network-advantage+dna-advantage,all:MACALLAN-CHASSIS;"
MCP_STARTUP_TRACEFLAGS="00000000:00000000"
CALL_HOME_DEBUG="0000000000000"
D_STACK_DAD=""
CONFIG_FILE=""
BOOTLDR=""
SWITCH_IGNORE_STARTUP_CFG="0"
MANUAL_BOOT="no"
AUTOREBOOT_RESTORE="0"
ENABLE_BREAK="yes"
RET_2_RTS=""
AUTO_SWITCH_CONSOLE_DISABLE="0"
BOOT="flash:cat9k_iosxe.16.12.03a.SPA.bin;"
D_STACK_DISTR_STACK_LINK2=""
ABNORMAL_RESET_COUNT="1"
ROMMON_AUTOBOOT_ATTEMPT="3"
BSI="0"
RET_2_RCALTS=""
RANDOM_NUM="421133355"
D_STACK_DISTR_STACK_LINK1="Te5/0/1,"
D_STACK_MODE="aggregation"
D_STACK_DOMAIN_NUM="100"
Save the configurations and power off chassis where new supervisor is placed into.
Connect StackWise-Virtual links between two chassis and prefer to leave the Dual-active detection link disconnected (if applicable).
Power on the chassis and monitor the boot process through console.
- If your SVL is running in "Bundle" boot mode, ensure that new supervisor is coming up with same software version as Active. If not, break into ROMMON again and boot it manually using the correct software version.
- If the SVL is running in "Install" boot mode, then "software auto-upgrade" must take care of pushing right software version and boot mode into new supervisor, without any manual intervention.
Active supervisor's log
*Sep 12 07:20:25.457: %ILPOWER-6-SET_ILPOWER: Set power allocated to POE to 4420 for slot 0
*Sep 12 07:20:30.621: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Chassis 2 R0/0: issu_stack: Incompatible software detected. Details: Active's subpackage boot mode does not match with member's super boot mode. Please boot switch 1 in subpackage mode.
*Sep 12 07:20:40.779: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.
*Sep 12 07:21:00.978: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade initiated for switch 1.
*Sep 12 07:21:01.031: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Chassis 2 R0/0: auto_upgrade_client: Searching stack for software to upgrade switch 1.
*Sep 12 07:21:01.053: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Chassis 2 R0/0: auto_upgrade_client: Found donor switch 2 to auto upgrade switch 1.
*Sep 12 07:21:01.074: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Chassis 2 R0/0: auto_upgrade_client: Upgrading switch 1 with software from switch 2.
Logs from new supervisor's console
Waiting for remote chassis to join
#########################################
Chassis number is 1
All chassis in the stack have been discovered. Accelerating discovery
Chassis 1 reloading, reason - System requested reload <<< reload is instructed by current active as part of software auto-upgrade
Sep 12 07:25:23.306: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload fru code
All the running configurations must be automatically synced from Active supervisor to the new one. Wait for these logs from active supervisor.
*Sep 12 07:33:39.803: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Sep 12 07:33:40.837: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
- Once SSO is completed, proceed with connecting the Dual-active detection (DAD) link and other network uplink ports on new supervisor, as applicable.
- Push the line cards back inside, to get them attached to backplane again
- Verify that all line cards have booted fine, passed online diagnostic tests and brought up their interfaces, including port-channel binding and so on.
Verify Post-Replacement
Check the StackWise Virtual related configurations and the states of the switch using these commands.
9400-3#sh redundancy
Redundant System Information :
------------------------------
Available system uptime = 1 hour, 31 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = Switch 2
Current Software state = ACTIVE
Uptime in current state = 31 minutes
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.3a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 28-Apr-20 09:37 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
Peer Processor Information :
----------------------------
Standby Location = Switch 1
Current Software state = STANDBY HOT
Uptime in current state = 4 minutes
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.3a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Tue 28-Apr-20 09:37 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
!
9400-3#sh stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 TenGigabitEthernet1/5/0/1
2 1 TenGigabitEthernet2/5/0/1
9400-3#sh module
Chassis Type: C9410R
Switch Number 1
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 48 48-Port UPOE w/ 24p mGig 24p RJ-45 C9400-LC-48UX JAE22360153
2 48 48-Port UPOE w/ 24p mGig 24p RJ-45 C9400-LC-48UX JAE215103V7
5 10 Supervisor 1 Module C9400-SUP-1 JAE221703NQ
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 00B7.71FA.D878 to 00B7.71FA.D8A7 1.0 16.12.2r 16.12.03a ok
2 4C77.6DBF.4A94 to 4C77.6DBF.4AC3 1.0 16.12.2r 16.12.03a ok
5 AC3A.675B.E9AC to AC3A.675B.E9B5 1.0 16.12.2r 16.12.03a ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
5 Standby sso sso
**snip**
Replace a Member of the C9500 StackWise-Virtual
In this example, you are considering a C9500 Stackwise Virtual setup with Switch-1 (the Active switch) as a faulty switch that needs to be replaced. The SVL is running in INSTALL boot mode.
Verify Pre-Replacement
Check the current StackWise-Virtual related configurations and the state of the switches. Ensure that boot variable is set correctly, points to packages.conf, and config-register is set to 0x2102.
C9500-1#show stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 TwentyFiveGigE1/0/1
TwentyFiveGigE1/0/2
2 1 TwentyFiveGigE2/0/1
TwentyFiveGigE2/0/2
C9500-1#show stackwise-virtual dual-active-detection Dual-Active-Detection Configuration: ------------------------------------- Switch Dad port ------ ------------ 1 TwentyFiveGigE1/0/3 2 TwentyFiveGigE2/0/3 <<<<<<<<<< Ports configured for Dual-Active Detection (DAD)
Note : Configs of these DAD ports do not show up in running-config
!
interface TwentyFiveGigE 1/0/3
end
!
interface TwentyFiveGigE 2/0/3
end
C9500-1#show switch
Switch/Stack Mac Address : f4db.e619.0480 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active f4db.e619.0480 15 V02 Ready
2 Standby f4db.e618.fa80 1 V02 Ready
C9500-1#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 4 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 4 minutes
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 19-Nov-19 10:04 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
Peer Processor Information :
----------------------------
Standby Location = slot 2
Current Software state = STANDBY HOT
Uptime in current state = 1 minute
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 19-Nov-19 10:04 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
C9500-1#show run all | in software auto
no software auto-upgrade source url
software auto-upgrade enable
If active switch needs to be replaced, perform a failover to Standby switch and wait for standby to take over active's role. (Skip this step if you are replacing Standby unit).
C9500-1#redundancy force-switchover System configuration has been modified. Save? [yes/no]: yes Building configuration... Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]
Replace
Power off the Switch that needs to be replaced. Disconnect all the cables from that switch.
C9500-1#show switch
Switch/Stack Mac Address : f4db.e619.0480 - Foreign Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
1 Member 0000.0000.0000 0 V02 Removed << switch 1 is powered down
*2 Active f4db.e618.fa80 1 V02 Ready
Power on the new switch. It must boot up in Stand-alone mode (Non-SVL). (Skip this step if the current Active SVL is running in INSTALL boot mode)
Check the software version on new unit. If it does not match with existing member of StackWise-Virtual unit, then pre-stage that to match the software version and license, with existing member of SVL. (You can load the correct software version through TFTP/FTP/SFTP or using an USB stick, and after matching the software version and license on new unit, proceed to next step.
Cisco IOS XE Software, Version 16.12.02
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 19-Nov-19 10:04 by mcpre
Configure StackWise Virtual on the new switch. You must use same SVL domain number to match existing member.
Switch#conf t Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#stackwise-virtual Please reboot the switch for Stackwise Virtual configuration to take effect Switch(config-stackwise-virtual)#domain 100 Switch(config-stackwise-virtual)#exit
Configure SVL and DAD ports. Usesame ports, that were used on the faulty switch.
Switch(config)#int range twe1/0/1-2 Switch(config-if-range)#stackwise-virtual link 1 WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/1 on reboot WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/2 on reboot Switch(config-if-range)#exit Switch(config)#int twe1/0/3 Switch(config-if)#stackwise-virtual dual-active-detection WARNING: All the extraneous configurations will be removed for TwentyFiveGigE1/0/3 on reboot.
Check that SVL configuration is applied correctly to the new switch.
Switch#show stackwise-virtual Stackwise Virtual Configuration: -------------------------------- Stackwise Virtual : Disabled Switch Stackwise Virtual Link Ports ------ ---------------------- ------ Stackwise Virtual Configuration After Reboot: --------------------------------------------- Stackwise Virtual : Enabled Domain Number : 100 Switch Stackwise Virtual Link Ports ------ ---------------------- ------ 1 1 TwentyFiveGigE1/0/1
TwentyFiveGigE1/0/2 Switch#show stackwise-virtual dual-active-detection Dual-Active-Detection Configuration: ------------------------------------- Switch Dad port ------ ------------ Distributed Stack DAD Configuration After Reboot: ------------------------------------------------ Switch Dad port ------ -------- 1 TwentyFiveGigE1/0/3
Save the configurations and power off the new switch.
Connect StackWise-Virtual links between existing SVL member and the new unit. Prefer to leave the Dual-active detection link disconnected.
Power on the new unit. If there is a conflict in switch number, then new unit must be re-numbered automatically.
Chassis is reloading, reason: Configured Switch num conflicts with peer, Changing local switch number to 1 and reloading to take effect
Sep 10 22:41:50.738: %PMAN-3-PROCHOLDDOWN: R0/0: The process nif_mgr has been helddown (rc 69)
*Sep 10 22:47:05.996: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.
All the running configurations are automatically synced from Active switch to the New switch. No additional configurations are needed. Wait for these logs from active switch.
*Sep 11 01:02:28.974: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
C9500-1#
*Sep 11 01:02:30.009: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Proceed with connecting the Dual-active detection (DAD) link and other network ports. (Once SSO is completed)
Verify Post-Replacement
Check the StackWise Virtual related configurations and the states of the switch using these commands.
C9500-1#show stackwise-virtual
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 TwentyFiveGigE1/0/1
TwentyFiveGigE1/0/2
2 1 TwentyFiveGigE2/0/1
TwentyFiveGigE2/0/2
C9500-1#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 14 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 2
Current Software state = ACTIVE
Uptime in current state = 14 minutes
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 19-Nov-19 10:04 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
Peer Processor Information :
----------------------------
Standby Location = slot 1
Current Software state = STANDBY HOT
Uptime in current state = 1 minute
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 19-Nov-19 10:04 by mcpre
BOOT = flash:packages.conf;
CONFIG_FILE =
Configuration register = 0x102
Replace a Redundant Supervisor of C9600 Dual-Sup Standalone Chassis
In this example, you are considering to replace the active supervisor at slot 3 of a C9606 chassis. (The switch is running in "Install" boot mode.)
Verify Pre-Replacement
Check that boot variable on switch is set correctly pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.
C9600R-1#show run all | in software auto
no software auto-upgrade source url
software auto-upgrade enable
Replace
If active supervisor needs to be replaced (like in this example of ours), perform a failover to standby supervisor and wait for it to take over active's role. (Skip this step if you are going to replace standby supervisor).
C9600R-1#redundancy force-switchover System configuration has been modified. Save? [yes/no]: yes Building configuration... Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]
Remove faulty supervisor from chassis and insert the new one, with a console cable plugged into it.
If your active supervisor is running in "Bundle" boot mode, then break into the ROMMON of new supervisor, while it is booting up. With help of an USB stick or OOB TFTP access, manually boot the supervisor with same software version as your active supervisor. Later on, after new standby joins SSO, copy the running software to it's local bootflash.
Preparing to autoboot. [Press Ctrl-C to interrupt] 3 (interrupted)
rommon 1 >
rommon 2 > boot disk0:cat9k_iosxe.16.12.04.SPA.bin
If your active supervisor is running in "Install" boot mode, automatic software upgrade must be kicked in by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor. Typically, no manual intervention is needed at this stage.
*Sep 12 21:32:04.886: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*Sep 12 21:32:04.886: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
*Sep 12 21:32:07.773: %REDUNDANCY-2-IPC: IOS versions do not match.
*Sep 12 21:32:07.823: %SMART_LIC-5-EVAL_START: Entering evaluation period
*Sep 12 21:32:28.980: %AUTO_UPGRADE_MODULAR-5-SMU_AUTO_UPGRADE_INITIATING: R1/0: auto_upgrade_client: Initiating SMU autoupgrade for RP 0
*Sep 12 21:32:30.867: %AUTO_UPGRADE-5-AUTO_UPGRADE_FINISH: R1/0: auto_upgrade_client: Finished installing software on RP 0.
*Sep 12 21:32:30.908: %AUTO_UPGRADE-5-AUTO_UPGRADE_RELOAD: R1/0: auto_upgrade_client: Reloading RP 0 to complete the auto upgrade.
** snip **
*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*Jun 16 19:56:10.356: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
** snip **
*Sep 12 21:36:37.786: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
*Sep 12 21:36:37.786: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
**snip**
*Sep 12 21:39:24.085: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Sep 12 21:39:25.124: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Verify Post-Replacement
Check the state of the supervisors once SSO is completed
C9606R-1#show mod
Chassis Type: C9606R
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2313L2WQ
2 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2314L36W
3 0 Supervisor 1 Module C9600-SUP-1 CAT2310L5C1
4 0 Supervisor 1 Module C9600-SUP-1 CAT2311L4DQ
5 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2310L57N
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 DC8C.37C9.AC00 to DC8C.37C9.AC7F 1.0 17.1.1[FC2] 16.12.04 ok
2 DC8C.37C9.FD00 to DC8C.37C9.FD7F 1.0 17.1.1[FC2] 16.12.04 ok
3 DC8C.3772.C780 to DC8C.3772.C7FF 1.0 17.1.1[FC2] 16.12.04 ok
4 DC8C.3772.E580 to DC8C.3772.E5FF 1.0 17.1.1[FC2] 16.12.04 ok
5 DC8C.3773.0280 to DC8C.3773.02FF 1.0 17.1.1[FC2] 16.12.04 ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Standby sso sso
4 Active sso sso
Chassis MAC address range: 64 addresses from 6cb2.ae4a.9680 to 6cb2.ae4a.96bf
C9606R-1#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 1 day, 11 hours, 32 minutes
Switchovers system experienced = 1
Standby failures = 1
Last switchover reason = user forced
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = slot 4
Current Software state = ACTIVE
Uptime in current state = 35 minutes
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 09-Jul-20 21:49 by mcpre
BOOT =
CONFIG_FILE =
Peer Processor Information :
----------------------------
Standby Location = slot 3
Current Software state = STANDBY HOT
Uptime in current state = 3 minutes
Image Version = Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 09-Jul-20 21:49 by mcpre
BOOT =
CONFIG_FILE =
Replace a Supervisor of C9600 Dual-Sup StackWise-Virtual
In this example, you are considering a C9600 stackwise-Virtual setup (one supervisor in each chassis), where chassis-1's supervisor (Active switch) has gone bad and needs to be replaced. The SVL is running in "Install" boot mode.
Verify Pre-Replacement
Check the current StackWise-Virtual related configurations and the state of supervisors. Ensure that boot variable on switch is set correctly pointing to correct package file (if boot mode is Install) or bin file (bundle boot mode) and auto-boot is enabled.
C9600_SVL#sh stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
2 1 FortyGigabitEthernet2/1/0/1
FortyGigabitEthernet2/1/0/2
1 1 FortyGigabitEthernet1/1/0/1 << supervisor of SW1 needs to be replaced
FortyGigabitEthernet1/1/0/2
C9600_SVL#show bootvar
BOOT variable = bootflash:packages.conf;
MANUAL_BOOT variable = no
BAUD variable = 9600
ENABLE_BREAK variable = yes
BOOTMODE variable does not exist
IPXE_TIMEOUT variable does not exist
CONFIG_FILE variable =
C9600_SVL#show run all | in software auto
no software auto-upgrade source url
software auto-upgrade enable
If your active supervisor is running in "Bundle" boot mode, please keep a copy of running software file (.bin file that you are running on active) in an USB stick or local TFTP server which can be accessed from new supervisor, through it's out-of-band (OOB) management port.
- If active supervisor needs to be replaced (like in our example), perform a failover to standby supervisor and wait for standby to take over active's role. Skip this step if you are replacing standby supervisor.
C9600_SVL#redundancy force-switchover System configuration has been modified. Save? [yes/no]: yes Building configuration... Compressed configuration from 11673 bytes to 4403 bytes[OK]Proceed with switchover to standby RP? [confirm]
Replace
Power off the chassis where supervisor needs to be replaced. In our example, it is chassis-1.
Remove the line cards from the backplane from respective chassis (where supervisor needs to be replaced), except one where StackWise-Virtual links (SVL) are attached to. For those line cards, where SVLs are configured, remove all connections except the SVLs itself. That way, when new supervisor is inserted and being pre-staged, the remote switches of the connections (Multi-chassis etherchannel) do not put their local ports into err-disabled state (LACP and so on).
C9600_SVL#show module
Chassis Type: C9606R
Switch Number 1
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
Switch Number 2
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2310L4DW
2 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2310L59S
3 0 Supervisor 1 Module C9600-SUP-1 CAT2340L40Q
5 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2313L2W1
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 DC8C.379F.DB80 to DC8C.379F.DBFF 1.0 17.3.1r[FC2] 17.03.01 ok
2 DC8C.3772.FD80 to DC8C.3772.FDFF 1.0 17.3.1r[FC2] 17.03.01 ok
3 7C21.0E5D.0800 to 7C21.0E5D.087F 1.0 17.3.1r[FC2] 17.03.01 ok
5 DC8C.37A0.D180 to DC8C.37A0.D1FF 1.0 17.3.1r[FC2] 17.03.01 ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Active non-redundant sso
Chassis 2 MAC address range: 64 addresses from 2c4f.523b.bd00 to 2c4f.523b.bd3f
Insert the new supervisor into same slot where faulty supervisor was present and power it on. It must boot up in Stand-alone mode (Non-SVL), leave the Stackwise-virtual links and DAD links disconnected for now.
If your active supervisor is running in "Bundle" boot mode, then copy the software bin file (same as current active supervisor of SVL) to bootflash of new standby supervisor and change bootstring accordingly.
If your active supervisor is running in "Install" boot mode, then manual software upgrade is not needed. Software and boot mode of new supervisor must be upgraded automatically by current active supervisor, as soon it detects an incompatible software version or boot mode on new/standby supervisor.
Configure the new supervisor with Stackwise-virtual settings. (You must use same SVL domain number to match existing member).
Switch#conf t Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#stackwise-virtual Please reboot the switch for Stackwise Virtual configuration to take effect Switch(config-stackwise-virtual)#domain 100 Switch(config-stackwise-virtual)#exit
Configure SVL and DAD ports. Usesame ports, that were used on the faulty supervisor.
Switch(config)#int range fortyGigabitEthernet 1/0/1 -2
Switch(config-if-range)#stackwise-virtual link 1
Switch(config)#int range twentyFiveGigE 2/0/25 -26
Switch(config-if-range)#stackwise-virtual dual-active-detection
Check that SVL configuration is applied correctly to the new switch.
Switch#show stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual Configuration After Reboot:
---------------------------------------------
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 FortyGigabitEthernet1/0/1
FortyGigabitEthernet1/0/2
Switch#show stackwise-virtual dual-active-detection
In dual-active recovery mode: No
Dual-Active-Detection Configuration:
-------------------------------------
Switch Dad port Status
------ ------------ ---------
Distributed Stack DAD Configuration After Reboot:
------------------------------------------------
Switch Dad port Status
------ -------- ------
1 TwentyFiveGigE2/0/25 down
TwentyFiveGigE2/0/26 down
If software version is 16.12.x or higher, you can check the SVL settings in ROMMON from IOSd CLI.
Switch#show romvar
ROMMON variables:
BOARDID="38"
ETHER_PORT="2"
PS1="rommon ! >"
MAC_ADDR="7C:21:0E:5D:04:00"
DOPPLER_E_WA="1"
RETRY="0"
MODEL_NUM="C9600-SUP-1"
SYSTEM_SERIAL_NUM="CAT2340L3Y5"
MOTHERBOARD_SERIAL_NUM="CAT2340L3Y5"
TEMPLATE="core"
BAUD="9600"
AUTO_SWITCH_CONSOLE_DISABLE="0"
PSEUDO_OIR_REMOVE_SET="1"
CALL_HOME_DEBUG="0000000000000"
ENABLE_BREAK="yes"
RET_2_RTS=""
CRASHINFO="bootflash:crashinfo_RP_00_00_20200225-024401-UTC"
MCP_STARTUP_TRACEFLAGS="00000000:00000000"
CONFIG_FILE=""
BOOTLDR=""
RECOVERY_RELOAD_DISABLE=""
SWITCH_PRIORITY="1"
SWITCH_NUMBER="1"
SWITCH_IGNORE_STARTUP_CFG="0"
D_STACK_DISTR_STACK_LINK2=""
MANUAL_BOOT="no"
AUTOREBOOT_RESTORE="0"
ABNORMAL_RESET_COUNT="0"
ROMMON_AUTOBOOT_ATTEMPT="3"
BSI="0"
RET_2_RCALTS=""
RANDOM_NUM="1430571596"
BOOT="bootflash:cat9k_iosxe.16.12.02.SPA.bin;"
D_STACK_DISTR_STACK_LINK1="Fo1/0/1,Fo1/0/2,"
D_STACK_DAD="Twe2/0/25,Twe2/0/26,"
D_STACK_MODE="aggregation"
D_STACK_DOMAIN_NUM="100"
Save the configurations and power off chassis where new supervisor is placed into.
Connect StackWise-Virtual links between two chassis and prefer to leave the Dual-active detection link disconnected (if applicable).
Power on the chassis and monitor the boot process through console.
Note: If your SVL is running in "Bundle" boot mode, please ensure that new supervisor is coming up with same software version as Active. If not, break into ROMMON again and boot it manually using the correct software version.
Active supervisor's log-
*Sep 13 00:59:49.367: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 13 00:59:51.988: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 13 00:59:52.135: %BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: Chassis 2 R0/0: issu_stack: Incompatible software detected. Details: Active's subpackage boot mode does not match with member's super boot mode. Please boot switch 1 in subpackage mode.
*Sep 13 00:59:52.297: %AUTO_UPGRADE-5-AUTO_UPGRADE_START_CHECK: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade start checking for incompatible switches.
*Sep 13 00:59:53.311: %AUTO_UPGRADE-5-AUTO_UPGRADE_INITIATED: Chassis 2 R0/0: auto_upgrade_client: Auto upgrade initiated for switch 1.
*Sep 13 00:59:53.368: %AUTO_UPGRADE-5-AUTO_UPGRADE_SEARCH: Chassis 2 R0/0: auto_upgrade_client: Searching stack for software to upgrade switch 1.
*Sep 13 00:59:53.397: %AUTO_UPGRADE-5-AUTO_UPGRADE_FOUND: Chassis 2 R0/0: auto_upgrade_client: Found donor switch 2 to auto upgrade switch 1.
*Sep 13 00:59:53.423: %AUTO_UPGRADE-5-AUTO_UPGRADE_START: Chassis 2 R0/0: auto_upgrade_client: Upgrading switch 1 with software from switch 2.
Logs from new supervisor's console-
Waiting for remote chassis to join
##################################################################################
Chassis number is 1
All chassis in the stack have been discovered. Accelerating discovery
Chassis 1 reloading, reason - System requested reload <<< reload is instructed by current active as part of software auto-upgrade
All the running configurations are automatically synced from Active supervisor to the new one. Wait for these logs from active supervisor.
*Sep 13 01:14:18.552: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Sep 13 01:14:18.577: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Once SSO is completed, proceed with connecting the Dual-active detection (DAD) link as well.
- Push the line cards back inside (for chassis where supervisor is replaced), to get those attached to backplane again. Now, reconnect the cables.
- Verify that all line cards have booted fine, passed online diagnostic tests and brought up their interfaces, including port-channel binding and so on.
Verify Post-Replacement
Check the StackWise Virtual related configurations and the states of the switch using these commands.
C9600_SVL#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 1 hour, 27 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none
Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up
Current Processor Information :
-------------------------------
Active Location = Switch 2
Current Software state = ACTIVE
Uptime in current state = 1 hour, 27 minutes
Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Fri 07-Aug-20 21:32 by mcpre
BOOT = bootflash:packages.conf;
CONFIG_FILE =
Peer Processor Information :
----------------------------
Standby Location = Switch 1
Current Software state = STANDBY HOT
Uptime in current state = 0 minutes
Image Version = Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.1, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Fri 07-Aug-20 21:32 by mcpre
BOOT = bootflash:packages.conf;
CONFIG_FILE =
C9600_SVL#show stackwise-virtual
Stackwise Virtual Configuration:
--------------------------------
Stackwise Virtual : Enabled
Domain Number : 100
Switch Stackwise Virtual Link Ports
------ ---------------------- ------
1 1 FortyGigabitEthernet1/1/0/1
FortyGigabitEthernet1/1/0/2
2 1 FortyGigabitEthernet2/1/0/1
FortyGigabitEthernet2/1/0/2
C9600_SVL#show stackwise-virtual dual-active-detection
In dual-active recovery mode: No
Recovery Reload: Enabled
Dual-Active-Detection Configuration:
-------------------------------------
Switch Dad port Status
------ ------------ ---------
1 TwentyFiveGigE1/2/0/25 up
TwentyFiveGigE1/2/0/26 up
2 TwentyFiveGigE2/2/0/25 up
TwentyFiveGigE2/2/0/26 up
C9600_SVL#show module
Chassis Type: C9606R
Switch Number 1
Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+--------------+--------------
1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2252L0PR
2 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2334L0BA
3 0 Supervisor 1 Module C9600-SUP-1 CAT2340L3Y5
5 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2337L509
Mod MAC addresses Hw Fw Sw Status
---+--------------------------------+----+------------+------------------+--------
1 70B3.175A.8100 to 70B3.175A.817F 1.0 17.3.1r[FC2] 17.03.01 ok
2 10B3.D652.9900 to 10B3.D652.997F 1.0 17.3.1r[FC2] 17.03.01 ok
3 7C21.0E5D.0400 to 7C21.0E5D.047F 1.0 17.3.1r[FC2] 17.03.01 ok
5 4C71.0D7C.8400 to 4C71.0D7C.847F 1.0 17.3.1r[FC2] 17.03.01 ok
Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode
---+-------------------+-------------------------+---------------------------
3 Standby sso sso
Switch Number 2
**snip**
Replace a Supervisor of C9600 Quad-Sup StackWise-Virtual
In this example, you are considering a C9600 Quad Sup stackwise-Virtual setup (two supervisors in each chassis), where a supervisor has gone bad and needs to be replaced. The SVL is running in "Install" boot mode.
Replace and Verify
Pulling out the faulty supervisor
- If the supervisor to be replaced isGlobal Active supervisor (Sw-1 Slot 3 in the image shown), perform a failover so that the Global Standby (Sw-2 Slot 3 in the previous picture) takes over as Active. Wait until there is a new Global Standby and SSO gets completed. (In this case, Sw-1 Slot 4 to become new Global Standby).
- If the supervisor to be replaced isGlobal Standby supervisor (Sw-2 Slot 3 in the image shown), pull the supervisor out. Wait until there is a new Global Standby and SSO gets completed. (In this case, Sw-2 Slot 4 to become new Global Standby)
- If the supervisor to be replaced isICS supervisor(Sw-1 Slot 4 or Sw-2 Slot 4 in the image shown), pull the supervisor out.
Inserting the new supervisor
- If the new supervisor is running on 17.x code, the steps are straight forward. Just insert the new supervisor. If ICS supervisors have 17.x image, they automatically boot and become part of Quad-sup. Even if it is running different 17.x code than that of what is running one on the production setup, Software Auto-upgrade to automatically take care of upgrading the ICS supervisor with the same 17.x code in INSTALL mode.
- If the new supervisor is running on 16.x code, or if you are unsure of the code it is running on, try to insert the supervisor on a spare chassis and get it upgraded to 17.x code. If there is no spare chassis to upgrade, these steps have to be taken.
- This step is very important. Insert the ICS supervisor and use Ctrl+C to break them to rommon.If you miss to break into ROMMON and supervisor boots up on 16.x code, this could take down the complete chassis where the supervisor was inserted
Look for any SVL related rommon variables. These variables start with D_STACK. Usually a new supervisor do not have these variables set.
D_STACK_DISTR_STACK_LINK2=""
D_STACK_DAD="Fo1/0/13,Fo1/0/15,"
D_STACK_MODE="aggregation"
D_STACK_DOMAIN_NUM="255"
D_STACK_DISTR_STACK_LINK1="Fo1/0/10,Fo1/0/15,Fo1/0/16,Fo1/0/17,Fo1/0/3,Fo1/0/6,"
Unset all the previously shown variables
rommon 1 > unset D_STACK_DAD
rommon 1 > unset D_STACK_DISTR_STACK_LINK1
rommon 1 > unset D_STACK_DOMAIN_NUM rommon 1 > unset D_STACK_MODE
Look for variable "SWITCH_NUMBER=1". If the switch number is 2, set the variable to 1. If it is already 1, move to the next step.
rommon 1 > SWITCH_NUMBER=1
Set to manual boot the supervisor.
rommon 1 > MANUAL_BOOT=YES
Manually boot the ICS supervisor in bundle mode using USB/TFTP on 17.x code. Do not change boot variable in rommon. Just boot it manually from rommon.
- The supervisor is reset as it detects an existing ICS in SVL mode, so it gets converted from stand-alone to stackwise virtual mode. It could again fall back into rommon, since auto-boot is disabled.
Unset manual boot to enable auto-boot.
rommon 1 > unset MANUAL_BOOT
Manually boot the ICS supervisor in bundle mode using USB/TFTP on 17.x code. Do not change boot variable in rommon. Just boot it manually from rommon. This step boots the ICS in BUNDLE mode.
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
4.0 |
31-Aug-2023 |
Cleaned up some style issue to improve readability |
3.0 |
07-Jun-2023 |
Fixed an error in a note: "The new unit cannot be running same software version as existing stack" |
2.0 |
05-May-2023 |
Added Alt Text.
Updated Title, Introduction, SEO, Branding Requirements, Machine Translation, Style Requirements, Gerunds and Formatting. |
1.0 |
04-Nov-2020 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)