Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Before You Get Started

Insert Secondary Supervisor

Secondary With 10.4(x) Release

Same Release Image

Different Release Images

Secondary With 10.3(x) Release

Image Sync

BIOS Version

Force Download from Active

EOBC Boot Secondary Supervisor

Upgrade BIOS of Standby Supervisor

Upgrade EPLD of Supervisors

Summary

Introduction

This document describes how to migrate to Dual Supervisors for the Nexus 9800 modular switches.

Prerequisites

Requirements

Console access to both Supervisors is necessary.

Cisco recommends that you understand the basics of Upgrading Nexus 9000 switches.

For more information, refer to the documentation here:

Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide, Release 10.4(x)

Cisco recommends that you understand the basics of High Availability for Nexus 9000 switches.

For more information, refer to the documentation here:

Cisco Nexus 9000 Series NX-OS High Availability and Redundancy Guide, Release 10.4(x)

Components Used

The information in this document is based on Nexus 9808 running NX-OS software release 10.4(3)F with two N9K-C9800-SUP-A Supervisor Modules.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The Cisco Nexus 9000 Series switches are protected from the impact of hardware or software failures by redundant hardware components.

Dual supervisors provide redundancy for the control and management plane. The two supervisors operate in an active/standby capacity in which only one of the supervisor modules is active at any given time, while the other acts as a standby backup. The two supervisors constantly synchronize the state and configuration to provide a seamless and stateful switchover of most services if the active supervisor module fails.

Support for Dual Supervisors was introduced for the Nexus 9808 switches in 10.4(1)F and the Nexus 9804 switches in 10.4(2)F respectively.

Early platform deployments can contain a single supervisor initially and transition to dual supervisors.

It is important to understand how to correctly migrate to Dual Supervisors on this platform.

The example outputs referenced throughout this document are from a Nexus 9808 running NX-OS software release 10.4(3)F.

This content is also applicable to a Nexus 9804.

In this document the terms Secondary Supervisor and Standby Supervisor are used.

Secondary Supervisor refers to the second supervisor physically installed into the chassis but has not necessarily synced with the Active Supervisor.

Standby Supervisor refers to the second supervisor physically installed into the chassis and has synced with the Active Supervisor in the HA-Standby state.

Before You Get Started

Firstly, upgrade your Nexus 9800 switch with a single supervisor to a NX-OS software release version with dual supervisor support as mentioned in the previous section.

For deploying features only supported in 10.4(x), Cisco recommends upgrading to the latest available release of 10.4(x) referenced by the Recommended Cisco NX-OS Releases for Cisco Nexus 9000 Series Switches.

Insert Secondary Supervisor

Proceed with inserting Secondary Supervisor into supervisor slot 28 on the Nexus 9800.

Insert console cable into Secondary Supervisor and open a terminal session to monitor the boot process.

Secondary With 10.4(x) Release

A Secondary Supervisor with a 10.4(x) release pre-installed will recognize itself as the Standby Supervisor and slot 27 supervisor as the Active Supervisor.

The Secondary Supervisor will begin to sync with the Active Supervisor starting with their boot images.

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

disable watchdog

Filesystem type unknown, partition type 0x83

check_bootmode: grub: Continue grub

Trying to read config file /boot/grub/menu.lst.local from (hd0,4)

Filesystem type is ext2fs, partition type 0x83

Booting bootflash:/nxos64-cs.10.4.3.F.bin ...

Booting bootflash:/nxos64-cs.10.4.3.F.bin                  <<<

Trying diskboot

Filesystem type is ext2fs, partition type 0x83

Image valid

Image Signature verification was Successful.

Boot Time: 8/5/2024 17:41:4

mount: overlay mounted on /newroot/usr.

Installing klm_card_index

done

Setting nativeboot

Linking n9k mando SSD partition...

creating flash devices BOOT_DEV= sda

INIT: version 2.88 booting

Installing ata_piix module ... done.

Installing kheaders module ... done.

Unsquashing rootfs ...

Total size needed in bootflash is 158780

check bootflash : OK

Total size needed in bootflash is 55644

check bootflash : OK

Enabling 8250 serial driver spurious INTs workaround

Installing isan procfs ... done.

is_lxc: is_titan_eor: is_stby: suffix: klm_ftrace: /isanboot/lib/modules/klm_ftrace.o

Installing ftrace in non-lxc mode done

Installing SSE module with card index 21213 ... done.

Creating SSE device node 243 ... done.

Executing /etc/rc.d/rcS.d/S06exablaze start

Executing /etc/rc.d/rcS.d/S06exablaze start

MKR mando set cpa libs

MANDO RP CPA setup done

Loading I2C driver ... done.

Installing CCTRL driver for card_type 132 without NEED_GE[ 16.676681] ENABLING TOP LEVEL RP INTERRUPT

M ... [ 16.682862] 28 Stdby waiting for 27 to become active           <<<

[ 18.689037] Now Continuing..

done.

Same Release Image

If the boot image for both supervisors is the same, the Secondary Supervisor will complete boot process and move to establish High Availability standby state (ha-standby) with the Active Supervisor.

User Access Verification

N9K-C9808(standby) login: 2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 16.676681] ENABLING TOP LEVEL RP INTERRUPT - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 16.682862] 28 Stdby waiting for 27 to become active - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 18.689037] Now Continuing.. - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 53.624182] Initializing panic NMI - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 53.656637] fpga_setup_probe - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 85.571069] Initializing delayed work - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 85.575323] Finished INIT function for lfm - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 94.522919] MANDO RDN : register_rdn_int - kernel

2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 94.527406] register_rdn_int : Sent First Heartbeat - kernel

2024 Aug 5 17:43:18 stark %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> logflash: online - usbhsd

2024 Aug 5 17:43:43 stark %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Installing thirdparty RPMs - /thirdparty_install.py

2024 Aug 5 17:43:46 stark %$ VDC-1 %$ %USER-2-SYSTEM_MSG: ssnmgr_app_init called on ssnmgr up - aclmgr

2024 Aug 5 17:43:47 stark %$ VDC-1 %$ %USER-1-SYSTEM_MSG: VP aclqos tah stats get - pltfm_config

2024 Aug 5 17:44:11 stark %$ VDC-1 %$ %USER-1-SYSTEM_MSG: ASSERT@../utils/usd/libusd/usd_drv.c:3332 - s1hal_supe

2024 Aug 5 17:44:16 stark %$ VDC-1 %$ %USER-1-SYSTEM_MSG: ASSERT@../utils/usd/libusd/usd_drv.c:3332 - s1hal_supe (message repeated 1 time)

2024 Aug 5 17:44:16 stark %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp

2024 Aug 5 17:44:36 stark %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary

2024 Aug 5 17:44:36 stark %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: TMFPGA booted from Primary

N9K-C9808# show module

Mod Ports Module-Type Model Status

--- ----- ------------------------------------------------ --------------------- --------

27 0 Supervisor Module N9K-C9800-SUP-A active *

28 0 Supervisor Module N9K-C9800-SUP-A ha-standby <<<

Different Release Images

If the boot image for both supervisors is different 10.4(x) releases, the Secondary Supervisor will be rebooted to force NetBoot and download the Active Supervisor's boot image to the Secondary Supervisor.

In the this example, our Active Supervisor is running 10.4(3)F but the Secondary Supervisor booted to 10.4(1)F.

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

disable watchdog

Filesystem type unknown, partition type 0x83

check_bootmode: grub: Continue grub

Trying to read config file /boot/grub/menu.lst.local from (hd0,4)

Filesystem type is ext2fs, partition type 0x83

Booting bootflash:/nxos64-cs.10.4.1.F.bin ...

Booting bootflash:/nxos64-cs.10.4.1.F.bin     <<<

Trying diskboot

Filesystem type is ext2fs, partition type 0x83

Image valid

<...>

[ 93.294604] MANDO RDN : register_rdn_int

[ 93.299095] register_rdn_int : Sent First Heartbeat

< Reboots into NetBoot process >

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

disable watchdog

Finding driver for NIC vendor 8086 Device 15ab

i210 devide id = 1537

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

x check driver for NIC vendor 8086 Device 15ab

inside driver for NIC vendor 8086 Device 15ab

pci_base_addres 0= b2c0000c ioaddr= 0

mask issue

Found the device 8086:15ab at ioaddr 1000, membase b2c00000 at 4:0

Probing...

intelx probe

eobc MAC address 00000000:00000000:00000000:0000001c:00000000:00000000

board_type=5

network is ready.

bp_op = 1

bp_htype = 1

bp_hlen = 6

bp_xid = -160883712

bp_hwaddr = 00:00:00:1C:00:00

bp_hops = 0

bp_secs = 2132726648

*retry 1 currticks 27126 timeout = 720

*retry 2 currticks 27684 timeout = 720

bp_op = 1

bp_htype = 1

bp_hlen = 6

bp_xid = -160883712

bp_hwaddr = 00:00:00:1C:00:00

bp_hops = 0

bp_secs = 2132726648

errnum = 0

reqretry = 0

*2 retry 2 currticks 27684 timeout = 720

Filename: /img-sync/curr-ks.img

board_type=5

board_type=5

Address: 127.1.1.28

Netmask: 255.255.0.0

Server: 127.1.1.27

Gateway: 0.0.0.0

Filesystem type is tftp, using whole disk

.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Image valid

Image Signature verification was Successful.

Once the download has completed, the Secondary Supervisor will boot to the new image then move to establish ha-standby state with the Active Supervisor.

N9K-C9808# show module

Mod Ports Module-Type Model Status

--- ----- ------------------------------------------------ --------------------- --------

27 0 Supervisor Module N9K-C9800-SUP-A active *

28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

Mod Sw Hw Slot

--- ----------------------- ------ ----

27 10.4(3) 1.0 SUP1

28 10.4(3) 1.0 SUP2

Secondary With 10.3(x) Release

For a Secondary Supervisor with a 10.3(x) release pre-installed, one of the listed events can happen after insert:

1. Boot loop.

2. Boot as an independent supervisor.

3. Boot into loader > prompt

If the Secondary Supervisor is in the state of event 1 or 2, then you must break it into loader > prompt by typing Ctrl-C at "NXOS Grub" during the bootup process.

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2633PCKJ

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

3E772D0F6E8FC7406CEC11BBA5B2A84C00000000000000000000000000000000

Aikido FPGA Version: 0x11B

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1b

NXOS Grub <<<<< Ctrl+C here

disable watchdog

Aborting config file read and autoboot

No autoboot or failed autoboot. falling to loader

Loader Version 1.12

loader >

In the this example, the Secondary Supervisor started booting to 10.3(2)F then recognized itself as the Standby Supervisor and rebooted into loader > prompt.

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU9

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

disable watchdog

Filesystem type unknown, partition type 0x83

check_bootmode: grub: Continue grub

Trying to read config file /boot/grub/menu.lst.local from (hd0,4)

Filesystem type is ext2fs, partition type 0x83

Booting bootflash:/nxos64-cs.10.3.2.F.bin ...

Booting bootflash:/nxos64-cs.10.3.2.F.bin <<<

Trying diskboot

Filesystem type is ext2fs, partition type 0x83

Image valid

Image Signature verification was Successful.

Boot Time: 7/23/2024 22:49:10

mount: overlay mounted on /newroot/usr.

Installing klm_card_index

done

Setting nativeboot

Linking n9k mando SSD partition...

creating flash devices BOOT_DEV= sda

INIT: version 2.88 booting

Installing ata_piix module ... done.

Installing kheaders module ... done.

Unsquashing rootfs ...

Total size needed in bootflash is 153632

check bootflash : OK

Total size needed in bootflash is 54464

check bootflash : OK

Enabling 8250 serial driver spurious INTs workaround

Installing isan procfs ... done.

is_lxc: is_titan_eor: is_stby: suffix: klm_ftrace: /isanboot/lib/modules/klm_ftrace.o

Installing ftrace in non-lxc mode done

Installing SSE module with card index 21213 ... done.

Creating SSE device node 243 ... done.

Executing /etc/rc.d/rcS.d/S06exablaze start

Executing /etc/rc.d/rcS.d/S06exablaze start

Using cctrli_lite on Mando SUP...

Loading I2C driver ... done.

Installing CCTRL driver for card_type 132 without NEED_GEM ... done.

Loading IGB driver ...

Loading Intel igb driver for Mando...14.81: \nInstalling klm_ins_ixgbe for Mando... card_index=21213

Installing klm_ins_ixgbe ... done.

Not Micron SSD...

Checking all filesystems.

Extracting rpms from image...

/

MANDO RP CPA setup done

Installing SPROM driver ... 21213 IS_N9K done.

@@@ MANDO-RP

Installing pfmsvcs module ...done.

Installing nvram module ... done.

Installing if_index module with port mode 6 ... done.

Installing fcfwd

Installing RNI lcnd ... done.

Installing lcnd ... done.

\n S090setup-eth sup_start

autoneg unmodified, ignoring

no pause parameters changed, aborting

autoneg unmodified, ignoring

no pause parameters changed, aborting

Installing psdev ...

Installing veobc module ... done.

RTNETLINK answers: File exists

Clean up previous pcap files present in tmp directory

Checking SR card

Card Index is 21213

48.15: Inserting obfl & mtd spi driver ...

old data: 0 new data: 9f000000

old data: 0 new data: 0

old data: 20907001 new data: a0901406

[ 48.207892] fpga_setup_probe

48.26: Done..Inserting obfl & mtd spi driver ...

Making OBFL character devices

old data: c000000 new data: 9f000000

old data: 10c0000 new data: 0

old data: 989040 new data: a0901406

mounting plog for N9k!

48.31: mounting plog for N9k!

48.32: Done..mounting plog for N9k!

mounting recovery for N9k!

48.33: mounting recovery for N9k!

Mounting OBFL pstore for mtd

Inserting kernel_services module ... done.

Making kernel_services character devices

cgroups initialized

update-alternatives: Linking /usr/bin/unshare to /usr/bin/unshare.util-linux

Removing any system startup links for cgroups-init ...

Adding system startup for /etc/init.d/cgroups-init.

Running groupadd commands...

NOTE: docker-ce: Performing groupadd with [ -r docker]

update-alternatives: Linking /bin/vi to /usr/bin/vim.tiny

update-alternatives: Linking /usr/bin/vim to /usr/bin/vim.tiny

Removing bootvar to force RP(standby) to be at loader promptYou can use cmdline enable_standby to allow RP(standby) to boot

INIT: Switching/isan/etc/rc.cleanup ignoring inittab invocations

Stoping third party daemons

Sending all processes the TERM signal...

Sending all processes the KILL signal...

Unmounting filesystems...

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU9

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

disable watchdog

Filesystem type unknown, partition type 0x83

check_bootmode: grub: Continue grub

Trying to read config file /boot/grub/menu.lst.local from (hd0,4)

Filesystem type is ext2fs, partition type 0x83

Trying to read config file /boot/grub/menu.lst.local from (hd0,5)

Filesystem type is ext2fs, partition type 0x83

No autoboot or failed autoboot. falling to loader

Loader Version 1.11

loader >

To initiate image sync with the Active Supervisor to establish HA, we have 2 methods we can use depending on our BIOS version.

Image Sync

BIOS Version

You will need to determine the current BIOS version of the Secondary Supervisor.

The current version can be found on the 1st line of initial output when the supervisor is inserted & after reboot before the loader > prompt.

Cisco N9800(R) Series BIOS Ver 1.11 Primary <<<

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU9

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

Force Download from Active

If the BIOS version installed is 1.11 or HIGHER, image sync can be manually initiated to the Secondary Supervisor from the Active Supervisor using the reload module 28 force-dnld command.

The Secondary Supervisor must be at the loader > prompt when executing this command.

Similar to the Different Release Images section, the Secondary Supervisor will be rebooted to force NetBoot and download the Active Supervisor's boot image to the Secondary Supervisor.

From the Active Supervisor -

N9K-C9808# reload module 28 force-dnld

N9K-C9808# 2024 Jul 23 22:59:15 N9K-C9808 %$ VDC-1 %$ %PLATFORM-2-PFM_MODULE_RESET: Manual restart of Module 28 from Command Line Interface

From the Secondary Supervisor -

Loader Version 1.11

loader >

< Reload Initiated by Active - Reboots into NetBoot process >

Cisco N9800(R) Series BIOS Ver 1.11 Primary

Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz

Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4

Chipguard Init

TAM Library Version: 3.4.26

Aikido Version: 5.0.13

Aikido Chip Serial Number:

006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000

Aikido FPGA Version: 0x120

Aikido chip authentication successful

Aikido KEY Format: UEFI

Cisco PK Key

Cisco Key Key

Cisco DBX Key

Cisco OS Rel Key

Cisco OS Dev Key

Board type 5

IOFPGA @ 0xb2400000

SLOT_ID @ 0x1c

NXOS Grub

disable watchdog

Finding driver for NIC vendor 8086 Device 15ab

i210 devide id = 1537

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

x check driver for NIC vendor 8086 Device 15ab

inside driver for NIC vendor 8086 Device 15ab

pci_base_addres 0= b2c0000c ioaddr= 0

mask issue

Found the device 8086:15ab at ioaddr 1000, membase b2c00000 at 4:0

Probing...

intelx probe

eobc MAC address 00000000:00000000:00000000:0000001c:00000000:00000000

board_type=5

network is ready.

bp_op = 1

bp_htype = 1

bp_hlen = 6

bp_xid = -160883712

bp_hwaddr = 00:00:00:1C:00:00

bp_hops = 0

bp_secs = 2132726648

*retry 1 currticks 27126 timeout = 720

*retry 2 currticks 27684 timeout = 720

bp_op = 1

bp_htype = 1

bp_hlen = 6

bp_xid = -160883712

bp_hwaddr = 00:00:00:1C:00:00

bp_hops = 0

bp_secs = 2132726648

errnum = 0

reqretry = 0

*2 retry 2 currticks 27684 timeout = 720

Filename: /img-sync/curr-ks.img

board_type=5

board_type=5

Address: 127.1.1.28

Netmask: 255.255.0.0

Server: 127.1.1.27

Gateway: 0.0.0.0

Filesystem type is tftp, using whole disk

.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Image valid

Image Signature verification was Successful.

Once the download has complete, the Secondary Supervisor will boot to the new image then move to establish ha-standby state with the Active Supervisor.

N9K-C9808# show module

Mod Ports Module-Type Model Status

--- ----- ------------------------------------------------ --------------------- --------

27 0 Supervisor Module N9K-C9800-SUP-A active *

28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

Mod Sw Hw Slot

--- ----------------------- ------ ----

27 10.4(3) 1.0 SUP1

28 10.4(3) 1.0 SUP2

EOBC Boot Secondary Supervisor

If the BIOS version installed is LOWER than 1.11, image sync can ONLY be manually initiated by the Secondary Supervisor from within the loader > prompt using the eobc command.

This will force the switch to TFTP boot from the active supervisor then sync with it.

loader > ?

? Print the command list

boot Boot image

bootmode Display/Change current boot mode

dir List file contents on a device

eobc Booting image from active sup via EOBC channel

keyinfo BIOS KEY information

help Print the command list or the specific command usage

ip Setting IP address or gateway address

reboot Reboot the system

serial Serial console setting

set Set network configuration

show Show loader configuration

loader > eobc

Finding driver for NIC vendor 8086 Device 15ab

i210 devide id = 1537

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

loop driver for NIC vendor 8086 Device 15ab

x check driver for NIC vendor 8086 Device 15ab

inside driver for NIC vendor 8086 Device 15ab

pci_base_addres 0= b2c0000c ioaddr= 0

mask issue

Found the device 8086:15ab at ioaddr 1000, membase b2c00000 at 4:0

Probing...

intelx probe

< After a few minutes supervisor starts TFTP Boot process >

Address: 127.1.1.27

Netmask: 255.255.0.0

Server: 127.1.1.28

Gateway: 0.0.0.0

Filesystem type is tftp, using whole disk

................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Image valid

< After some time the TFTP boot process completes and the Image Sync process begins. >

Saving image for img-sync ...

Enough free disk space found on bootflash

< The sync process does not show progress status and can take some time. >

VERIFYING Md5 checksum for /bootflash/curr-ks.img ....

Verify OK!!

< Soon afterwards the Supervisor successfully completes the boot and sync process then comes up as the Standby Supervior. >

ethernet switching mode Fri Jun 21 15:46:14 UTC 2024

prepare span CLI

[ 666.302971] MANDO RDN : register_rdn_int

[ 666.307460] register_rdn_int : Sent First Heartbeat

User Access Verification

N9K-C9808(standby) login:

The entire boot and sync process can take more than 15 minutes.

It can be monitored from the console.

On the Active Supervisor you can confirm the status of the Secondary Supervisor is ha-standby.

N9K-C9808# show module

Mod Ports Module-Type Model Status

--- ----- ------------------------------------------------ --------------------- --------

27 0 Supervisor Module N9K-C9800-SUP-A active *

28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

Mod Sw Hw Slot

--- ----------------------- ------ ----

27 10.4(3) 1.0 SUP1

28 10.4(3) 1.0 SUP2

Performing either method mentioned is only required one time.

It is recommended to delete the 10.3(x) release from bootflash of the Standby Supervisor leaving the 10.4(x) release remaining.

Upgrade BIOS of Standby Supervisor

The Active Supervisor will already be on the most up to date BIOS version if a supported upgrade doing install all nxos was done on it.

The processes shown in the Different Release Images, Force Download from Active and EOBC Boot Secondary Supervisor sections do NOT upgrade the BIOS on the Standby Supervisor.

The next step is to upgrade the BIOS on the Standby Supervisor to match the Active.

If just the Standby Supervisor BIOS is the only one being upgraded it does not need to be reloaded.

N9K-C9808# install all nxos bootflash:nxos64-cs.10.4.3.F.bin

Installer will perform compatibility check first. Please wait.

Installer is forced disruptive

Verifying image bootflash:/nxos64-cs.10.4.3.F.bin for boot variable "nxos".

[####################] 100% -- SUCCESS

Verifying image type.

[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos64-cs.10.4.3.F.bin.

[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos64-cs.10.4.3.F.bin.

[####################] 100% -- SUCCESS

Performing module support checks.

[####################] 100% -- SUCCESS

Notifying services about system upgrade.

[####################] 100% -- SUCCESS

Compatibility check is done:

Module bootable Impact Install-type Reason

------ -------- -------------- ------------ ------

27 yes disruptive copy-only default upgrade is not hitless

28 yes disruptive none default upgrade is not hitless

Images will be upgraded according to following table:

Module Image Running-Version(pri:alt) New-Version Upg-Required

------ ---------- ---------------------------------------- -------------------- ------------

27 nxos 10.4(3) 10.4(3) no

27 bios v01.09(12/19/2022):v01.03(12/29/2021) v01.12(11/28/2023) yes

28 nxos 10.4(3) 10.4(3) no

28 bios v01.12(11/28/2023):v01.03(12/29/2021) v01.12(11/28/2023) no

Switch will be reloaded for disruptive upgrade.

Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Performing runtime checks.

[####################] 100% -- SUCCESS

Syncing image bootflash:/nxos64-cs.10.4.3.F.bin to standby.

[####################] 100% -- SUCCESS

Setting boot variables.

[####################] 100% -- SUCCESS

Performing configuration copy.

[####################] 100% -- SUCCESS

Module 20: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

Module 27: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.

[####################] 100% -- SUCCESS

Install has been successful.

BIOS version on both Standby and Active Supervisors now match but EPLD version does not:

N9K-C9808(standby)# show hardware internal dev-version

-------------------------------------------------------------------

Name InstanceNum Version Date

-------------------------------------------------------------------

IO FPGA 0 0x1001b

TM FPGA 0 0x10006

BIOS version v01.12(11/28/2023)

Alternate BIOS version v01.03(12/29/2021)

N9K-C9808# show hardware internal dev-version

-------------------------------------------------------------------

Name InstanceNum Version Date

-------------------------------------------------------------------

IO FPGA 0 0x10017

TM FPGA 0 0x10002

FM Module-20 FPGA 1 0x10000

FM Module-22 FPGA 3 0x10000

FM Module-24 FPGA 5 0x10000

FM Module-26 FPGA 7 0x10000

BIOS version v01.12(11/28/2023)

Alternate BIOS version v01.03(12/29/2021)

Upgrade EPLD of Supervisors

As mentioned in the Cisco Nexus 9000 Series NX-OS High Availability and Redundancy Guide, Release 10.4(x), it is a mandatory requirement to perform an EPLD upgrade for Nexus 9800 Dual Supervisor deployments.

As further elaborated in Cisco bug ID CSCwk42621 This table shows the minimum EPLD version on the Supervisors for HA to fully function:

IO FPGA

0x10020 or HIGHER

TM FPGA

0x10006 or HIGHER

You can upgrade individual modules or the entire chassis at once.

Depending on which modules are upgraded, this procedure can be disruptive.

It is recommended to upgrade all modules if possible.

The number of modules being upgraded will affect how long the entire upgrade will take.

Cisco recommends performing the EPLD upgrade during a maintenance window to minimize potential disruption.

N9K-C9808# install epld bootflash:n9000-epld.10.4.3.F.img module ?

WORD X or x-y (Max Size 5)

all Install all the modules

N9K-C9808# install epld bootflash:n9000-epld.10.4.3.F.img module all

Compatibility check:

Module Type Upgradable Impact Reason

------ ----------------- ---------- ---------- ------

20 FM Yes disruptive Module Upgradable

22 FM Yes disruptive Module Upgradable

24 FM Yes disruptive Module Upgradable

26 FM Yes disruptive Module Upgradable

27 SUP Yes disruptive Module Upgradable

28 SUP Yes disruptive Module Upgradable

Retrieving EPLD versions.... Please wait.

Images will be upgraded according to following table:

Module Type EPLD Running-Version New-Version Upg-Required

------ ---- ------------- --------------- ----------- ------------

20 FM MI FPGA 0x10000 0x10002 Yes

22 FM MI FPGA 0x10000 0x10002 Yes

24 FM MI FPGA 0x10000 0x10002 Yes

26 FM MI FPGA 0x10000 0x10002 Yes

27 SUP TM FPGA 0x10006 0x10006 No

27 SUP IO FPGA 0x1001b 0x10020 Yes

28 SUP TM FPGA 0x10002 0x10006 Yes

28 SUP IO FPGA 0x10017 0x10020 Yes

The above modules require upgrade.

Do you want to continue (y/n) ? [n] y

Do you want to upgrade the Active Supervisor? [n] y

Proceeding to upgrade Modules.

Proceeding to upgrade Modules.

Starting Module 20 EPLD Upgrade

Module 20 : MI FPGA [Programming] : 0.00% ( 0 of 64 sectors)

Module 20 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)

Module 20 EPLD upgrade is successful..

Module 20 EPLD upgrade is successful.

Starting Module 22 EPLD Upgrade

Module 22 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)

Module 22 EPLD upgrade is successful..

Module 22 EPLD upgrade is successful.

Starting Module 24 EPLD Upgrade

Module 24 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)

Module 24 EPLD upgrade is successful..

Module 24 EPLD upgrade is successful.

Starting Module 26 EPLD Upgrade

Module 26 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)

Module 26 EPLD upgrade is successful..

Module 26 EPLD upgrade is successful.

Module 27 : IO FPGA [Programming ] : 0.00% ( 0 of 64 total sectors)

Module 27 : IO FPGA [Programming ] : 100.00% ( 64 of 64 total sectors)

Module 27 EPLD upgrade is successful.

Starting Module 28 EPLD Upgrade

Module 28 : IO FPGA [Programming] : 100.00% ( 64 of 64 sectors)

Module 28 EPLD upgrade is successful..

Module Type Upgrade-Result

------ ------------------ --------------

27 SUP Success

28 SUP Success

EPLDs upgraded. Performing switchover.

Module 28 EPLD upgrade is successful.

Reloading the Chassis...

Reseting Standby SUP (Module 27) FPGAs. Please wait...

Module 27 will reload.

Resetting Module 20 FPGAs. Please wait...

Module 20 will reload.

Resetting Module 22 FPGAs. Please wait...

Module 22 will reload.

Resetting Module 24 FPGAs. Please wait...

Module 24 will reload.

Resetting Module 26 FPGAs. Please wait...

Module 26 will reload.

Processing Active sup reset...

Reseting Active SUP (Module 28) FPGAs. Please wait...

Reload in 10 seconds .....

Resetting Module 20 FPGAs. Please wait...

Module 20 will reload.

After EPLD upgrade and reload the Active and Standby Supervisor roles can be switched:

N9K-C9808# show module

Mod Ports Module-Type Model Status

--- ----- ------------------------------------------------ --------------------- --------

27 0 Supervisor Module N9K-C9800-SUP-A active *

28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

To change the active supervisor you can use the "system switchover" command to reload the Active Supervisor and make the Standby Supervisor the Active.

From the Active Supervisor:

N9K-C9808# system switchover

From the Standby Supervisor as it becomes the Active:

N9K-C9808(standby) login: [ 784.909143] Got RP P2PM intr, Becoming Active !!

User Access Verification

N9K-C9808 login: admin

Password:

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Copyright (C) 2002-2024, Cisco and/or its affiliates.

All rights reserved.

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under their own

licenses, such as open source. This software is provided "as is," and unless

otherwise stated, there is no warranty, express or implied, including but not

limited to warranties of merchantability and fitness for a particular purpose.

Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or

GNU General Public License (GPL) version 3.0 or the GNU

Lesser General Public License (LGPL) Version 2.1 or

Lesser General Public License (LGPL) Version 2.0.

A copy of each such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://opensource.org/licenses/gpl-3.0.html and

http://www.opensource.org/licenses/lgpl-2.1.php and

http://www.gnu.org/licenses/old-licenses/library.txt.

N9K-C9808# show module

Mod Ports Module-Type Model Status

--- ----- ------------------------------------------------ --------------------- --------

27 0 Supervisor Module powered-up

28 0 Supervisor Module N9K-C9800-SUP-A active *

Summary

1. Install Secondary Supervisor

2. Perform process to image sync Secondary with Active Supervisor depending on pre-installed release

3. Upgrade BIOS on Standby Supervisor

4. Upgrade EPLD for all modules