Follow the guidelines in the "Guidelines and Limitations for ISSU" section to ensure that the ISSU works smoothly.

Make sure that the network is stable and no changes are made while the ISSU is in progress.

Ensure feature compatibility between the current running release and the target release.

Make sure that interfaces are not in a spanning-tree designated forwarding state. Also, make sure that bridge assurance is not configured on any interface. The vPC peer-link is an exception to these requirements.

Verify that the current STP topology is consistent with the ISSU requirements. Use the show spanning-tree issu-impact command to display the STP configuration and whether or not there are potential STP issues.

Use the show lacp issu-impact command to display if a port or a peer switch is configured in the rate fast mode. The ISSU process is aborted if the system has any LACP fast timers configured.

The following is a list of important guidelines and limitations for ISSU:

• Non-disruptive standard ISSU on Cisco Nexus 3172PQ, 3172TQ, 3132Q, 3132Q-X, 3064, 3064-X, 3064-T, 3048, 3016 (4 GB low-memory platforms) is supported from Cisco Nexus 7.0(3)I7(7) or later releases to the Cisco NX-OS 9.2(X) release.

• You can perform an ISSU for segment routing or VXLAN.

• Standard ISSUs are supported on the following platforms:

  • Cisco Nexus 3172 switches

  • Cisco Nexus 3132Q and 3132Q-XL switches

  • Cisco Nexus 3064 switches

  • Cisco Nexus 3048 switches

  • Cisco Nexus 3016 switches

• Beginning with Cisco NX-OS Release 7.0(3)I5(1), Cisco Nexus 3000 series platforms with 16 GB of memory or higher support enhanced ISSU. The upgrade will be disruptive.

• The minimum free bootflash space required to perform ISSU is as follows:

  • For compact image: 200 MB

  • For non-compact image: 300 MB

• The minimum free bootflash space required to perform ISSU is 550 MB.

• CLI and the SNMP configuration change requests are denied during ISSU operations.

• STP topology changes are not expected during an ISSU.

• The Guest Shell is disabled during an ISSU, and it is reactivated after the upgrade. During an ISSU, all First-Hop Redundancy Protocols (FHRPs) cause the other peer to become active if the node undergoing the ISSU is active.

• Beginning with Cisco NX-OS Release 9.2(1), a simplified NX-OS numbering format is used for the platforms that are supported in the release. In order to support a software upgrade from releases prior to Release 7.0(3)I7(4) that have the old release format, an installer feature supplies an I9(1) label as a suffix to the actual release during the install all operation. This label is printed as part of the image during the install operation from any release prior to 7.0(3)I7(4) to Release 9.2(1), and it can be ignored. See the following example.

• switch# install all nxos bootflash:nxos.9.2.1.bin
  Installer will perform compatibility check first. Please wait. 
  Installer is forced disruptive
  
  Verifying image bootflash:/nxos.9.2.1.bin for boot variable "nxos".
  [####################] 100% -- SUCCESS
  
  Verifying image type.
  [####################] 100% -- SUCCESS
  
  Preparing "nxos" version info using image bootflash:/nxos.9.2.1.bin.
  [####################] 100% -- SUCCESS
  
  Preparing "bios" version info using image bootflash:/nxos.9.2.1.bin.
  [####################] 100% -- SUCCESS
  
  Performing module support checks.
  [####################] 100% -- SUCCESS
  
  Notifying services about system upgrade.
  [####################] 100% -- SUCCESS
  
  Compatibility check is done:
  Module  bootable   Impact       Install-type  Reason
  ------  --------  ------------  ------------  ------
    1       yes      disruptive     reset       Incompatible image for ISSU
  
  Images will be upgraded according to following table:
  Module   Image                 Running-Version(pri:alt)  New-Version          Upg-Required
  ------  -------  -------------------------------------- --------------------  ------------
    1      nxos                               7.0(3)I7(3)          9.2(1)I9(1)           yes
    1      bios     v05.31(05/17/2018):v05.26(11/06/2017)   v05.31(05/17/2018)            no
  
  
  Switch will be reloaded for disruptive upgrade.
  Do you want to continue with the installation (y/n)?  [n] y
  

• Cisco Nexus 3048, 3064, 3132, and 3172 switches with a model number that does not end in -XL must run a compact NX-OS software image due to limited bootflash space. This compact image can be created using the NX-OS Compact Image procedure; alternatively, a compact NX-OS software image can be downloaded directly from Cisco's Software Download website. This requirement does not apply to any other model of Nexus 3000 or 3100 series switch.

  • The MD5/SHA512 checksum published on Cisco's Software Download website for a compact NX-OS software image may not match the MD5/SHA512 checksum of a compact image created through the NX-OS Compact Image procedure.

Compact NX-OS software images are available for download on Cisco's Software Download website for a few NX-OS software releases. These compact images have a published MD5/SHA512 checksum that can be used to verify the integrity of the NX-OS binary image file. The compact NX-OS software images can be downloaded from Cisco's Software Download website for the following NX-OS software releases:

• 9.3(4) and later

• 9.2(4)

• 7.0(3)I7(8) and later

Note	The MD5/SHA512 checksum published on Cisco's Software Download website for a compact NX-OS software image may not match the MD5/SHA512 checksum of a compact image created through the NX-OS Compact Image procedure.

There are two ways to initiate the NX-OS Compact Image procedure. The following subsections describe each option in further detail.

See the following guidelines for the BIOS upgrade:

Cisco Nexus 3000 Series platforms require a BIOS upgrade to load Release 7.0(3)I2(1) or later images. Only BIOS versions 3.x.x are compatible with Release 7.0(3)I2(1) and later releases on all Cisco Nexus 3000 Series switches. This is taken care in the regular install all method of upgrade but it needs to be explicitly upgraded prior to the fast-reload upgrade.

Only after the BIOS upgrade is complete, it allows the system to perform fast-reload to the newer releases.

To avoid link down to link up transitions during the control plane outage time, the laser is turned off for administratively up ports that are operationally down. This situation occurs during the upgrade reboot starting state. After the upgrade reboot and a stateful restart, the laser is turned back on. This action prevents the link state from transitioning from down to up during an upgrade.

Fast reboot has the following timing requirements for the configurations that it supports:

• Time taken to reset the ASIC and disrupt the data plane after control plane disruption—Less than 90 seconds, when the control plane is disrupted.

• Time taken to resume forwarding traffic—Less than 30 seconds from ASIC reset.

Fast reboot is supported only with limited configurations and topologies. Some of supported configurations and guidelines are listed in this section. When reloading system software by using the fast-reload command, use the supported follow these guidelines:

• Configuration changes—You cannot enter configuration mode during a reload or an upgrade. You should save, commit, or discard any active configuration sessions before upgrading or downgrading the Cisco NX-OS software image. The active configuration session is deleted without a warning during a reload.

  Use the show configuration session summary command to verify that there are no active configuration sessions.

  switch# show configuration session summary
  There are no active configuration sessions

  Ensure that you check the compatibility of configurations before using the fast-reload command.

  Note	Do not use the fast-reload command for upgrades that may lead to kernel or BIOS changes.

  For more information on configuration sessions, see the Cisco Nexus 3000 Series NX-OS System Management Configuration Guide.

  Note	The CLI and SNMP configuration change requests are denied.

• Topology—You should make topology changes such as Spanning Tree Protocol (STP) before you perform an upgrade. You should perform module installations or removals only before or after an upgrade. However, you should not make changes to the Layer 2 and routing topologies, and the default root bridge should not be configured.

• Scheduling—You should upgrade when your network is stable and steady. Ensure that everyone who has access to the switch or the network is not configuring the switch or the network during this time. You cannot configure a switch during an upgrade.

• Space—Verify that sufficient space is available in the location where you are copying the images. The internal bootflash requires approximately 650 MB of free space.

• Hardware—Avoid power interruptions during an installation procedure. Power interruptions can corrupt the software image.

• Connectivity to remote servers—Configure the IPv4 address or IPv6 address for the 10/100/1000 BASE-T Ethernet port connection (interface mgmt0). Ensure that the switch has a route to the remote server. The switch and the remote server must be in the same subnetwork if you do not have a router to route traffic between subnets.

• Link Aggregation Control Protocol (LACP) fast timers—To allow fast-reload, ensure that LACP fast timers are not configured.

• Retrieve compatible images in one of two ways:

  • Locally—Images are locally available on the switch.

  • Remotely—Images are in a remote location and you specify the destination using the remote server parameters and the filename to be used locally.

• Command—Use the following commands to prepare for and install the new software:

  • Use the ping command to verify connectivity to the remote server.

  • Use the dir command to verify the required space is available for the image files to be copied.

  • Use the show install all impact command to identify the upgrade impact. This command also displays whether the upgrade is disruptive or the reason why the upgrade is disruptive, whether the switch needs to be rebooted, and the reason why it needs to be rebooted.

  Note	We recommended that you log in to the console port to begin the upgrade process.

• Between control plane disruption and data plane disruption, the CPU stops responding.

• Configuration—Fast reboot currently supports the following configuration:

  • BGP v4 and v6

  • 16-way ECMP

  • 48 downlink L2 ports

  • 4 SVIs

  • Less than 10 VLANs

  • 2000 v4 routes and 2000 v6 routes

  • RACLs

  • ARPs

  • STP edge port configuration

• Repaved fast-reload also supports the change in configurations that earlier required a complete reload, for example, portmode profile, URPF enable/disable, and TCAM re-carving.

• Do not use the fast-reload option for upgrade to/from a version that has reached EOL.

• The fast reload feature also supports the change in the configurations that earlier required a complete reload, for example, portmode profile, URPF enable/disable, and TCAM re-carving.

• Fast reload does not upgrade the BIOS.

• If you upgrade from a Cisco NX-OS release that does not support the CoPP feature to a release that does support the CoPP feature, you must run the setup utility after the upgrade to enable CoPP on the device.

• Fast reload upgrades are supported from Cisco NX-OS Release 7.0(3)I7(4) or 7.0(3)I7(5) to a Cisco NX-OS 9.x release.

• Beginning with Cisco NX-OS Release 9.3(10), Fast Reload support is “hidden” in the NX-OS CLI, but still available if the command is executed. Fast Reload feature is not in the Release 9.3(x) train going forward.

• Cisco removes any public documentation claiming fast reload performance of under 30 seconds for the Cisco Nexus 3164Q switches.

• If you downgrade to a release lower than Cisco NXOS release 6.0(2)U2(1) using the install-all command, fast reload does not work.

ASCII configuration based fast-reload is also supported in addition to the PSS/binary configuration based fast-reload.

Note	The copy file startup and fast reload are supported only for specific configurations, namely Layer 3 ports with port channels, eBGP, and a few physical l2 ports having SVI only towards the hosts.

To use ASCII-file based fast reload, use the following command:

copy configuration-file startup-config

fast-reload nxos bootflash:nxos-image-name

The configuration-file is an ASCII file that contains the system configurations that fast reload uses on upgrade or fast reload. It can be copied from the remote location also. If the NXOS software image is not specified, the image existing on the switch is reloaded. If the NXOS software image provided is a higher version than the existing version, an upgrade is triggered.

Note	To ensure that subsequent fast reboot operations, use the NXOS software image as the boot variables, specify the save-config option while running the fast-reload command. If the save-config option is not specified, the fast-reload command does not save the boot variables.

You can perform a faster reload and upgrade to a newer release with minimal data downtime compared to install all command.

The feature is similar to existing fast-reload support on Cisco Nexus 3000 Series switches from Release 6.0(2)U2(1) onwards with a few additional steps to upgrade BIOS before upgrading to Release 7.0(3)I2(1) or later releases.

Caution

The fast-reload to Release 7.0(3)I2(1) is supported only from Release 6.0(2)U6(3a) or later releases, after the BIOS has been upgraded. Using fast-reload from Release 6.0(2)U6(3a) to Release 7.0(3)I2(1) without upgrading the BIOS will result in the switch not booting up. See BIOS upgrade for more information.

Note	If guest shell or any virtual-services are enabled, the install all command should be used. Use the show virtual-service list command to verify the presence or the state of the guest shell or any virtual services..

Cisco NX-OS software allows you to enable BGP graceful restarts (GR) with fast reboot. You can now use the fast-reload trigger-gr command to enable BGP GR. Use this command only when all BGP peers are GR-capable.

To enable BGP GR with fast reboot on Cisco Nexus 3000 and 3100 Series switches, use the following command:

fast-reload [save-config] [trigger-gr] [nxos bootflash:nxos-image-name] [non-interruptive] command to perform a fast reload.

switch# fast-reload trigger-gr nxos bootflash:nxos.9.2.1.bin non-interruptive

You can upgrade the software on a switch by using fast reboot. To upgrade, you must specify the NXOS software image.

Before You Begin

Ensure that the version of the NXOS software image specified in the fast-reload command is higher than the version of the image currently existing on the switch.

switch# fast-reload nxos bootflash:nxos.9.2.1.bin

Downgrading the system software by using fast reboot is not supported. To downgrade the image software, use the install all command instead of fast reboot.

The following list summarizes the upgrade process in a non-vPC topology:

1. The install all command triggers the installation upgrade.

2. The compatibility checks display the impact of the upgrade.

3. The installation proceeds or not based on the upgrade impact.

4. The current state is saved.

5. The system unloads and runs the new image.

6. The stateful restart of the system software and application occurs.

7. The installer resumes with the new image.

8. The installation completes.

The following example displays the upgrade process:

switch# install all nxos bootflash:<nxos-image-name>

The following list summarizes the upgrade process on a switch in a vPC topology that holds either the Primary or Operational Primary vPC roles. Steps that differ from a switch upgrade in a non-vPC topology are in bold.

Note	In vPC topologies, the two peer switches must be upgraded individually. An upgrade on one peer switch does not automatically update the vPC peer switch.

1. The install all command issued on the vPC primary switch triggers the installation upgrade.

2. The compatibility checks display the impact of the upgrade.

3. The installation proceeds or not based on the upgrade impact.

4. The configuration is locked on both vPC peer switches.

5. The current state is saved.

6. The system unloads and runs the new image.

7. The stateful restart of the system software and application occurs.

8. The installer resumes with the new image.

9. The installation is complete.

When the installation is complete, the vPC primary switch is upgraded.

Note	The vPC primary switch is running the upgraded version, and the vPC secondary switch is running the original software version.

The following list summarizes the upgrade process on a switch in a vPC topology that holds either the Secondary or Operational Secondary vPC roles. Steps that differ from a switch upgrade in a non-vPC topology are in bold.

1. The install all command issued on the vPC secondary switch triggers the installation upgrade.

2. The compatibility checks display the impact of the upgrade.

3. The installation proceeds or not based on the upgrade impact.

4. The current state is saved.

5. The system unloads and runs the new image.

6. The stateful restart of the system software and application occurs.

7. The installer resumes with the new image.

8. The configuration is unlocked on the primary and secondary switches.

9. The installation is complete.

In vPC topologies, the two peer switches usually must be upgraded individually. An upgrade on one peer switch does not automatically update the vPC peer switch.

However, Cisco NX-OS Releases 7.0(3)F3(3c) and 7.0(3)F3(4) are not compatible with Cisco NX-OS Release 9.2(x) for vPC peer switches. Both vPC peers must be upgraded simultaneously to Cisco NX-OS Release 9.2(x) to avoid one switch running a 7.0(3)F3(x) release and the other switch running 9.2(x). Optionally, if the switches are being upgraded from Cisco NX-OS Release 7.0(3)F3(4), you can use the following procedure to minimize the traffic impact during upgrade.

Note	This procedure not to be used on Broadcom or Cloudscale-based switches.

1. Switch A and B are running a Cisco NX-OS release. Switch A is the primary switch, and switch B is the secondary switch. Use the copy r s command on both switches.

   
   primary_switch# show vpc role
   vPC Role status
   ----------------------------------------------------
   vPC role : primary 
   vPC system-mac : 00:23:04:ee:be:64 
   vPC system-priority : 32667
   vPC local system-mac : 70:df:2f:eb:86:1f 
   vPC local role-priority : 90 
   vPC peer system-mac : 70:df:2f:eb:1c:ab 
   vPC peer role-priority : 100 
   primary_switch#
   
   secondary_switch# show vpc role
   vPC Role status
   ----------------------------------------------------
   vPC role : secondary 
   vPC system-mac : 00:23:04:ee:be:64 
   vPC system-priority : 32667
   vPC local system-mac : 70:df:2f:eb:1c:ab 
   vPC local role-priority : 100 
   vPC peer system-mac : 70:df:2f:eb:86:1f 
   vPC peer role-priority : 90 
   secondary_switch#
   
   
   primary_switch# copy r s v
   [########################################] 100%
   Copy complete.
    
   secondary_switch# copy r s v
   [########################################] 100%
   Copy complete.
    
   

2. Bring down the peer link (PL) on the primary switch. The secondary switch brings down its vPC legs.

   
   primary_switch# conf t
   Enter configuration commands, one per line. End with CNTL/Z.
   primary_switch(config)# int port-channel 100
   primary_switch(config-if)# shutdown 
    
   Reload the secondary switch with Release 9.2.1 image (change bootvar /reload)
    
   secondary_switch(config)# boot nxos nxos.9.2.1.bin
   Performing image verification and compatibility check, please wait....
   secondary_switch(config)# 
   secondary_switch(config)# copy r s v
   [########################################] 100%
   Copy complete.
    
    
   secondary_switch# reload
   This command will reboot the system. (y/n)? [n] y
   
    
   After reload
   --------------------- 
   secondary_switch# show vpc
   Legend:
   (*) - local vPC is down, forwarding via vPC peer-link
   vPC domain id : 100 
   Peer status : peer link is down 
   vPC keep-alive status : peer is alive 
   Configuration consistency status : failed 
   Per-vlan consistency status : success 
   Configuration inconsistency reason: Consistency Check Not Performed
   Type-2 inconsistency reason : Consistency Check Not Performed
   vPC role : none established 
   Number of vPCs configured : 20 
   Peer Gateway : Enabled
   Dual-active excluded VLANs : -
   Graceful Consistency Check : Disabled (due to peer configuration)
   Auto-recovery status : Disabled
   Delay-restore status : Timer is off.(timeout = 90s)
   Delay-restore SVI status : Timer is off.(timeout = 10s)
   Operational Layer3 Peer-router : Disabled
   vPC Peer-link status
   ---------------------------------------------------------------------
   id Port Status Active vlans 
   -- ---- ------ -------------------------------------------------
   1 Po100 down -
    
   secondary_switch#
    
   primary_switch(config-if)# show vpc 
   Legend:
   (*) - local vPC is down, forwarding via vPC peer-link
   vPC domain id : 100 
   Peer status : peer link is down 
   vPC keep-alive status : peer is alive 
   Configuration consistency status : success 
   Per-vlan consistency status : success 
   Type-2 consistency status : success 
   vPC role : primary 
   Number of vPCs configured : 20 
   Peer Gateway : Enabled
   Peer gateway excluded VLANs : -
   Dual-active excluded VLANs and BDs : -
   Graceful Consistency Check : Enabled
   Auto-recovery status : Enabled, timer is off.(timeout = 240s)
   Operational Layer3 Peer-router : Disabled
   vPC Peer-link status
   ---------------------------------------------------------------------
   id Port Status Active vlans 
   -- ---- ------ --------------------------------------------------
   1 Po100 down -
   
   

3. Configure vPC auto-recovery under the vPC domain on the secondary switch. Enable vpc upgrade (exec command).

   
   secondary_switch(config)# vpc domain 100
   secondary_switch(config-vpc-domain)# auto-recovery 
   secondary_switch(config-vpc-domain)# end
   
   secondary_switch# show running-config vpc
   !Command: show running-config vpc
   !Running configuration last done at: Wed May 16 06:34:10 2018
   !Time: Wed May 16 06:34:14 2018
   version 9.2(1) Bios:version 01.11 
   feature vpc
   vpc domain 100
   peer-switch
   role priority 100
   peer-keepalive destination 10.1.31.30 source 10.1.31.29
   delay restore 90
   peer-gateway
   auto-recovery
   ipv6 nd synchronize
   ip arp synchronize
   interface port-channel100
   vpc peer-link
   interface port-channel2001
   vpc 101
    
   
   secondary_switch# show vpc upgrade
   vPC upgrade : TRUE
   SVI Timer : 0
   Delay Restore Timer : 0
   Delay Orphan Port Timer : 0
   secondary_switch#
   
   secondary_switch# show vpc upgrade   >> Hidden command 
   vPC upgrade : FALSE
   SVI Timer : 10
   Delay Restore Timer : 90
   Delay Orphan Port Timer : 0
    
    
   secondary_switch# vpc upgrade   >> Hidden command 
   
   

4. After Layer 3 routes are learned on the secondary switch, reload the primary switch with the new release image. The secondary switch takes over the primary role and brings up its vPC legs in approximately 5 seconds.

   
   primary_switch(config)# show boot 
   Current Boot Variables:
   sup-1
   NXOS variable = bootflash:/nxos.9.2.1.bin
   No module boot variable set
   Boot Variables on next reload:
   sup-1
   NXOS variable = bootflash:/nxos.9.2.1.bin
   
   No module boot variable set
   primary_switch(config)# end
   
   primary_switch# show boot 
   Current Boot Variables:
   sup-1
   NXOS variable = bootflash:/nxos.9.2.1.bin
   No module boot variable set
   Boot Variables on next reload:
   sup-1
   NXOS variable = bootflash:/nxos.9.2.1.bin
   
   No module boot variable set
   primary_switch# reload
   This command will reboot the system. (y/n)? [n] y
    
   secondary_switch# show vpc
   Legend:
   (*) - local vPC is down, forwarding via vPC peer-link
   vPC domain id : 100 
   Peer status : peer link is down 
   vPC keep-alive status : peer is not reachable through peer-keepalive
   Configuration consistency status : failed 
   Per-vlan consistency status : success 
   Configuration inconsistency reason: Consistency Check Not Performed
   Type-2 inconsistency reason : Consistency Check Not Performed
   vPC role : primary 
   Number of vPCs configured : 20 
   Peer Gateway : Enabled
   Dual-active excluded VLANs : -
   Graceful Consistency Check : Disabled (due to peer configuration)
   Auto-recovery status : Enabled, timer is off.(timeout = 240s)
   Delay-restore status : Timer is off.(timeout = 0s)
   Delay-restore SVI status : Timer is off.(timeout = 0s)
   Operational Layer3 Peer-router : Disabled
   vPC Peer-link status
   ---------------------------------------------------------------------
   id Port Status Active vlans 
   -- ---- ------ -------------------------------------------------
   1 Po100 down -
   vPC status
   
   

5. When the primary switch comes back up, the peer link on it is operationally up.

   
   primary_switch# show vpc
   Legend:
   (*) - local vPC is down, forwarding via vPC peer-link
   vPC domain id : 100 
   Peer status : peer adjacency formed ok 
   vPC keep-alive status : peer is alive 
   Configuration consistency status : success 
   Per-vlan consistency status : success 
   Type-2 consistency status : success 
   vPC role : primary, operational secondary
   Number of vPCs configured : 20 
   Peer Gateway : Enabled
   Dual-active excluded VLANs : -
   Graceful Consistency Check : Enabled
   Auto-recovery status : Disabled
   Delay-restore status : Timer is off.(timeout = 90s)
   Delay-restore SVI status : Timer is off.(timeout = 10s)
   Operational Layer3 Peer-router : Disabled
   vPC Peer-link status
   ---------------------------------------------------------------------
   id Port Status Active vlans 
   -- ---- ------ -------------------------------------------------
   1 Po100 up 1,101-400
   
   

   For downgrade, reload both switches at the same time.