Generic

The guidelines that apply to all upgrades irrespective of the releases are

• A pre-upgrade generic checklist includes:

  • Schedule the upgrade when your network is stable and steady.

  • Avoid any power interruption, which could corrupt the software image, during the installation procedure.

  • On devices with dual supervisor modules, both supervisor modules must have connections on the console ports to maintain connectivity when switchovers occur during a software upgrade. For more information about your specific chassis, see the relevant Hardware Installation Guide.

  • Perform the installation on the active supervisor module, not the standby supervisor module.

• When you use install all with no-reload option, the saved configuration cannot be used before you reload the device. Saving configuration in this state can result in incorrect startup configuration once you reload the device with new version of NX-OS.

• During upgrade, while performing device reload, if ASCII replay is triggered without binary restore, primary key gets lost. The primary key must be reconfigured after device reload. Use the key config-key ascii command to reconfigure the primary key and avoid encryption issues. However, upgrade with binary restore retains the primary key after the reboot.

• ISSU is blocked if boot poap enable is configured.

• Make sure that both vPC peers are in the same mode (regular mode or enhanced mode) before performing a non-disruptive upgrade.

  Note	vPC peering between an enhanced ISSU mode (boot mode lxc) configured switch and a non-enhanced ISSU mode switch is not supported.

• During an ISSU, the software reload process on the first vPC device locks its vPC peer device by using CFS messaging over the vPC communications channel. Only one device at a time is upgraded. When the first device completes its upgrade, it unlocks its peer device. The second device then performs the upgrade process, locking the first device as it does so. During the upgrade, the two vPC devices temporarily run different releases of NX-OS; however, the system functions correctly because of its backward compatibility support.

• ISSU is not supported when onePK is enabled. You can run the show feature | include onep command to verify that this feature is disabled before performing an ISSU or enhanced ISSU.

• Occasionally, while the switch is operationally up and running, the Device not found logs are displayed on the console. This issue is observed because the switch attempts to find an older ASIC version and the error messages for the PCI probe failure are enabled in the code. There is no functionality impact or traffic loss due to this issue.

• For secure POAP, ensure that DHCP snooping is enabled and set firewall rules to block unintended or malicious DHCP servers. For more information on POAP, see the Cisco Nexus 9000 Series Fundamentals Configuration Guide.

• When you upgrade from an earlier release to a NX-OS release that supports switch profiles, you have the option to move some of the running-configuration commands to a switch profile. For more information on configuration, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide.

• Guest Shell is disabled during an ISSU and reactivated after the upgrade. All applications running in the Guest Shell are affected.

• While performing an ISSU, VRRP and VRRPv3 display these messages:

  • If VRRPv3 is enabled:

    2015 Dec 29 20:41:44 MDP-N9K-6 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: ISSU ERROR: Service
    "vrrpv3" has sent the following message: Feature vrrpv3 is configured. User can change
    vrrpv3 timers to 120 seconds or fine tune these timers based on upgrade time on all Vrrp
    Peers to avoid Vrrp State transitions. – sysmgr

  • If VRRP is enabled:

    2015 Dec 29 20:45:10 MDP-N9K-6 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: ISSU ERROR: Service "vrrp-
    eng" has sent the following message: Feature vrrp is configured. User can change vrrp
    timers to 120 seconds or fine tune these timers based on upgrade time on all Vrrp Peers to
    avoid Vrrp State transitions. – sysmgr

• An error occurs when you try to perform an ISSU if you changed the reserved VLAN without entering the copy running-config save-config and reload commands.

Software image and SMU

• A simplified NX-OS numbering format is used for platforms that are supported in NX-OS 10.1(x) releases. In order to support a software upgrade from releases prior to NX-OS Release 7.0(3)I7(4) that have the old release format, an installer feature supplies an I9(x) label as a suffix to the actual release during the install all operation. This label is printed as part of the image during the install operation from any release prior to NX-OS Release 7.0(3)I7(4) to 10.1(x), and it can be ignored. See the following example.

  switch# install all nxos bootflash:nxos.9.3.1.bin
  Installer will perform compatibility check first. Please wait. 
  Installer is forced disruptive
  
  Verifying image bootflash:/nxos.9.3.1.bin for boot variable "nxos".
  [####################] 100% -- SUCCESS
  
  Verifying image type.
  [####################] 100% -- SUCCESS
  
  Preparing "nxos" version info using image bootflash:/nxos.9.3.1.bin.
  [####################] 100% -- SUCCESS
  
  Preparing "bios" version info using image bootflash:/nxos.9.3.1.bin.
  [####################] 100% -- SUCCESS
  
  Performing module support checks.
  [####################] 100% -- SUCCESS
  
  Notifying services about system upgrade.
  [####################] 100% -- SUCCESS
  
  Compatibility check is done:
  Module  bootable   Impact       Install-type  Reason
  ------  --------  ------------  ------------  ------
    1       yes      disruptive     reset       Incompatible image for ISSU
  
  Images will be upgraded according to following table:
  Module   Image                 Running-Version(pri:alt)  New-Version          Upg-Required
  ------  -------  -------------------------------------- --------------------  ------------
    1      nxos                               7.0(3)I7(3)          9.3(1)I9(1)           yes
    1      bios     v07.61(04/06/2017):v07.61(04/06/2017)   v05.33(09/08/2018)           yes
  
  
  Switch will be reloaded for disruptive upgrade.
  Do you want to continue with the installation (y/n)?  [n] y
  

• Beginning with NX-OS Release 10.2(2)F, Nexus 9504 and 9508 platform switches, and Nexus 9508-R, R2, and RX line cards support NX-OS 64-bit images. Disruptive upgrade from earlier releases to 10.2(2)F 64-bit NX-OS image is supported. NX-OS 32-bit image is not supported on these platform switches anymore.

• Beginning from NX-OS Release 10.2(1), Nexus 9300 and 9500 platform switches support 64-bit image.

• Beginning with NX-OS Release 10.1(x), when an existing SMU is active, if you install a bundle that contains the existing active SMU, the installer installs only the non-existing SMUs.

• Software image compaction is only supported on Nexus 9300-series platform switches.

• The compressed image of Nexus 3000-series is hardware dependent and can only be used on the same device that it got compressed or downloaded from CCO. Do not use the Nexus 3000-series compressed image on Nexus 9000-series.

• Non-disruptive upgrade to 64-bit image is supported from NX-OS Release 9.3(9) onwards. For information about supported platforms, see the Cisco Nexus 9000 and 3000 Upgrade and ISSU Matrix.

• The install all command is the recommended method for software upgrades because it performs configuration compatibility checks and BIOS upgrades automatically. In contrast, changing the boot variables and reloading the device bypasses these checks and the BIOS upgrade and therefore is not recommended.

  Note	For Nexus 9500 platform switches with -R line cards, you must save the configuration and reload the device to upgrade from NX-OS Release 7.0(3)F3(5) to 9.3(1). To upgrade from NX-OS Release 9.2(2) or 9.2(3), we recommend that you use the install all command.

• You can detect an incomplete or corrupt NX-OS software image prior to performing an upgrade by verifying the MD5, SHA256 or SHA512 checksum of the software image. To verify the MD5 checksum of the software image, run the show file bootflash:<IMAGE-NAME>md5sum command and compare the resulting value to the published MD5 checksum for the software image on the Software Download website. To verify the SHA512 checksum of the software image, run the show file bootflash:<IMAGE-NAME>sha512sum command and compare the resulting value to the published SHA512 checksum for the software image on the Software Download website.

• The install all command is the recommended method for software upgrades because it performs configuration compatibility checks and BIOS upgrades automatically. In contrast, changing the boot variables and reloading the device bypasses these checks and the BIOS upgrade and therefore it is not recommended.

EPLD

• ISSU supports EPLD image upgrades using install all nxos <nxos-image> epld <epld-image> command, during disruptive system (NX-OS) upgrade.

• To perform an EPLD upgrade after an ISSU upgrade from NX-OS Release 7.x to NX-OS Release 9.3(x), before starting the EPLD upgrade, add the copy run start command.

• ISSU is not supported if EPLD is not at NX-OS Release 7.0(3)I3(1) or later.

Release specific

• Upgrade from release 10.4(6)F to 10.5(1)F, 10.5(2)F, or 10.5(3)F releases is not supported and can result in configuration loss or its corruption. To upgrade from 10.4(6)M or 10.4(7)M release to 10.6(x) releases, the recommended path is to first upgrade to 10.5(4)M and then to 10.6(x). See CSCwr21007 in the Cisco Nexus 9000 Series NX-OS Release Notes, Release 10.4(6)M and Release 10.4(7)M.

• When ISSU is executed from NX-OS Release 10.2.1 or 10.2.2 to 10.2.3 or greater 10.2.x releases, BIOS upgrade is not supported for N9K-X9716D-GX, N9K-C9504-FM-G, and N9K-C9508-FM-G cards. To upgrade BIOS after ISSU is completed, use any one of the following options:

  • Execute these commands:

    1. install all nxos <image>

    2. install all bios-force

  • Use the bios-force option directly: install all nxos <image> bios-force .

• While performing an ISSU from NX-OS Release 9.3(5), 9.3(6), 9.3(7), 10.1(1), or 10.1(2) to NX-OS Release 10.2(1) or higher release, ISSU is blocked.

• ISSU is blocked when the delay configuration is present in track list Boolean/weight.

• If the IPv6 ND timeouts during ISSU, then the IPv6 BFD session may flap after the ISSU.

• When you upgrade from NX-OS Release 7.0(3)I7(1), 7.0(3)I7(2) or 7.0(3)I7(3) to NX-OS Release 10.2(x), the upgrade fails with below error message:

  switch(config)# install all nxos bootflash:nxos64-cs.10.2.3.F.bin
  Installer will perform compatibility check first. Please wait.
  Installer is forced disruptive
  Verifying image bootflash:/nxos64-cs.10.2.3.F.bin for boot variable "nxos".
  [####################] 100% -- SUCCESS
  Verifying image type.
  [####################] 100% -- SUCCESS
  Preparing "nxos" version info using image bootflash:/nxos64-cs.10.2.3.F.bin.
  [####################] 100% -- SUCCESS
  Preparing "bios" version info using image bootflash:/nxos64-cs.10.2.3.F.bin.
  [####################] 100% -- SUCCESS
  Pre-upgrade check failed. Return code 0x40930076 (Parallel downgrade to target version is not supported). <<<<<

  • Make sure that you follow below procedure to upgrade to 10.2(x) release:

    • Upgrade from 7.0(3)I7(1), 7.0(3)I7(2) or 7.0(3)I7(3) to 7.0(3)I7(5) and above code or 9.x code

    • Upgrade from 7.0(3)I7(5) or 9.x code to 10.2(x) code

• When upgrading directly to NX-OS Release 10.1(x) from any release prior to 7.0(x), the upgrade is disruptive. For a non-disruptive upgrade, an intermediate upgrade to NX-OS Release 9.x is required. We recommend upgrading to the latest release of NX-OS Release 9.3(x) as an intermediate hop for the upgrade. For information about the supported upgrade paths, see the Cisco Nexus 9000 and 3000 Upgrade and ISSU Matrix.

• When upgrading from NX-OS Release 7.0(3)I6(1) or 7.0(3)I7(1) to NX-OS Release 10.1(x), if the Nexus 9000 Series switches are running vPC and they are connected to an IOS-based switch via Layer 2 vPC, there is a likelihood that the Layer 2 port channel on the IOS side will become error disabled. The workaround is to disable the spanning-tree etherchannel guard misconfig command on the IOS switch before starting the upgrade process.

  Once both the Nexus 9000 Series switches are upgraded, you can re-enable the command.

• If you are upgrading from NX-OS Release 7.0(3)I5(2) to NX-OS Release 10.1(x) by using the install all command, BIOS will not be upgraded due to CSCve24965. When the upgrade to NX-OS Release 10.1(x) is complete, use the install all command again to complete the BIOS upgrade, if applicable.

• When upgrading to NX-OS Release 10.1(x) from 7.0(3)I2(x) or before and running EVPN VXLAN configuration, an intermediate upgrade to 7.0(3)I4(x) or 7.0(3)I5(x) or 7.0(3)I6(x) is required.

• When upgrading from NX-OS Release 9.2(4) or earlier releases to NX-OS Release 9.3(4) or later, running configuration contains extra TCAM configuration lines. You can ignore these extra lines as they do not have an effect on the upgrade and configuration.

• When performing an ISSU from NX-OS Release 9.3(1) or 9.3(2) to NX-OS Release 9.3(3) or later, ensure that the features with user-defined ports, such as <ssh port>, are within the prescribed port range. If the port range is incorrect, follow the syslog message recommendation. For more information about the port range, see Cisco Nexus 9000 Series NX-OS IP SLAs Configuration Guide.

• For any prior release version upgrading to NX-OS Release 9.3(5) using ISSU, if the following logging level commands are configured, they are missing in the upgraded version and must be reconfigured:

  • logging level evmc value

  • logging level mvsh value

  • logging level fs-daemon value

• For any prior release version upgrading to NX-OS Release 9.3(6) using ISSU, if the following logging level commands are configured, they are missing in the upgraded version and must be reconfigured:

  • logging level evmc value

  • logging level mvsh value

• The limitation that applies to software upgrades from 7.0(3)I5 to 10.1(x) or 9.2(3) to 10.1(x) is:

  If you have the same NetFlow configuration in both VLAN and SVI, you must remove the NetFlow flow monitor from the VLAN configuration prior to the upgrade. Once upgraded, reconfigure NetFlow by creating a new flow monitor and adding it to the VLAN configuration. Failure to perform these steps results in error messages and the inability to modify the VLAN NetFlow configuration in the upgraded software.

• When upgrading from NX-OS Releases 7.0(3)I4(8), 7.0(3)I5(3), and 7.0(3)I6(1) to NX-OS Release 10.1(x) results in a disruptive upgrade. If syncing images to standby SUP failed during the disruptive upgrade from NX-OS Releases 7.0(3)I4(8), 7.0(3)I5(3,) or 7.0(3)I6(1) to 10.1(x), you should manually copy the image to the standby SUP and perform the disruptive upgrade.

Switch specific

• During an ISSU on a Nexus 9300 Series switch, all First-Hop Redundancy Protocols (FHRPs) will cause the other peer to become active if the node undergoing the ISSU is active.

• Non-disruptive ISSU is not supported on interfaces with 2.5G or 5G speed on N9K-C93108TC-FX3P platform. For more information, see CSCwq38959.

• Beginning with NX-OS Release 10.2(2)F, FCoE/FC NPV is supported on N9K-C9336C-FX2-E platform switches.

  ISSU with with FCoE (Fiber Channel over Ethernet)/FC (Fiber Channel) NPV (N-port Virtualization) is supported on some Nexus 9000 switches. An ISSU allows you to upgrade the device software while the switch continues to forward traffic. You can perform an in-service software upgrade (ISSU), also known as a nondisruptive upgrade, for some Nexus 9000 switches. The default upgrade process is disruptive. Using the nondisruptive option helps ensure a nondisruptive upgrade.

  Fibre Channel N-port Virtualization (NPV) can co-exist with VXLAN on different fabric uplinks but on same or different front panel ports on the Nexus 93180YC-FX, N9K-C9336C-FX2-E, and N9k-C93360YC-FX2 switches.

• After performing ISSU on Nexus 9300 platform switches, you may see the MTS_OPC_CLISH message on the vPC peers. MTS_OPC_CLISH is the last MTS code that is sent from the back-end component to the VSH to specify the end of the show command output.

  If the user executes a show command that produces more output and keeps the session on for more than 3 minutes, the following warning message may be displayed on the console. As a workaround, you can set the terminal length as 0 using the terminal length 0 command or the show <command> | no-more option.

  
  --More--2018 Jun 5 19:11:21 Th-agg1 %$ VDC-1 %$ Jun 5 19:11:20 %KERN-2-SYSTEM_MSG: [12633.219113] 
  App vsh.bin on slot 1 vdc 1 SUP sap 64098(cli_api queue) did not drop MTS_OPC_CLISH with 
  msg_id 0x675ecf from sender sap 64132(NULL) in 180 sec, contact app owner - kernel
  

  
  (config)# show ip mroute detail 
  IP Multicast Routing Table for VRF "default"
  
  Total number of routes: 4801
  Total number of (*,G) routes: 2400
  Total number of (S,G) routes: 2400
  Total number of (*,G-prefix) routes: 1
  
  (*, 225.0.0.1/32), uptime: 00:09:32, igmp(1) pim(0) ip(0) 
    RPF-Source: 10.10.10.3 [11/110]
    Data Created: No
    VPC Flags
      RPF-Source Forwarder
    Stats: 15/720 [Packets/Bytes], 0.000   bps
    Stats: Inactive Flow
    Incoming interface: Ethernet1/1, RPF nbr: 12.0.0.2
    LISP dest context id: 0  Outgoing interface list: (count: 1) (bridge-only: 0)
      Vlan2001, uptime: 00:09:32, igmp (vpc-svi) 
  
  
  (60.60.60.2/32, 225.0.0.1/32), uptime: 00:09:31, ip(0) mrib(1) pim(0) 
    RPF-Source: 60.60.60.2 [20/110]
    Data Created: Yes
    VPC Flags
  --More--2018 Jun  5 19:11:21 Th-agg1 %$ VDC-1 %$ Jun  5 19:11:20 %KERN-2-SYSTEM_MSG: [12633.219113] App vsh.bin on slot 1 vdc 1 SUP 
  sap 64098(cli_api queue) did not drop MTS_OPC_CLISH with msg_id 0x675ecf from sender sap 64132(NULL) in 180 sec, 
  contact app owner - kernel
  

  There is no functionality impact or traffic loss due to this issue. All the MTS messages are drained once the show command displays the complete output, the user enters CTRL+c, or the session gets closed.

• Before enabling the FHS on the interface, we recommend that you carve the ifacl TCAM region on Nexus 9300 and 9500 platform switches. If you carved the ifacl TCAM region in a previous release, you must reload the system after upgrading to NX-OS Release 10.1(x). Uploading the system creates the required match qualifiers for the FHS TCAM region, ifacl.

• Before enabling the FHS, we recommend that you carve the ing-redirect TCAM region on Nexus 9200 and 9300-EX platform switches. If you carved the ing-redirect TCAM region in a previous release, you must reload the system after upgrading to NX-OS Release 10.1(x). Uploading the system creates the required match qualifiers for the FHS TCAM region, ing-redirect.

• When upgrading from Nexus 94xx, 95xx, and 96xx line cards to Nexus 9732C-EX line cards and their fabric modules, upgrade the NX-OS software before inserting the line cards and fabric modules. Failure to do so can cause a diagnostic failure on the line card and no TCAM space to be allocated. You must use the write_erase command followed by the reload command.

• Upgrading from NX-OS Release 7.0(3)I1(2), Release 7.0(3)I1(3), or Release 7.0(3)I1(3a) requires installing a patch for Nexus 9500 platform switches only. For more information on the upgrade patch, see the Install upgrade patches.

Disruptive and non-disruptive ISSU

• On performing a non-disruptive ISSU from NX-OS Release 7.0(3)I6(1) to any higher version, a traffic loss might occur based on the number of VLANs configured. To avoid traffic loss, it is recommended to increase the routing protocol's graceful restart timer to higher value. The recommended value of the graceful restart timer is 600 seconds. You can further increase or decrease this value based on the scale of the configuration.

• Beginning with NX-OS Release 10.2(3)F, non-disruptive ISSU is supported for VPC fabric peering on all Nexus 9300-X TORs. Both standard and enhanced non-disruptive upgrades are supported. Note that ISSU should be started or triggered when there is no failure. An example for failure would be one of the VPC legs is down.

• The recommended routing protocol graceful restart timer is 600 seconds and nve source-interface hold-down-time is 400 seconds.

• It is recommended to set disable-fka on VFC interfaces in E or F mode, when invoking ND native ISSU on switch mode testbed. If not, it can be disruptive.

• Beginning from NX-OS Release 10.2(8)M onwards, Nexus 9300-FX3 supports non-disruptive upgrade.

• Beginning with NX-OS Release 10.2(3)F, for switches that are in LXC mode and for non-desruptive upgrade, a new option skip-kernel-upgrade is added to install command.

• MPLS strip, GRE strip, and any underlying ACL configuration is not ISSU compatible when you perform ND ISSU to NX-OS Release 10.2(2)F from a previous release.

  After ND ISSU to NX-OS Release 10.2(2)F or 10.2(3)F from a previous release, post GRE strip dot1q tunnel VLAN_tag might be missing. Workaround for this issue is to remove and add port ACL from L2 interfaces for GRE strip enabled interface.

• For a device that is running on NX-OS Release 10.1(2), 10.2(1)F, and 10.2(2)F, ND-ISSU is not supported if Layer 2 sub-interfaces are configured.

• When performing ND ISSU using BGP non-default hold timers, ensure that the BGP graceful-restart timer is reasonably long enough, for example, 180 seconds.

• If there is a VRF scale, for a non-disruptive ISSU under each VRF, you must configure graceful restart timer to 300 seconds.

• OpenFlow and LACP fast timer rate configurations are not supported for Non-Disruptive ISSU.

• Beginning with NX-OS Release 9.3(5), standard, nondisruptive ISSU, on switches that are configured with uRPF, is supported on:

  • Nexus 9300-EX platform switches

  • Nexus 9300-FX/FX2 platform switches, and

  • Nexus 9300-GX platform switches

  Note	Prior to NX-OS Release 9.3(5), if any of the above switches were configured with uRPF, standard, non-disruptive ISSU was not supported.

Enhanced mode

• ND ISSU can be performed in LXC mode in two methods:

  • ND ISSU in LXC mode - Switchover-based ISSU that is similar to EOR. Second SUP is brought up in new container and switchover is done. The second SUP now becomes the new active. There is no change to the kernel.

  • Fallback ND LXC ISSU - This is only done when the above switchover-based ISSU cannot be done (SRG Kernel incompatible or less memory). The kernel is upgraded.

  • skip-kernel-upgrade option will force ND ISSU in LXC mode - Switchover-based ISSU (even in case when running) and target kernels are incompatible.

• For switches that are in LXC boot mode, when upgrading image from NX-OS Release 10.1(1) or earlier releases to NX-OS Release 10.2(x), the upgrade will be disruptive.

Feature specific

• While upgrading from NX-OS releases prior to 10.2(2)F to 10.2(2)F or later releases, configure the mode tap-aggregation command before attaching TapAgg ACLs on Layer 2 interface.

• If you upgrade from a NX-OS release that supports the CoPP feature to a NX-OS release that supports the CoPP feature with additional classes for new protocols, you must either run the setup utility using the setup command or use the copp profile command for the new CoPP classes to be available. For more information on these commands of configuring control plane policing, see the Cisco Nexus 9000 Series NX-OS Security Configuration Guide.

• Beginning with NX-OS Release 10.1(2), CoPP is supported on N9K-X9624D-R2 and N9K-C9508-FM-R2 platform switches.

• Beginning with NX-OS Release 10.1(2), RACL is supported on N9K-X9624D-R2 and N9K-C9508-FM-R2 platform switches.

• Beginning with NX-OS Release 10.1(1), during the disruptive upgrade to the 64-bit image or a downgrade from 64-bit to 32-bit image, if feature ITD is enabled, refer to Guidelines and Limitations for ITD in the Cisco Nexus 9000 Series NX-OS Intelligent Traffic Director Configuration Guide, Release 10.1(x), if the upgrade or downgrade proceeds with an ASIC reload.

• Beginning with NX-OS Release 10.1(1), Fs_daemon does not support snmpwalk on devices with more than 5000 files. When performing snmpwalk on a device with more than 5000 files, the error resourceUnavailable (This is likely a out-of-memory failure within the agent) is an expected behavior.

• When you upgrade a Nexus 9000 device to NX-OS Release 10.1(x), if a QSFP port is configured with the manual breakout command and is using a QSA, the configuration of the interface Ethernet 1/50/1 is no longer supported and must be removed. To restore the configuration, you must manually configure the interface Ethernet 1/50 on the device.

• When upgrading from NX-OS Release 9.3(3) to NX-OS Release 9.3(6) or later, if you do not retain configurations of the TRM enabled VRFs from NX-OS Release 9.3(3), or if you create new VRFs after the upgrade, the auto-generation of ip multicast multipath s-g-hash next-hop-based command, when feature ngmvpn is enabled, will not happen. You must enable the command manually for each TRM enabled VRF. For the configuration instructions, see the Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide.

• Upgrading from NX-OS Release 9.3(1), 9.3(2) or 9.3(3) to a higher release, with Embedded Event Manager (EEM) configurations that are saved to the running configuration, may cause a DME error to be presented. The error is in the output of the show consistency-checker dme running-config enhanced command, specifically, the event manager commands. If this error occurs, delete all EEM applet configurations after completing the ISSU, then reapply the EEM configurations.

• When redistributing static routes, NX-OS requires the default-information originate command to successfully redistribute the default static route starting in 7.0(3)I7(6).

FEX

• Beginning with NX-OS Release 10.2(2)F, ND ISSU is supported for FEX and you need to re-adjust the BGP graceful-restart restart time command for the upgrade to work non-disruptively. This must be done for each FEX upgrade one-by-one.

  The following example shows the time taken to re-adjust bgp-graceful restart-time for each non-disruptive FEX upgrade.

  In the Non-disruptive upgrade with FEX, each FEX will upgrade taking about 90 secondss 
  (1.5 minutess) sequentially (one-by-one and not a parallel upgrade). 
    Total non-disruptive upgrade time for all FEX = No. of fex * time taken per fex
                                       For 10 FEX = 10 * 90
                                                  = 900 seconds or 15 minutes

• When you upgrade a Nexus 9000 switch from NX-OS Release 7.x with an attached FEX in straight-through mode to 9.x and then to 10.x, the FEX Layer 2 Host Interface (HIF) configuration can be lost after upgrading to a 10.x Release. This occurs due to a design change in handling Layer 2 FEX HIF ports at boot time from Release 9.x to 10.x.

  Note	The issue occurs only for FEX connected in a straight-through mode and not for dual-homed (A-A) mode.

  To resolve this, run the following non-intrusive commands before upgrading the switch from 9.x to 10.x:

  1. Apply no switchport only on all Layer 3 (L3) physical and Layer 3 (L3) port-channel interfaces. For example,

     switch(config)# interface e1/1
     switch(config-if)# no switchport
     

  2. Configure system default switchport globally and save the configuration. For example,

     switch(config)# system default switchport
     switch(config)# copy r s
     

  Note	The issue does not occur if: the switch was originally booted in 9.x with an attached FEX and then upgraded to 10.x. the switch was upgraded from 7.x to 9.x without an attached FEX, and the FEX was added later in 9.x before upgrading to 10.x.

• An upgrade that is performed via the install all command for NX-OS Release 7.0(3)I2(2b) to Release 10.1(x) might result in the VLANs being unable to be added to the existing FEX HIF trunk ports. To recover from this, the following steps should be performed after all FEXs have come online and the HIFs are operationally up:

  1. Enter the copy run bootflash:fex_config_restore.cfg command at the prompt.

  2. Enter the copy bootflash:fex_config_restore.cfg running-config echo-commands command at the prompt.

• In NX-OS Release 7.0(3)I6(1) and earlier, performing an ASCII replay or running the copy file run command on a FEX HIF configuration requires manually reapplying the FEX configuration after the FEX comes back up.

FC/FCoE NPV

• Beginning with NX-OS Release 10.1(1), ISSU is supported on FC/FCoE switch mode on Nexus 93360YC-FX2. For more information about the FC/FCoE switch mode and supported hardware, see the Cisco Nexus 9000 Series NX-OS SAN Switching Configuration Guide.

• Beginning with NX-OS Release 10.1(1), enhanced ISSU is supported on FC/FCoE switch mode for Nexus 93180YC-FX and 93360YC-FX2 switches. For more information about the FC/FCoE switch mode and supported hardware, see the Cisco Nexus 9000 Series NX-OS SAN Switching Configuration Guide.

• Beginning with NX-OS Release 10.1(1), Enhanced ISSU is supported on FC/FCoE NPV mode for Nexus 93180YC-FX and 93360YC-FX2 switches. For more information about the FC/FCoE NPV mode and supported hardware, see the Cisco Nexus 9000 Series NX-OS FC-NPV and FCoE NPV Configuration Guide.

Unsupported PIDs

The table displays the list of unsupported PIDs from various NX-OS Releases.

• The only supported method of downgrading a Nexus 9000 Series switch is to utilize the install all command. Changing the boot variables, saving the configuration, and reloading the switch is not a supported method to downgrade the switch.

  Disable the Guest Shell if you need to downgrade from NX-OS Release 9.3(x) to an earlier release.

  • Performing an ISSU downgrade from NX-OS Release 9.3(x) to Release 7.0(3)I4(1) with an FCoE (Fiber Channel over Ethernet) NPV (N-port Virtualization) configuration causes the port channel to crash with a core file:

    [################ ] 38%2016 Apr 18 20:52:35 n93-ns1 %$ VDC-1 %$ %SYSMGR-2- SERVICE_CRASHED: Service "port-channel" (PID 14976) hasn't caught signal 11 (core will be saved)

  • ISSU (non-disruptive) downgrade is not supported

• On Nexus 9500 switches with N9508-E2 Fabric module, downgrade from any 9.x or 10.x supported releases to any unsupported releases of 7.x is not supported.

• When downgrading from the NX-OS Release 9.3(x) to earlier releases, any ACL with the statistics per-entry command enabled and applied as RACL needs the statistics per-entry command removed from the running configuration before downgrading. Otherwise, the interfaces on which this ACL is applied as a RACL will be error disabled after the downgrade.

• Prior to downgrading a Nexus 9500-series switch, with -FX or -FX+EX line cards, from NX-OS Release 10.1(x) to earlier releases (9.2(x) or 7.x), the TCAM region that applies to NetFlow (ing-netflow) should be carved to zero (0) using the following command:

  hardware access-list tcam region ing-netflow 0

  The configuration change is required because the default ing-netflow TCAM region in 9.3(1) and onwards is 512 while the default in 9.2(x) and earlier is 0.

• When downgrading from the NX-OS Release 10.1(x) to a release prior to 9.3(x), make sure that the ACL TCAM usage for ingress features does exceed the allocated TCAM space in the absence of the label sharing feature. Label sharing is a new feature in NX-OS Release 9.3(x). Otherwise, interfaces with RACLs that could not fit in the TCAM will be disabled after the downgrade.

• Software downgrades should be performed using the install all command. Changing the boot variables, saving the configuration, and reloading the switch is not a supported method to downgrade the switch.

• This limitation applies to Nexus platform switches that support Trust Anchor Module (TAM):

  The TACACS global key cannot be restored when downgrading from NX-OS Release 9.3(3) and higher to any earlier version. TAM was updated to version-7 in 9.3(3), but earlier NX-OS versions used TAM version-3.

• iCAM must be disabled before downgrading from Release 9.2(x) or Release 9.3(x) → 7.0(3)I7(1). Only Release 9.3(1) → Release 9.2(4) can be performed if iCAM is enabled.

• Beginning with NX-OS Release 9.3(3), new configuration commands exist for SRAPP (with sub-mode options for MPLS and SRTE). The SRAPP configuration on the switch running release 9.3(3) (or later) will not be present if the switch is downgraded to an earlier release.

• On devices with dual supervisor modules, both supervisor modules must have connections on the console ports to maintain connectivity when switchovers occur during a software downgrade. For more information about your specific chassis, see the relevant Hardware Installation Guide.

• NX-OS automatically installs and enables the guest shell by default. However, if the device is reloaded with a NX-OS image that does not provide guest shell support, the existing guest shell is automatically removed and a %VMAN-2-INVALID_PACKAGE message is issued. As a best practice, remove the guest shell with the guestshell destroy command before downgrading to an earlier NX-OS image.

• You must delete the switch profile (if configured) when downgrading from a NX-OS release that supports switch profiles to a release that does not. For more information, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide.

• Software downgrades are disruptive. In-service software downgrades (ISSDs), also known as non-disruptive downgrades, are not supported.

• When downgrading from NX-OS Release 9.3(3) or later to 7.0(3)I7(7), disable BFD for the BGP neighbor prefix peer using the no bfd command.

• While downgrading from the NX-OS Release 10.2(1)F or higher to an earlier release, the install all command is blocked when the delay config is present in track list Boolean/weight.

• While performing ISSD from NX-OS Release 10.2(3)F to NX-OS Release 10.2(2)F with epbr L2 applied on interfaces, remove the policies from interfaces before performing ISSD to avoid the duplicate tracks issue.

• Beginning with NX-OS Release 10.2(3)F, if you have configured the lldp chassis-id switch command, then you must disable the command before performing ISSD.

• Beginning with 10.2(3)F, although application of ePBR policy to access ports is supported, downgrading with this configuration is not recommended.

• During downgrade, where both source and target images support Type-6 encryption, while performing device reload, if ASCII replay is triggered without binary restore, primary key gets lost. The primary key must be reconfigured after device reload. Use the key config-key ascii command to reconfigure the primary key and avoid encryption issues. However, downgrade with binary restore retains the primary key after the reboot, provided both source and target images support Type-6 encryption.

  If you downgrade the system from an image that supports Type-6 encryption to an image that does not support Type-6 encryption, compatibility check fails.

Beginning with NX-OS Release 10.2(3)F, Nexus 9000 switches support a new command, show upgrade history details, which displays login details (user name/session ID).

View the NX-OS upgrade history details by entering the show upgrade history details command. The output displays user login details (user name/session ID) under LOGIN column on the switch along with upgrade history. Here is an sample output of the show upgrade history details command.

switch# show upgrade history details 
     TYPE          VERSION                                  DATE                  LOGIN                          STATUS                            
NXOS system image  10.2(3)                   21 Jan 2022 10:01:06     admin/10.30.216.212        Installation End
NXOS system image  10.2(3)                   21 Jan 2022 10:00:53     admin/10.30.216.212        Installation started
NXOS system image  10.2(3)                   21 Jan 2022 01:03:52     admin/10.30.216.212        Installation End