Field Notice: FN74371 - Cisco Nexus 9000 and 3000 Series Configuration Corruption Risk on Upgrade/Downgrade with Specific Releases - Software Upgrade Recommended

Upgrades or downgrades between certain releases of Cisco NX-OS Software for Cisco Nexus 3000 and Nexus 9000 Series platforms may fail or cause configuration corruption. The issue occurs when initially upgrading or downgrading between Release 10.4(6)M or a later 10.4(x) release and Release 10.5(1)F, 10.5(2)F, or 10.5(3)F. Disruptive and non-disruptive upgrades are affected, but workarounds are available.

This risk only exists between specific releases, as detailed in the following tables. 

Upgrading from a release listed in the left column to one of the releases in the same row in the right column may cause this issue.

Downgrading from one of the releases listed in the left column to the release in the same row in the right column may cause this issue.

The following upgrading scenarios will not cause this issue and are safe:

• Upgrades from any 10.4(x) release to Release 10.5(4)M or later
• Upgrades from Release 10.4(1), 10.4(2), 10.4(3), 10.4(4), or 10.4(5) to any 10.5(x) release

The following downgrading scenarios will not cause this issue and are safe:

• Downgrades from release 10.5(4) or later to any 10.4(x) release
• Upgrades to 10.5(4)M followed by downgrades to Release 10.4(6)M or any later 10.4(x) release.

If the issue has occurred after an upgrade from Release 10.4(6)M or a later 10.4(x) release to Release 10.5(1)F, 10.5(2)F, or 10.5(3)F, you cannot address the issue by downgrading to 10.4(6) or an earlier 10.4 release. You must take one of the following actions:

• Do an ASCII reload. Some configurations may be missing after the reload and may need to be applied (pasted).
• Do a write erase, reload, then reconfigure.

No non-disruptive recovery procedure is available if the issue has already occurred. See Workaround/Solution section for additional information.

If the upgrade from Release 10.4(6)M or a later 10.4(x)M sequential release was not performed and you are not having the issue, downgrade from 10.5(1, 2 or 3)F to 10.4(1,2,3,4, or 5), then, if needed, upgrade to 10.4(6)M or a later 10.4(x)M sequential release.

This issue can affect Cisco Nexus 3000 or 9000 platforms (TOR and modular chassis) when performing disruptive or non-disruptive upgrades or downgrades between the affected releases only. See the table in Problem Description.

When this issue occurs, the upgrade or downgrade process will exit without proceeding, or configuration corruption will occur after the upgrade or downgrade. The config corruption may show up as garbled VLAN names, missing configurations, spurious segment IDs, missing items from operational VLAN listings or switch virtual interface (SVI) listings, numbering issues, and other configuration errors. SVIs for affected VLANs may be in a down state, and startup-config parsing for VLANs may fail. Segment IDs appear even though related features (such as VXLAN/VN-segment) are not enabled.

After disruptive or non-disruptive upgrades or disruptive downgrades, configuration will be corrupted and not complete. Examples of known symptoms include:

• VLAN Corruption: VLANs show garbled names and spurious segment IDs, and they are missing from operational VLAN listings.
• SVI Issues: SVIs for affected VLANs are in down state.
• TLV Errors: Startup-config parsing for VLANs fails with TLV errors.
• Invalid Segment IDs: Segment IDs appear even though VXLAN/VN-segment features are not enabled.
• Configuration Instability: Running-config for VLANs changes automatically over time without user action.
• Upgrade Failure: In some cases, the upgrade is not possible because the system does not progress after the compatibility check and exits without continuing.

Important: Reloading will not recover the devices from this issue, nor will consequent downgrade to the earlier release. The only way to recover is to do an ASCII reload or write erase and reconfigure the device. See Workaround/Solution section.

Recommended Solution

Upgrade from 10.4(6)M, 10.4(7)M and 10.4(continued higher releases)M directly to 10.5(4)M release or 10.5(continued higher releases)M and skip 10.5(1)F, 10.5(2)F, and 10.5(3)F.

Recovery Options (If Already Affected)

Option 1: ASCII Reload

Performing ASCII reload resolves the issue and restores missing configuration:

switch# reload ascii
!!!WARNING! This command will erase binary configuration
in this VDC and reboot the system with ascii configuration.
Do you wish to proceed anyway? (y/n)  [n] y

Note: The configuration needs to be verified after the ASCII reload, and restored if needed, because part of config may be missing. For more information, see the Understand the Reload ASCII Command TechNote.

Option 2: Write Erase and Restore

After performing an upgrade or downgrade on an affected path, write erase and reload the device and recover the configuration from backup. After the reload, the switch will boot up with no configuration, so be sure to set up a way to access it before doing the reload (such as console access or POAP).

switch# write erase
Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n)  [n] y
switch# reload
This command will reboot the system. (y/n)?  [y]

Avoidance: Alternative Upgrade Path to Get to 10.5(1,2,3)F

If upgrade from 10.4(6,7,8, etc.)M to 10.5(1,2 or 3)F is absolutely necessary, the following upgrade path can be taken (disruptive upgrade):

10.4(6)M → 10.5(4)M → 10.5(3)F

Cisco Nexus 9000 or 3000 platforms (TOR or modular chassis) are at risk of this upgrade/downgrade problem only if one of the following is true:

• Currently running Cisco NX-OS Release 10.4(6)M, 10.4(7)M, 10.4(8)M or a later 10.4(x)M release and plan to do a disruptive or non-disruptive upgrade to Release 10.5(1)F, 10.5(2)F, or 10.5(3)F
• Currently running Cisco NX-OS Release 10.5(1)F, 10.5(2)F, or 10.5(3)F and plan to downgrade to Release 10.4(6)M, 10.4(7)M, 10.4(8)M or a later 10.4(x)M release

To check current Cisco NX-OS Software release, use the show version command. Look for NX-OS release number in the output to find your current release. Check the filename for your target release.