Operations Allowed During Mixed Versions on Cisco ACI Switches

Operations allowed during mixed versions on Cisco ACI switches

The Cisco Application Centric Infrastructure (ACI) fabric essentially has a requirement that all nodes (Cisco Application Policy Infrastructure Controller (APIC), leaf switches, and spine switches) should have the same software release or have a compatible software release, where the Cisco APIC nodes have the standard release format of x.y(z), and the leaf and spine switches have the switch-specific standard release format of 1x.y(z). For example, if the Cisco APIC nodes are on software release 4.1(1), the leaf switches and spine switches should be on the switch-specific compatible version of 14.1(1).

However, this could be a challenging requirement when attempting to upgrade the software for a huge Cisco ACI fabric with a large number of switch nodes, because you would usually split the switch nodes into several different groups (maintenance groups) in this situation, which would allow you to perform the upgrade one group at a time to avoid any service disruptions. Depending on the number of switch nodes or maintenance groups, and the validation process for network traffic, services, and applications, you would be able to upgrade some maintenance groups on one day, but you might have to wait to upgrade the remaining maintenance groups on another day.

To help users with this requirement, there are two options:

  • Enhanced Mixed Version Support – support all features, configurations and operations that are supported on the older version V1 of the mixed versions V1 and V2.

  • Limited Mixed Version Support – support some features, configurations and operations listed below.

Mixed version support conditions

Both Enhanced mixed version support and Limited mixed version support are available under the following conditions

  • Up to two versions at the same time. For example, V1 and V2 are supported. V1, V2 and V3 are not supported.

  • APICs must be always V2 while switches can be either V1 or V2 where V2 is the newer version.

  • Both switches in a vPC pair must be on the same version (both V1 or both V2)

Table 1. Limited mixed version support or Enhanced mixed version support

V1

V2

Supported Operations

2.2(x) or later

Any versions in the supported upgrade path in the matrix.

Limited Mixed Version Support

Version pairs specifically mentioned as Enhanced Mixed Version Support in the matrix.

Version pairs specifically mentioned as Enhanced Mixed Version Support in the matrix.

Enhanced Mixed Version Support

Enhanced mixed version support

Starting with APIC release 6.2(1), Enhanced mixed version support was introduced under the conditions mentioned in the Mixed version support conditions section.

When the conditions above are fulfilled, the Cisco ACI fabric supports all features, configurations and operations that are supported on the older version V1 of the mixed versions V1 and V2 where V1 is the older and V2 is the newer.

Enhanced mixed version support allows users to operate the fabric as if all versions are still running on V1 without limitations, even when APICs and some switches are running V2. There is no time limit for operating the fabric in enhanced mixed version mode. You can continue to run the fabric in a mixed-version state for as long as needed, not only during an upgrade window. While operating in this mode, the supported feature set on V1 nodes is limited to the capabilities of the lower software version. The system raises fault F4768 when a node is detected as incompatible with a feature, regardless of whether the feature is currently configured.

This can be helpful for situations such as the following:

  • Upgrading all switches requires multiple maintenance windows.

  • V2 is required only for a new switch model.

  • V2 is required only for a subset of switches.

  • The fabric needs to remain in a mixed-version state for an extended period of time.

Beginning with Cisco APIC release 6.2(2), the enhanced mixed version support is extended to allow certain V2 software features to be used during mixed-version operations. The availability of a V2 feature during mixed-version operations depends on the feature's compatibility scope. There are two types of feature scopes:

  • Node scope — A feature with node scope can be enabled and used on switch nodes that are running the newer software version (V2), even when other switch nodes in the fabric are still running the older software version (V1). Switch nodes that are still running V1 do not support this feature. The system raises a fault when a V1 node is detected as incompatible with a node-scoped feature, regardless of whether the feature is currently configured.

  • Fabric scope — A feature with fabric scope requires all switch nodes in the fabric to be running the same software version (V2) before the feature can be enabled. The system raises a fault on all affected switch nodes when a node is detected as incompatible with a fabric-scoped feature, regardless of whether the feature is currently configured.

Mixed version compatibility faults

When a V2 feature is configured in a mixed-version fabric and the feature is not supported on one or more nodes due to version incompatibility, Cisco APIC raises a fault to notify the administrator. The fault provides details about the feature, the scope, and the affected nodes.

The following table describes the fault that is raised for mixed version feature incompatibility.

Table 2. Mixed version feature compatibility faults

Fault Code

Severity

Cause

Description

F4768

Minor

config-unsupported

Raised when a feature is incompatible with one or more nodes in the fabric due to a version mismatch during mixed-version operations. The fault description indicates the feature name, the feature scope (Node or Fabric), and the affected node or nodes.

Guidelines and limitations

  • Only V1 features are supported on switch nodes that are running V1, regardless of whether other nodes in the fabric are running V2.

  • For node-scoped features, the feature is supported only on the switch nodes that are running V2. The system raises fault F4768 when a V1 node is detected as incompatible with a node-scoped feature, regardless of whether the feature is currently configured.

  • For fabric-scoped features, all switch nodes in the fabric must be upgraded to V2 before the feature can be enabled. If any V1 switch nodes exist in the fabric, the system raises Fault F4768 on all affected nodes.

  • To verify the supported version combinations for mixed-version operations, refer to the software operating matrix for your specific release versions.

  • For information about the enhanced mixed version scope for a specific feature, refer to the release notes for the release in which the feature was introduced, or refer to the configuration guide for that feature.

Downgrading only switches using enhanced mixed version support

Enhanced Mixed Version enables switches to operate on an older software version (V1) while APICs continue running a newer version (V2). However, the standard policy-driven downgrade procedure—managed through APIC—requires downgrading the APICs first. As a result, even if a V1 and V2 combination supports EMV, you cannot downgrade only the switches to V1 using the regular procedure while APICs remain on V2.

If you need to downgrade only the switches to the older version due to an emergency, and the V1 and V2 releases support Enhanced Mixed Version, use the RMA process described below:

Switch-only downgrade procedure:

  1. Delete the existing switch update group for the switches you need to downgrade.

  2. Enable Auto Firmware Update on Switch Discovery and set the target firmware version to V1.

  3. Remove the switch using the Decommission & Remove option, as outlined in the Troubleshoot ACI Fabric Discovery – Device Replacement guide.

  4. The switch will initialize and reboot.

  5. Discover and register the switch again with the same node ID and name.

  6. The switch will automatically boot up with the V1 image during discovery using Auto Firmware Update.

Limited mixed version support

Limit Mixed Version Support has been available since release 2.2(1). When the conditions mentioned in Mixed Version Support Conditions are fulfilled, some limited operations listed below can be performed even when all Cisco ACI switches have yet to be upgraded to the same version if the operation is regarding a feature that was already supported on the older (from) version.


Note

Operations listed below are only for upgrade scenarios. Those are not applicable and not supported when downgrading the fabric, that is when APICs are running an older version than the switches. Downgrading switches while APICs are still running the newer version is not supported.

Table 3. Supported operations with limited mixed version support for upgrades from release 2.2(x)

Features

Operations

Troubleshooting

  • Exporting configuration

  • Collecting techsupport

Physical network

Reboot, cable replacement etc.

Other

Policy changes for features introduced prior to the major release.

*This operation is supported only when the upgrade is within the same release train. For example, an upgrade from 3.2(5d) to 3.2(5f), where the releases are still part of the 3.2(5) release train, but the upgrade occurs between the d and the f versions of that release train.

Table 4. Supported operations with limited mixed version support for upgrades from release 2.3(x) or later

Features

Operations

Contracts

  • Creating, updating, and deleting filters, subjects, and contracts.

  • Exporting and importing contracts.

  • Adding and deleting provided and consumed contracts in relationship with EPGs.

  • Adding and deleting provided and consumed contracts in vzAny.

Endpoint group

  • Creating and deleting EPGs.

  • Adding and deleting VMM, physical, Layer 2 external, and Layer 3 external domain association.

  • Adding, deleting, and updating static port assignment and statically linking with the node.

  • Moving an end point from one EPG to another EPG.

  • Moving an end point from uSeg EPG to base EPG.

Microsegmentation

Adding and updating uSeg EPG

vMotion

vMotion across a leaf switch.

VM operation

On and off of virtual machines.

Bridge domain

Creating, updating, and deleting bridge domains.

VMM Domain

The following operations are supported only in VMware vDS and Cisco AVS.

  • Creating and deleting VMM domains.

  • Adding and updating VLAN pools.

  • Adding and deleting multicast pools.

  • Adding and updating VMware vCenter.

  • Adding and updating vSwitch policies.

Layer 2 or Layer 3 Out

Adding, updating, and deleting Layer 2 external and Layer 3 external domains.

Access Policy

Adding, updating, and deleting switch policies, interface policies, policy group, Attached Entity Profiles (AEP).

Troubleshooting

  • Adding, updating, and deleting SPAN configuration.

  • Adding, updating, and deleting syslog server.

  • Exporting configuration

  • Collecting techsupport

Physical network

  • Enabling and disabling port status.

  • On and off of a physical server.

  • Moving physical server within and across leaf switches.

  • Reloading spine switches and leaf switches.

    When the reload is stateless, meaning that it is a clean reload in which the configuration is wiped and pulled again from the Cisco APICs, the switches must have the same release as the Cisco APICs.

  • Reloading a spine switch line card, Fibre Channel card, CS card, and SUP card.

  • Decommissioning spine switches and leaf switches.

  • Removing spine switches and leaf switches using the Remove from Controller option.

  • Registering a new spine switch and leaf switch

    The new switch must have the same release as the Cisco APICs.

  • Adding and deleting a virtual port channel domain

  • Flapping primary link, secondary link, and all the links in the virtual port channel.

  • Flapping all the port channel links, flapping one link in the port channel, flapping NIF ports on FEX, and flapping front panel ports on the leaf switch.

  • Replace cables.

Fabric Policy

  • Adding, updating, and deleting NTP server, SNMP, BGP route reflector, Layer 2 MTU policy.

  • Updating Cisco APIC connectivity preferences.

Other

Policy changes for features introduced prior to the major release*

*This operation is supported only when the upgrade is within the same release train. For example, an upgrade from 3.2(5d) to 3.2(5f), where the releases are still part of the 3.2(5) release train, but the upgrade occurs between the d and the f versions of that release train.

Guidelines and limitations for mixed versions on Cisco ACI-mode switches

This topic provides definitions for Cisco APIC releases and describes operational limitations when using mixed software versions on Cisco ACI-mode switches.

  • The following definitions are used to describe a Cisco APIC release:

    • A Cisco APIC major release contains support for new software features and additional hardware updates. Examples of major releases include 2.2(1n) and 2.1(1h).

    • A Cisco APIC minor or maintenance release (MR) contains the bugs fixes and patches from the existing release. Examples of minor or maintenance releases include 2.0(1m) and 2.0(2f).

    • A Cisco APIC patch release contains fixes for specific defects. Examples of patch releases include 2.1(1h) and 2.1(1i).

  • Prior to Cisco APIC release 3.0, a red banner warning is displayed, informing you of version differences on the Cisco ACI nodes in the fabric. This banner warning has been removed since Cisco APIC release 3.0.

  • If your Cisco APIC is running the 5.2(4) release or later and your switches are running an Cisco ACI -mode switch software release that is earlier than 15.2(4), a vPC domain's interfaces become suspended/down when its peer node is decommissioned. Graceful insertion (maintenance mode) of the vPC peer node will also lead to the same issue because the switch is automatically decommissioned, rebooted, and recommissioned. In the following example scenarios, you will encounter this issue: