Guidelines for ACI switch upgrades and downgrades

Rule 1 – Divide your leaf and spine switches into at least two groups

For example:

• Group ODD: leaf 101, leaf 103, spine 1001

• Group EVEN: leaf 102, leaf104, spine 1002

Rule 2 – Determine how spine switches should be grouped

• Always keep at least one MP-BGP route reflector (RR) spine switch up and running in each pod.

• Always keep at least one spine switch with IPN connectivity up and running in each pod.

• Never perform a graceful upgrade for a spine switch if the given pod has only one spine switch (in the case of multi-pod).

  See Graceful Upgrade or Downgrade of ACI Switches for details.

For example:

Where:

• RR means a Route Reflector spine switch

• IPN means a spine switch connected to IPN

Rule 3 – Determine how leaf switches should be grouped

• Always keep one of the leaf switches in the same vPC pair up and running

• Always keep one of the leaf switches connected to each Cisco Application Policy Infrastructure Controller ( APIC ) up and running

For example:

Where:

• vPC xx means one vPC pair

• APICx means a leaf switch connected to the Cisco APIC

Rule 4 – Understand the concurrent capacity in switch update groups

General

• Each update/maintenance group should contain a maximum of 80 remote leaf switches.

• The concurrent capacity (switches that are upgraded or downgraded simultaneously) decides how many switches should be upgraded or downgraded simultaneously within the same update/maintenance group. However, we recommend that you create separate update groups to upgrade or downgraded switches on different schedules instead of relying on the concurrent capacity setting because the concurrent capacity setting doesn’t let you manage which switches in the same group are to be upgraded or downgraded at the same time.

• If both leaf nodes in the same vPC pair are in the same switch upgrade or downgrade group, only one of the leaf nodes is upgraded or downgraded at a time regardless of the concurrent capacity.

• Starting from Cisco APIC release 4.1(1), when graceful upgrade or downgrade is enforced and there are no other operational spine switches in the same pod, the upgrade or downgrade is rejected regardless of the concurrent capacity setting.

Prior to Cisco APIC release 4.2(5):

• Even in the same update group, switches are upgraded or downgraded only one pod at a time.

• The default concurrent capacity per group is 20.

If you have more than 20 switches in the same group, you can use upgrade scheduler to change the capacity to unlimited.

See the Upgrading the Leaf and Spine Switch Software Version section for details:

• Releases prior to 4.x: Upgrade or downgrade leaf and spine switches through APIC running prior to release 4.x

• Releases 4.x or 5.0: Upgrade or downgrade the leaf and spine switches through a Cisco APIC running releases 4.x or 5.0

From Cisco APIC release 4.2(5):

• Switches in the same update group are upgraded or downgraded simultaneously, regardless of pods.

• The default concurrent capacity per group is unlimited.

The above enhancements from Cisco APIC release 4.2(5) take effect as soon as the Cisco APIC s are upgraded to 4.2(5) or later. For instance, when the Cisco APIC s are upgraded to 4.2(5) and the switches are still at release 13.2(10), the above enhancements will be effective when the switch is upgraded from 13.2(10) to 14.2(5).

This enhancement will help you reduce the time it takes to upgrade your switches.

Rule 5 – Save time by downloading switch images beforehand

Even after you have downloaded Cisco APIC and switch images to the Cisco APIC 's firmware repository, the switches still need to download the image from the Cisco APIC s. In later releases, this operation can be performed separately from the actual upgrade procedure. This is called pre-download and is equivalent to Step 7 in Upgrade or downgrade Cisco ACI fabrics .

Prior to switch release 14.1(1):

Not supported. Switches download the image from Cisco APIC s when the upgrade or downgrade is triggered.

Switch release 14.(1) - 15.0(x):

• Pre-download can be performed through the upgrade scheduler.

• Following is the recommended procedure:

  1. Create update groups with the scheduler set far into the future (such as 10 years in the future). This will trigger switches to download the image from Cisco APIC s immediately.

  2. When it’s the time to start the upgrade in the maintenance window, edit the same groups and change the Upgrade Start Time to Now .

• If the current version of the switches is 14.2(5) or later, the Cisco APIC GUI shows the progress of the pre-download.

Switch release 15.1(1) or later:

• Pre-download is built in the GUI workflow natively without using a scheduler.

  1. Switches download the image from the Cisco APIC s when you create an update group and click Begin Download .

  2. When the pre-download is completed, each switch shows Ready to Install .

  3. Perform Begin Install for the same group to trigger the upgrade.

The above enhancement (pre-download) from switch release 14.1(1) takes effect only after both the Cisco APIC s and the switches are upgraded or downgraded to the corresponding versions. For example, when the Cisco APIC s are upgraded to 4.2(7) and the switches are still on 13.2(10), pre-download is not available to upgrade the switches from 13.2(10) to 14.2(7). On the other hand, when the Cisco APIC s are upgraded to 5.2(1) and the switches are still in 14.2(7), pre-download is performed through the new Cisco APIC GUI using Begin Download for switch upgrades from 14.2(7) to 15.2(1).

Graceful Upgrade or Downgrade of ACI Switches

If you want to isolate a switch from user traffic when performing an upgrade or downgrade procedure, it's helpful to become familiar with the different terms and methods available to better understand what is supported and what is not supported in these situations:

• Graceful Insertion and Removal (GIR) : The operation used to isolate a switch from user traffic.

• Maintenance mode : Used to isolate a switch from user traffic for debugging purposes. You can put a switch in maintenance mode by enabling the Maintenance (GIR) field in the Fabric Membership page in the Cisco APIC GUI, located at Fabric > Inventory > Fabric Membership (right-click on a switch and choose Maintenance (GIR) ).

  If you put a switch in maintenance mode , that switch is not considered as a part of the operational ACI fabric infra and it will not accept regular Cisco APIC communications. Therefore, performing a firmware upgrade or downgrade for a switch in this state is not supported, because the process may fail or may get stuck in an incomplete status indefinitely if you attempt to perform a firmware upgrade or downgrade on the switch while the switch is in this state.

• Graceful Upgrade : Used to reload a switch after it is isolated from user traffic during an upgrade procedure. Switches are programmed to reboot automatically at a certain point during the firmware upgrade process; this operation will automatically perform GIR prior to that reboot. You can find the Graceful Maintenance option (releases prior to release 5.1) or the Graceful Upgrade option (release 5.1 and later) for a switch in an update group in Admin > Firmware in the Cisco APIC GUI.

  If you wish to halt the procedure after the switch is isolated from user traffic and before it is reloaded in order to ensure the user traffic is flowing through redundant paths, such an operation is currently not supported in ACI.

Guidelines for ACI Switch Graceful Upgrade

All guidelines from Guidelines for ACI switch upgrades and downgrades also apply to Graceful Upgrade . However, this section provides more information on several guidelines that are specifically critical for Graceful Upgrade .

• As suggested in Rule 2 – Determine how spine switches should be grouped , do not upgrade all spine switches in a pod at one time, especially when you are performing a Graceful Upgrade in a Multi-Pod setup.

  Otherwise, the upgrade will fail, leaving the spine switches isolated from the fabric indefinitely. This is because, as part of the Graceful Upgrade process, IPN connectivity is brought down explicitly on each spine switch being upgraded gracefully so that it can isolate itself from the fabric. Upgrading in this way results in the entire pod, including the spine switches themselves, to lose communication with Cisco APIC s and switches in other pods without the means to self-recover.

  Due to this reason, if you are performing a Graceful Upgrade , you must put the spine switches from the same pod into different maintenance/update groups such that the switches get upgraded separately. If the pod has only one spine switch, you must disable the Graceful Upgrade (or Graceful Maintenance ) option prior to the upgrade. In case you fail to follow this procedure, refer to the workaround provided in CSCvn28063 .

  To avoid this issue, Cisco APIC 4.1(1) release introduced a safe mechanism to reject the upgrade of the last spine switch in a pod when Grace Upgrade is enforced. This block mechanism is also described in Rule 4 – Understand the concurrent capacity in switch update groups .

• As suggested in Rule 3 – Determine how leaf switches should be grouped , you must put Cisco APIC -connected leaf switches into different maintenance/update groups so that two leaf switches connected to the same Cisco APIC are not upgraded at the same time.