ACI Upgrade or Downgrade Architecture

High Level Summary of APIC Upgrades and Downgrades

  • APIC upgrades and downgrades involve a defined sequence of steps to ensure compatibility and data integrity across all cluster members.

  • Each APIC is upgraded or downgraded separately, with image installation and database conversion processes managed to minimize disruption.

  • Recent APIC releases introduce enhancements such as centralized orchestration and improved post-upgrade validation procedures.

Sequence and Key Considerations for APIC Upgrades and Downgrades

  1. Image is uploaded to the firmware repository and synced to all APIC cluster members.

  2. Upgrade or downgrade is triggered to a specific target version.

  3. Each APIC installs the new image in the first grub partition in parallel to speed up the process.

  4. After image installation, each APIC sequentially performs a database conversion process:

    1. The Data Management Engine (DME) processes, including the nginx web server, shut down. UI/API access and backend applications on that APIC become unavailable.

    2. The database files are converted from the initial version to the target version. The duration depends on database size and configuration. For APIC release 6.0(3) or newer, the process is enhanced for shorter wait times. For APIC release 6.2(1) or newer, APIC 1 orchestrates the conversion for all distributed databases, while other APICs convert only their local databases, reducing synchronization risks.


      Note

      It’s critical that there is no disruptive action taken to the APIC at this stage, as it could result in data loss or partial configuration if this stage does not complete successfully. See Guidelines and limitations for upgrading or downgrading for more information.

    3. The APIC reloads after successful database conversion and boots up on the target software version.

  5. After an APIC reloads and comes back online, the same sequence is repeated for the next APIC in the cluster. The APIC that came back online initiates post-upgrade activities as a final database check. This process continues until all cluster members are upgraded or downgraded.

  6. For APIC versions prior to 6.0(6), the upgrade is considered complete when all APICs are online and Fully-Fit, regardless of post-upgrade activities. Starting from 6.0(6), the upgrade status transitions to “Post Upgrade Pending” until post-upgrade activities are complete on each APIC node, after which the status becomes “Completed”.


    Note

    The post-upgrade activities on APIC should be successfully completed before proceeding with switch upgrades. It is recommended to run the pre-upgrade validation script before both the APIC cluster upgrade and switch upgrades, as these may occur in different maintenance windows. For versions prior to 6.0(6), running the script before switch upgrades is highly encouraged, even within the same window, as it checks pre-upgrade validations and the status of post-upgrade activities to ensure the fabric is ready for switch upgrades after the APIC cluster upgrade.

  7. Starting from APIC 6.1(2), the same upgrade steps are performed on standby APIC nodes after all active APIC nodes are upgraded. See the Getting Started Guide for Cisco APIC 6.1 for details.

APIC upgrade detailed summary

The following section provides a detailed summary of APIC upgrades.

Understand APIC upgrade and downgrade stages

The APIC upgrade process progresses through several stages, each building on the completion of the previous one.

APIC upgrade process overview

Beginning with Cisco ACI release 6.2(1), the APIC upgrade process has been enhanced. The upgrade always starts on APIC1, which performs several cluster-wide upgrade stages and is the first APIC to be upgraded. When APIC1 is ready to upgrade itself, it hands off control to APIC2 using an internal API call. APIC1 then operates in node upgrade mode and completes its own upgrade. During this time, UI is redirected to APIC 2. APIC2 hands control back to APIC1 and UI is redirected. APIC1 continues upgrading the remaining APICs in the cluster in the same way. When all APICs have been upgraded to the target version, APIC1 performs the final cluster-wide stages and completes the upgrade cycle.

  1. The upgrade process always starts on APIC1

  2. APIC1 performs several cluster-wide upgrade stages and is the first APIC to be upgraded.

  3. When APIC1 is ready to upgrade itself, it hands off control to APIC2 using an internal API call.

  4. APIC1 then operates in node upgrade mode and completes its own upgrade.

  5. During this time, UI is redirected to APIC 2 during this time.

  6. APIC2 hands control back to APIC1 and UI is redirected.

  7. APIC1 continues upgrading the remaining APICs in the cluster in the same way.

  8. When all APICs have been upgraded to the target version, APIC1 performs the final cluster-wide stages and completes the upgrade cycle.

APIC upgrade stage table

The table below explains what occurs at each stage of the upgrade process.

Table 1. APIC Upgrade Stages

Name

Stage Level

Description

Catastrophic Failure Check

Cluster Wide

Initiate pre-upgrade validations to ensure upgrade can be performed.

Cluster Upgrade Preparation

Cluster Wide

Verify upgrade callbacks and set the cluster version to the requested upgrade version.

Upgrade Preparation

Node Level

Setup pre-upgrade state for all states.

Stage New OS

Node Level

Sets up the environment for data conversion using the extracted target image.

Freeze Database

Node Level

Freeze database for data conversion and monitor permdown completion.

Stop APIC Services

Node Level

Shut down services.

Stage Database Conversion

Node Level

Setup for data conversion.

Database Conversion

Node Level

Performs data conversion.

Install New OS

Node Level

Install the new OS.

Reboot

Node Level

Based on the upgrade, restart container or kexec the box to perform the reload operation.

Finalize OS Configuration

Node Level

Finalization of state done.

Cleanup

Node Level

Verify post upgrade activity.

Validate Cluster Health

Node Level

Verify that all nodes are in a healthy state.

Finalize Database State

Node Level

Verify if cluster is healthy

Validate Cluster Upgrade

Node Level

Perform post cluster upgrade checks.

Finalize Cluster State

Cluster Wide

Verify if post upgrade is completed on all shards and verify the APIC version

Validate Cluster State

Cluster Wide

Reset upgrade call backs on all APICs

Default interface policies in the 5.2(4) release and later

When you upgrade to the 5.2(4) or later release, the Cisco Application Policy Infrastructure Controller (APIC) creates the following default interface policies automatically:

  • CDP (cdpIfPol)

    • system-cdp-disabled

    • system-cdp-enabled

  • LLDP (lldpIfPol)

    • system-lldp-disabled

    • system-lldp-enabled

  • LACP (lacpLagPol)

    • system-static-on

    • system-lacp-passive

    • system-lacp-active

  • Link Level (fabricHIfPol)

    • system-link-level-100M-auto

    • system-link-level-1G-auto

    • system-link-level-10G-auto

    • system-link-level-25G-auto

    • system-link-level-40G-auto

    • system-link-level-100G-auto

    • system-link-level-400G-auto

  • Breakout Port Group Map (infraBrkoutPortGrp)

    • system-breakout-10g-4x

    • system-breakout-25g-4x

    • system-breakout-100g-4x

During the upgrade, if there is already a policy with the exact same name and the exact same parameters as any of these policies, the system takes ownership of those policies and the policies become read-only. If instead the parameters are different, such as the system-cdp-disabled has a setting "enabled," then the policies will continue to be user policies. That is, a user can modify the policies.

How ACI switch upgrades and downgrades work

When performing an upgrade or downgrade of an ACI switch node, there is a certain sequence of events that occur to the device(s) being upgraded or downgraded. Most of these events happen in the background, so it's important to understand what you should expect to see when you trigger an upgrade of an ACI switch node.

Summary

The key components involved in the ACI switch upgrade and downgrade process are:

  • APIC: Pushes the image to the switch and manages the upgrade process

  • Switch filesystem and bootflash: Provides storage space for image extraction

  • BIOS and EPLD images: May require updates during the upgrade process

  • SSD firmware: Automatically upgraded if necessary (starting with release 2.1(4))

Workflow

These stages describe how ACI switch upgrades and downgrades work:

  1. The APIC pushes the image to the switch and prepares the system for upgrade.
    • The image is pushed from the APIC to the switch.
    • The filesystem and bootflash of the switch is checked to ensure that there is enough space to extract the image.
  2. The system extracts the image and updates the boot partitions.
    • The image is extracted, and the primary grub partition is updated to the target version. The older version is moved into the recovery partition.
  3. The system upgrades firmware components if applicable.
    • The BIOS and EPLD images are upgraded if applicable.
  4. The switch performs a clean reload and rejoins the fabric with the new software version.
    • The switch will do a clean reload, and will re-join the ACI fabric running the newer version of software.
    Starting with release 2.1(4), support was added for the third-party Micron Solid State Drive (SSD) firmware auto update. As part of the standard Cisco APIC software upgrade process, the switches will reboot when they upgrade. During that boot-time process, the system will also check the current SSD firmware and will automatically perform an upgrade to the SSD firmware, if necessary. If the system performs an SSD firmware upgrade, the switches will then go through another clean reboot afterward.

Switch upgrades

A switch upgrade is a network maintenance process that

  • updates the firmware, software, or hardware components of network switches

  • enhances functionality, security, and performance of the network infrastructure, and

  • ensures compatibility with newer network protocols and features.

Switch upgrade summary information

The following sections provide a detailed summary of switch upgrades.

How ACI switch upgrade and downgrade stages work

During an ACI switch node upgrade or downgrade, the upgrade or downgrade progress will advance based on the stages which have completed.

Summary

The key components involved in the ACI switch upgrade and downgrade process are:

  • APIC: Downloads firmware to the switch and initiates the upgrade process

  • Switch: Receives firmware, extracts images, updates software components, and performs reboot

  • Bootflash: Storage component that is checked and validated during the upgrade

  • EPLD and BIOS: Hardware components that are upgraded during the process

  • Fabric: Network infrastructure that the switch rejoins after successful upgrade

Workflow

These stages describe how the ACI switch upgrade and downgrade process progresses:

  1. At 0% progress, firmware upgrade is queued and firmware is downloaded to the switch from the APIC.
  2. At 5% progress, firmware upgrade begins and the upgrade installer is initiated.
  3. At 45% progress, bootflash check completes and image extraction stage begins.
  4. At 60% progress, image extraction stage completes and the grub partition is updated with new software information.
  5. At 70% progress, the software has been updated on the switch.
  6. At 80% progress, EPLD and BIOS upgrade begins.
  7. At 95% progress, EPLD and BIOS upgrade completes and switch reboot is initiated.
  8. At 100% progress, upgrade is successful and the switch rejoins the fabric after clean reload running target version of software.

Guidelines and limitations for upgrading or downgrading

Downgrading Cisco APICs from Cisco APIC 6.2 or newer releases to a release older than 6.2(1) is not supported. See Checklists for Downgrade for details.

  • If at any point in time you believe the upgrade or downgrade has either stalled or failed, it is critical that you do not take any of the actions listed below:

    • Do not reload any Application Policy Infrastructure Controller ( APIC ) in the cluster.

    • Do not decommission any Cisco APIC in the cluster.

    • Do not change the firmware target version back to the original version.

    Instead, follow these guidelines:

    1. View the installer log files outlined in the Troubleshooting section if applicable (see APIC Installer Log Files and ACI Switch Installer Log Files ). This will help in understanding if there is still activity ongoing on the devices being upgraded or downgraded.

    2. Collect the tech-support files outlined in the Troubleshooting section (see Collecting Tech-Support Files ).

    3. Contact Cisco TAC if the upgrade or downgrade does not complete successfully and upload the tech-support files to the TAC case after it has been created.

  • The log record objects are stored only in one shard of a database on one of the Cisco APIC s. Because of this, the log records are not accessible while the Cisco APIC is rebooting for an upgrade or downgrade, unlike other objects that can still be read through another Cisco APIC .

  • To upgrade to the Cisco APIC 6.0(2) release or later, you must perform the following procedure:

    1. Download the Cisco APIC 6.0(2) or later image and upgrade the APIC cluster to the downloaded release. Before this step is completed, do not download the Cisco Application Centric Infrastructure ( ACI )-mode switch images to the Cisco APIC . The 6.0(2) release has both 32-bit and 64-bit switch images, but releases prior to 6.0(2) do not support 64-bit images. As a result, downloading the 64-bit images at this time might cause errors or unexpected results. However, if your Cisco APIC s have the 5.2(8) release or later, except for the 6.0(1) release, you can download the switch images to the Cisco APIC before this step the same as you would with any other upgrade procedure prior to 6.0(2).

    2. Download both the 32-bit and 64-bit Cisco ACI -mode switch images to the Cisco APIC . Downloading only one of the images may result in errors during the upgrade process.

      Beginning with the 6.0(3) release, the switch determines which image to install from the Cisco APIC based on the available memory of the switch instead of based on a static mapping. If the available memory of the switch is less than or equal to 24 GB, the switch installs the 32-bit image. If the available memory of the switch is greater than or equal to 32 GB, the switch may be upgraded to the 32-bit image first, then upgraded again to the 64-bit image, which results in two reboots during the upgrade process.

      Modular spine switches install the 64-bit image regardless of the switch's available memory.

    3. Create the maintenance groups and trigger the upgrade procedure as usual. Cisco APIC automatically deploys the correct image to the respective switch during the upgrade process.

  • If you change the switch firmware group during the upgrade process, the upgrade process will not be completed and you may encounter some unexpected upgrade behavior.