Cisco APIC Installation and ACI Upgrade and Downgrade Guide
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
When performing an upgrade of an APIC cluster, there is a certain sequence of events that occur to allow for the upgrade of
each APIC separately, along with ensuring that the data on the upgraded APIC will be compatible with the target image. Most
of these events happen in the background, so it’s important to understand what you should expect to see when you trigger an
upgrade of the APIC cluster.
Image is uploaded to the firmware repository. The image is synced to all APIC cluster members.
Upgrade is triggered to a specific target version.
Each APIC in the cluster goes through the process to install the new image in the first grub partition. It’s important to
note that this happens in parallel to speed up the upgrade process.
Once the image installation is completed, each APIC takes its turn to go through a data conversion process of the database
files in a sequential order. When this occurs, the following events happen:
The Data Management Engine (DME) processes shut down. This includes the nginx web server which services all API requests.
Because of this, you will lose access to the UI/API, as well as any other backend application that runs on that APIC.
The database files are converted from the initial version to the target version. The amount of time this takes is dependent
on the size of the configuration deployed on the ACI fabric. Because of this, the total time to complete the conversion will
vary between deployments.
Note
It’s critical that there is no disruptive action taken to the APIC at this stage, as it could result in data loss or partial
configuration if this stage does not complete successfully. See Operations You Must Avoid During an Upgrade/Downgrade for more information.
The APIC will then reload after the database conversion process has completed successfully and will boot up on the version
of software defined in the target version.
Once the APIC that performed the reload comes back online, the sequence of events outlined in Step 4 happen to the next APIC
in the cluster. This process repeats itself until all members of the cluster have been upgraded.
Detailed Summary of APIC Upgrade
The following sections provide a detailed summary of APIC upgrades.
Understanding APIC Upgrade Stages
The stages that the APICs go through during an upgrade process will vary, depending on
the current running version of your software and the version of the software that you
are upgrading to.
Upgrading from a Pre-4.2(5) Release to Release 4.2(5) or Later
If the current running version of your software is earlier than Cisco APIC Release 4.2(5), and you are upgrading to Release 4.2(5) or later, this section
provides information on the stages that each APIC will go through during the upgrade
process.
Before you begin the upgrade, each APIC will be shown at 100%, which shows
that the installation, upgrade, or downgrade that was previously performed
on each APIC was completed successfully.
Once you begin the upgrade process, the status will change from 100% to 0%
for all of the APICs, going through the following stages:
The status will first show as Firmware upgrade
queued.
The status will then change to Firmware upgrade in
progress.
Next, the APIC that is selected first by the installer will begin upgrading
and will advance to 5%, as shown in the following figure.
Note
The APIC that is selected first to begin the upgrade process is a random
selection, depending on which APIC is called first by the installer.
That means that the first APIC that will begin upgrading in the cluster
is not necessarily the APIC with the lowest-numbered name.
During this stage, an error and warning message similar to the following
might appear:
This is normal and expected behavior, and is due to the fact that the APIC is
being rebooted as part of the upgrade process.
Once the first APIC in your cluster has reached 5%, each APIC will then advance
through the following stages in the upgrade process, which will be shown in the
Upgrade Progress area:
First APIC called by installer: 0% → 5% → 100%
Remaining APICs in cluster: 0% → 100%
The following table provides more details on what happens at each stage of this
upgrade process:
Upgrade Progress
Description
0%
Displayed when the upgrade installer is initiated and the upgrade
process has started.
5%
During this stage, the following configurations occur for all
APICs in the cluster, with the status of the first APIC called
by the installer remaining at 5% and the status of the remaining
APICs in the cluster remaining at 0%:
The first APIC called by the installer performs internal sanity checks such as a preparation for database conversions to be
compatible with the new firmware and a firmware image status check on each APIC.
Internal sanity checks are completed, and the target version gets pre-loaded into the APICs.
All APICs in the cluster upgrade sequentially, going in
the order of the first APIC called by the installer,
then the second APIC, then the third APIC. In this
stage, each APIC waits for the other APICs ahead of it
to complete before that APIC begins upgrading. In other
words, the first APIC begins upgrading first, with the
second and third APICs waiting until the first APIC has
completed the upgrade process. Once the first APIC has
completed this stage, the second APIC begins the upgrade
process, as the third APIC waits.
All APICs go through the data conversion phase of the
upgrade process, in sequential order. At this stage of
the upgrade process, if the upgrade process fails, the
system will roll back to the previous version of the
software.
Each APIC will reboot during this stage, once the data
conversion part of this stage is completed. As each APIC
goes through a reboot, you will see the following:
The following error and warning message might
appear:
Request failed due to server-side error
or web-socket connection closed due to unknown
reason
This is normal and expected behavior, and is due
to the fact that the APIC is being rebooted as
part of the upgrade process.
The APIC will disappear briefly from the list of
APIC controllers in the GUI, and will then
reappear in the list after the reboot has
completed and the upgrade has completed
successfully.
When the Cisco APIC that the browser is connected to is upgraded and it
reboots, the browser first displays an error message,
then you will not be able to see anything in the browser
that you used to log into this APIC. However, you can
log into any of the remaining APICs in the cluster to
continue to monitor the progress of the upgrade process,
if you want.
100%
Displayed when that APIC has successfully completed the entire
upgrade process.
Upgrading from Release 4.2(5) or Later to a Later Release
If the current running version of your software is Cisco APIC Release 4.2(5) or later, and you are upgrading to a later release, this section
provides information on the stages that each APIC will go through during the upgrade
process.
Before you begin the upgrade, each APIC will be shown at 100%, which shows
that the installation, upgrade, or downgrade that was previously performed
on each APIC was completed successfully.
Once you begin the upgrade process, the status will change from 100% to 0%
for all of the APICs, as shown in the following figure.
Next, the APIC that is selected first by the installer will begin upgrading
and will advance to 5%, as shown in the following figure.
Note
The APIC that is selected first to begin the upgrade process is a random
selection, depending on which APIC is called first by the installer.
That means that the first APIC that will begin upgrading in the cluster
is not necessarily the APIC with the lowest-numbered name.
The APIC that is selected second by the installer will then begin upgrading
and will advance to 5%.
The APIC that is selected third by the installer will then begin upgrading
and will advance to 5%.
If you have more than three APICs in your cluster, the process will continue
until all the APICs in your cluster are at 5%.
Once all the APICs in your cluster have reached 5%, each APIC will then advance
through the following stages in the upgrade process, which will be shown in the
Upgrade Progress area:
First APIC called by installer: 0% → 5% → 10% → 25% → 50% → 75% →
100%
The following table provides more details on what happens at each stage of this
upgrade process:
Upgrade Progress
Install Stage
Install Stage Status
Description
0%
Ready for next upgrade
Queued
Displayed when the upgrade installer is initiated and the upgrade
process has started.
5%
Checking compatibility
Ensure hardware and software compatibilities for controller
Displayed when the upgrade installer is initiated and the upgrade
process has started.
10%
Checking controller health
Performing internal sanity checks to prepare for the upgrade
In this stage, the first APIC called by the installer performs internal sanity checks such as a preparation for database conversions
to be compatible with the new firmware and a firmware image status check on each APIC. The first APIC will move to 10% in
this stage, while the other APICs in the cluster will remain at 5%.
25%
Performing upgrade
Install target version on controller
Displayed when the internal sanity checks have completed, and the target version is getting pre-loaded into the APIC.
Note that the first APIC, which is performing the configuration
checks and pre-upgrade configurations on the other APICs in the
cluster, will move from 10% to 25% in this stage, whereas the
remaining APICs in the cluster will jump from 5% directly to 25%
in this stage.
50%
Waiting for other controllers to upgrade
Waiting for other controllers to complete migrating
configuration
The APICs in the cluster do not upgrade simultaneously, all at
one time, but rather upgrade sequentially, going in the order of
the first APIC called by the installer, then the second APIC,
then the third APIC. In this stage, each APIC waits for the
other APICs ahead of it to complete before that APIC begins
upgrading. In other words, the first APIC begins upgrading
first, with the second and third APICs waiting until the first
APIC has completed the upgrade process. Once the first APIC has
completed this stage, the second APIC begins the upgrade
process, as the third APIC waits.
75%
Migrating configuration
Now performing conversion on the controller
Displayed at the data conversion phase of the upgrade process.
Again, because of the sequential order of the upgrade process
between all of the APICs in the cluster, one APIC will move from
50% to 75% in this stage, while the other two APICs will remain
at 50%. Once that first APIC has completed this phase of the
upgrade process, the second APIC in the cluster will begin this
phase and will move from 50% to 75%, with the remaining APICs in
the cluster remaining at 50% until the second APIC has completed
this phase of the upgrade process.
At this stage of the upgrade process (between the 50% stage and
the 75% stage), if the upgrade process fails, the system will
roll back to the previous version of the software.
Each APIC will reboot during this stage, once the data conversion
part of this stage is completed. As each APIC goes through a
reboot, you will see the following:
The following error and warning message might appear:
Request failed due to server-side error or
web-socket connection closed due to unknown
reason
This is normal and expected behavior, and is due to the
fact that the APIC is being rebooted as part of the
upgrade process.
The APIC will disappear briefly from the list of APIC
controllers in the GUI, and will then reappear in the
list after the reboot has completed and the upgrade has
completed successfully.
When the Cisco APIC that the browser is connected to is upgraded and it reboots,
the browser first displays an error message, then you will not
be able to see anything in the browser that you used to log into
this APIC. However, you can log into any of the remaining APICs
in the cluster (the APICs that were still at 50% at the point
that the APIC was reloaded) to continue to monitor the progress
of the upgrade process, if you want.
100%
Ready for next upgrade
Successful
Displayed when that APIC has successfully completed the entire
upgrade process.
Default Interface Policies in the 5.2(4) release and later
When you upgrade to the 5.2(4) or later release, the Cisco Application Policy Infrastructure
Controller (APIC) creates the following default interface policies automatically:
CDP (cdpIfPol)
system-cdp-disabled
system-cdp-enabled
LLDP (lldpIfPol)
system-lldp-disabled
system-lldp-enabled
LACP (lacpLagPol)
system-static-on
system-lacp-passive
system-lacp-active
Link Level (fabricHIfPol)
system-link-level-100M-auto
system-link-level-1G-auto
system-link-level-10G-auto
system-link-level-25G-auto
system-link-level-40G-auto
system-link-level-100G-auto
system-link-level-400G-auto
Breakout Port Group Map (infraBrkoutPortGrp)
system-breakout-10g-4x
system-breakout-25g-4x
system-breakout-100g-4x
During the upgrade, if there is already a policy with the exact same name and the exact same parameters as any of these policies,
the system takes ownership of those policies and the policies become read-only. If instead the parameters are different, such
as the system-cdp-disabled has a setting "enabled," then the policies will continue to be user policies. That is, a user can modify the policies.
High Level Summary of Switch Upgrade
When performing an upgrade of an ACI switch node, there is a certain sequence of events that occur to the device(s) being
upgraded. Most of these events happen in the background, so it’s important to understand what you should expect to see when
you trigger an upgrade of an ACI switch node.
The image is pushed from the APIC to the switch.
The filesystem and bootflash of the switch is checked to ensure that there is enough space to extract the image.
The image is extracted, and the primary grub partition is updated to the target version. The older version is moved into the
recovery partition.
The BIOS and EPLD images are upgraded if applicable.
The switch will do a clean reload, and will re-join the ACI fabric running the newer version of software.
Starting with release 2.1(4), support was added for the third-party Micron Solid State Drive (SSD) firmware auto update. As
part of the standard Cisco APIC software upgrade process, the switches will reboot when they upgrade. During that boot-time
process, the system will also check the current SSD firmware and will automatically perform an upgrade to the SSD firmware,
if necessary. If the system performs an SSD firmware upgrade, the switches will then go through another clean reboot afterward.
Detailed Summary of Switch Upgrade
The following sections provide a detailed summary of switch upgrades.
Understanding Switch Upgrade Stages
During an ACI switch node upgrade, the upgrade progress will advance based on the stages which have completed.
The following table provides more details on what happens at each stage of this upgrade process:
Upgrade Progress
Install Stage
Description
0%
Firmware upgrade queued
Displayed when firmware is being downloaded to the switch from the APIC.
5%
Firmware upgrade in progress
Displayed when the upgrade installer is initiated, and the upgrade process has started.
45%
Firmware upgrade in progress
Displayed after the bootflash check has completed and the image extraction stage has begun.
60%
Firmware upgrade in progress
Image Extraction stage has completed and the grub partition is being updated with the new software information.
70%
Firmware upgrade in progress
The software has been updated on the switch.
80%
Firmware upgrade in progress
The EPLD and BIOS upgrade has begun.
95%
Firmware upgrade in progress
The EPLD and BIOS upgrade has completed, and switch reboot has been initiated.
100%
Upgraded Successfully
The switch has re-joined the fabric after the clean reload running target version of software.
Understanding APIC Downgrade Stages
The ACI APIC and switch downgrade stages are identical to the upgrade stages described in High Level Summary of APIC Upgrade, just that the version of software would be lower than the running version.
Operations You Must Avoid During an Upgrade/Downgrade
If at any point in time you believe the upgrade/downgrade has either stalled or failed, it is critical that you do not take
any of the actions listed below:
Don't reload any APIC in the cluster.
Don't decommission any APIC in the cluster.
Don't change the firmware target version back to the original version.
Instead, follow the guidelines below:
View the installer log files outlined in the Troubleshooting section if applicable (see APIC Installer Log Files and ACI Switch Installer Log Files). This will help in understanding if there is still activity ongoing on the devices being upgraded.