Field Notice: FN74116 - Risk of Configuration Loss in Cisco Application Policy Infrastructure Controllers During Power Event - Software Upgrade Recommended

Available Languages

Updated:March 20, 2024

Document ID:FN74116

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Affected Software Product	Affected Release	Affected Release Number	Comments
APIC Software	5	5.2(3e), 5.2(3f), 5.2(3g), 5.2(4d), 5.2(4e), 5.2(4f), 5.2(4h), 5.2(5c), 5.2(5d), 5.2(5e), 5.2(6e), 5.2(6g), 5.2(6h), 5.2(7f), 5.2(7g)
APIC Software	6	6.0(1g), 6.0(1j), 6.0(2h), 6.0(2j)

Defect Information

Defect ID	Headline
CSCwf54771	Several replicas empty after ungraceful reload of APIC

Problem Description

Cisco Application Policy Infrastructure Controllers (APICs) are vulnerable to data loss during any ungraceful reload, such as a power outage or physical removal of power. This may result in the Cisco APICs coming back in a cluster diverged state or the configuration being lost upon power up.

A configuration backup with a known AES encryption key can be used to restore any missing configuration to the Cisco APICs after an ungraceful reload.

It is strongly recommended that all Cisco Application Centric Infrastructure (ACI) operators have recent configuration backups available on a system outside of the Cisco APIC and that all encryption passwords are clearly documented.

Background

This problem exists for all Cisco APIC appliances that are running an affected 5.2 or 6.0 software release and experience a physical loss of power. Software reboots or power cycles through the Cisco Integrated Management Controller (IMC) are not affected.

Problem Symptom

One or more of the following symptoms may be observed after an ungraceful reload or power outage:

User configuration is missing on Cisco APICs and switches.
Data Management Engine (DME) services fail with a fatal error that indicates that it could not open or read the clone database.
- Example: Cannot create or open DB: /var/run/mgmt/db/ifc_plgnhandler/S24_R3_clone/ifc_plgnhandler.db, rv = 26
The Cisco APIC cluster is partially diverged.

Workaround/Solution

Solution

Upgrade the Cisco ACI fabric to a fixed software release prior to any power event. Software is available from the Software Center on Cisco.com.

Affected Release	Fixed Release
5.2	5.2(8h) or later
6.0	6.0(3d) or later

If a power outage or ungraceful reload has occurred, call the Cisco technical Assistance Center (TAC) to perform the following steps:

Remove any empty or corrupted clone directories from the filesystem. This must be performed as a root user.
Restart the affected processes and ensure that the Cisco APIC pulls a working database copy from another Cisco APIC.

How to Identify Affected Products

To determine whether a Cisco APIC is affected by the software defect, check the current software release using the show version command, as shown in the following example:

apic1# show version
 Role        Pod         Node        Name                      Version
 ----------  ----------  ----------  ------------------------  --------------------
 controller  1           1           apic1                     6.0(4c)
 controller  1           2           apic2                     6.0(4c)
 controller  1           3           apic3                     6.0(4c)

A Cisco APIC is affected by the software defect if the cluster is diverged and there are log lines that show last term 0 last index 0. These two conditions can be verified with the following commands:

apic1# show faults code F0321 controller
Code            : F0321
Severity        : critical
Last Transition : 2024-01-23T15:44:27.585-05:00
Lifecycle       : raised
DN              : topology/pod-1/node-1/av/node-1/fault-F0321
Description     : Controller 1 is unhealthy because: Data Layer Partially Diverged
                 
apic1# zgrep "last term = 0 last index = 0" /var/log/dme/log/svc_ifc_*
svc_ifc_policymgr.bin.log.60.gz:5124||2024-01-23T15:42:21.784791378+00:00||instance||INFO||||MIT at /var/run/mgmt/db/ifc_policymgr/S8_R1_clone/ifc_policymgr.db last term = 0 last index = 0||../common/src/framework/./core/shard/Instance.cc||1194
svc_ifc_policymgr.bin.log.60.gz:14609||2024-01-23T15:43:05.266039342+00:00||instance||INFO||||MIT at /var/run/mgmt/db/ifc_policymgr/S8_R1_clone/ifc_policymgr.db last term = 0 last index = 0||../common/src/framework/./core/shard/Instance.cc||1194
svc_ifc_policymgr.bin.log.60.gz:14654||2024-01-23T15:43:19.863083592+00:00||instance||INFO||||MIT at /var/run/mgmt/db/ifc_policymgr/S8_R1_clone/ifc_policymgr.db last term = 0 last index = 0||../common/src/framework/./core/shard/Instance.cc||1194

Revision History

Version	Description	Section	Date
1.0	Initial Release	—	2024-MAR-20

For More Information

For further assistance or for more information about this field notice, contact the Cisco Technical Assistance Center (TAC) using one of the following methods:

Receive Email Notification About New Field Notices

To receive email updates about Field Notices (reliability and safety issues), Security Advisories (network security issues), and end-of-life announcements for specific Cisco products, set up a profile in My Notifications.