Field Notice: FN - 72335 - APIC Upgrade to Release to 4.1(1) or Later Prompts "ifc_reader" Crash - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	24-May-22	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	NX-OS System Software-ACI	14.0	14.0(1h), 14.0(2c), 14.0(3c), 14.0(3d)
NON-IOS	NX-OS System Software-ACI	13.2	13.2(10e), 13.2(10f), 13.2(1l), 13.2(1m), 13.2(2l), 13.2(2o), 13.2(3i), 13.2(3j), 13.2(3n), 13.2(3o), 13.2(3r), 13.2(3s), 13.2(41d), 13.2(4d), 13.2(4e), 13.2(5d), 13.2(5e), 13.2(5f), 13.2(6i), 13.2(7f), 13.2(7k), 13.2(8d), 13.2(9b), 13.2(9f)
NON-IOS	NX-OS System Software-ACI	13.1	13.1(1i), 13.1(2m), 13.1(2o), 13.1(2p), 13.1(2q), 13.1(2s), 13.1(2t), 13.1(2u), 13.1(2v)
NON-IOS	NX-OS System Software-ACI	13.0	13.0(1k), 13.0(2h), 13.0(2k), 13.0(2n)
NON-IOS	NX-OS System Software-ACI	13	13.2(9h)
NON-IOS	NX-OS System Software-ACI	12.3	12.3(1e), 12.3(1f), 12.3(1i), 12.3(1l), 12.3(1o), 12.3(1p)
NON-IOS	NX-OS System Software-ACI	12.2	12.2(1n), 12.2(1o), 12.2(2e), 12.2(2f), 12.2(2i), 12.2(2j), 12.2(2k), 12.2(2q), 12.2(3j), 12.2(3p), 12.2(3r), 12.2(3s), 12.2(3t), 12.2(4f), 12.2(4p), 12.2(4q), 12.2(4r)
NON-IOS	NX-OS System Software-ACI	12.1	12.1(1h), 12.1(1i), 12.1(2e), 12.1(2g), 12.1(2k), 12.1(3g), 12.1(3h)
NON-IOS	NX-OS System Software-ACI	12.0	12.0(1m), 12.0(1n), 12.0(1o), 12.0(1p), 12.0(1q), 12.0(1r), 12.0(2f), 12.0(2g), 12.0(2h), 12.0(2l), 12.0(2m), 12.0(2n), 12.0(2o)
NON-IOS	NX-OS System Software-ACI	12	12.1(3j), 12.1(4a)
NON-IOS	NX-OS System Software-ACI	11.3	11.3(1g), 11.3(1h), 11.3(1i), 11.3(1j), 11.3(2f), 11.3(2h), 11.3(2i), 11.3(2j), 11.3(2k)
NON-IOS	NX-OS System Software-ACI	11.2	11.2(1i), 11.2(1k), 11.2(1m), 11.2(2g), 11.2(2h), 11.2(2i), 11.2(2j), 11.2(3c), 11.2(3e), 11.2(3h), 11.2(3m)
NON-IOS	NX-OS System Software-ACI	11.1	11.1(1j), 11.1(1o), 11.1(1r), 11.1(1s), 11.1(2h), 11.1(2i), 11.1(3f), 11.1(4e), 11.1(4f), 11.1(4g), 11.1(4i), 11.1(4l), 11.1(4m)
NON-IOS	NX-OS System Software-ACI	11.0	11.0(1b), 11.0(1c), 11.0(1d), 11.0(1e), 11.0(2j), 11.0(2m), 11.0(3f), 11.0(3i), 11.0(3k), 11.0(3n), 11.0(3o), 11.0(4h), 11.0(4o), 11.0(4q)
NON-IOS	NX-OS System Software-ACI	1.2	1.2(10a), 1.2(1a), 1.2(2a), 1.2(3a), 1.2(3b), 1.2(3c), 1.2(4a), 1.2(6a), 1.2(7a), 1.2(8a), 1.2(9a), 1.2(9b), 1.2(9c), 1.2(9d)
NON-IOS	APIC Software	4.0	4.0(1h), 4.0(2c), 4.0(3c), 4.0(3d)
NON-IOS	APIC Software	3.2	3.2(10e), 3.2(10f), 3.2(1l), 3.2(1m), 3.2(2l), 3.2(2o), 3.2(3i), 3.2(3n), 3.2(3o), 3.2(3r), 3.2(3s), 3.2(41d), 3.2(4d), 3.2(4e), 3.2(5d), 3.2(5e), 3.2(5f), 3.2(6i), 3.2(7f), 3.2(7k), 3.2(8d), 3.2(9b), 3.2(9f), 3.2(9h)
NON-IOS	APIC Software	3.1	3.1(1i), 3.1(2m), 3.1(2o), 3.1(2p), 3.1(2q), 3.1(2s), 3.1(2t), 3.1(2u), 3.1(2v)
NON-IOS	APIC Software	3.0	3.0(1k), 3.0(2h), 3.0(2k), 3.0(2n)
NON-IOS	APIC Software	2.3	2.3(1e), 2.3(1f), 2.3(1i), 2.3(1l), 2.3(1o), 2.3(1p)
NON-IOS	APIC Software	2.2	2.2(1n), 2.2(1o), 2.2(2e), 2.2(2f), 2.2(2i), 2.2(2j), 2.2(2k), 2.2(2q), 2.2(3j), 2.2(3p), 2.2(3r), 2.2(3s), 2.2(3t), 2.2(4f), 2.2(4p), 2.2(4q), 2.2(4r)
NON-IOS	APIC Software	2.1	2.1(1h), 2.1(1i), 2.1(2e), 2.1(2g), 2.1(2k), 2.1(3g), 2.1(3h), 2.1(3j), 2.1(4a)
NON-IOS	APIC Software	2.0	2.0(1m), 2.0(1n), 2.0(1o), 2.0(1p), 2.0(1q), 2.0(1r), 2.0(2f), 2.0(2g), 2.0(2h), 2.0(2l), 2.0(2m), 2.0(2n), 2.0(2o)
NON-IOS	APIC Software	1.3	1.3(1g), 1.3(1h), 1.3(1i), 1.3(1j), 1.3(2f), 1.3(2h), 1.3(2i), 1.3(2j), 1.3(2k)
NON-IOS	APIC Software	1.2	1.2(1i), 1.2(1k), 1.2(1m), 1.2(2g), 1.2(2h), 1.2(2i), 1.2(2j), 1.2(3c), 1.2(3e), 1.2(3h), 1.2(3m)
NON-IOS	APIC Software	1.1	1.1(1j), 1.1(1o), 1.1(1r), 1.1(1s), 1.1(2h), 1.1(2i), 1.1(3f), 1.1(4e), 1.1(4f), 1.1(4g), 1.1(4i), 1.1(4l), 1.1(4m)
NON-IOS	APIC Software	1.0	1.0(1e), 1.0(1h), 1.0(1k), 1.0(1n), 1.0(2j), 1.0(2m), 1.0(3f), 1.0(3i), 1.0(3k), 1.0(3n), 1.0(3o), 1.0(4h), 1.0(4o), 1.0(4q)

Defect Information

Defect ID	Headline
CSCvv12524	APIC reader crash on uribv4Nexthop.type query after APIC 1 upgrades from 3.2 to 4.2

Problem Description

After a Cisco Application Policy Infrastructure Controller (APIC) has completed the upgrade process from a release earlier than 4.1(1) to Release 4.1(1) or later, the ifc_reader process crashes. A software upgrade is recommended.

Background

We have identified a software issue with your selected release which can affect the use of this software. Review this field notice in order to determine if the issue applies to your environment. You can proceed to download this software if you have no concerns with the issue.

For more comprehensive information about what is included in this software, refer to the Cisco APIC documentation.

Problem Symptom

After a Cisco APIC has completed the upgrade process from a release earlier than 4.1(1) to Release 4.1(1) or later and the Cisco APIC has reloaded, the ifc_reader crashes about six times in seven minutes. Afterward, the ifc_reader service stops, which causes Cisco APIC communication issues. The ifc_reader data management engine (DME) issues are not reflected in the attribute/value (AV) health values, rvread command output, nor the Cisco APIC GUI. The acidiag avread and rvread commands along with the Cisco APIC GUI report a fully fit cluster. Cisco APIC GUI alarms raise a "split fabric" alert, and crashes in the NGINX process can be observed.

 

Switch Symptoms

After the Cisco APICs are upgraded from a release earlier than 4.1(1) to Release 4.1(1) or later, all of the switches start to see NGINX DME crashes every few minutes. The rate of crashes increases with the rate of uribv4Nexthop.type API queries, which results in switch queries.

After the NGINX process receives 250 instances of the offending query, the switch will cut off the interfaces since the switch has reached a failed state. This leads to a loss of network connectivity on the affected devices.

 

Conditions

This issue is triggered when these conditions are met:

1. A user or monitoring system sends a query to the Cisco APIC or leaf switch with this URL and filter:
   • URL: /api/class/uribv4Route.json
   • Filter: (eq(uribv4Nexthop.type,"")
2. The Cisco APIC software has been upgraded to a 4.2 release.
3. The Cisco Application Centric Infrastructure (ACI) switches still run the older release.

Workaround/Solution

Solution

Customers should upgrade to the software release 4.2(6d) or later.

 

Workaround

These applications send the known uribv4Nexthop.type query which can induce the crashes during mixed OS mode:

• The ConnectivityCompliance APIC App by AlgoSec
• The AlgoSec Firewall Analyzer

 

Note that any repeated query for uribn4Nexthop.type can induce unexpected behaviors in a mixed OS state. These two applications send the query as part of a pre-packaged application.

 

If any source/app/script/etc sends queries for uribv4Nexthop.type, which includes the two applications, the first action should be to disable/stop queries from it until the entire fabric is upgraded to a fixed version.

 

If the APIC ifc_reader is unable to recover on its own after the queries are stopped:

• Contact TAC, which can restart the ifc_reader service as root.
• If that does not resolve the issue, use acidiag restart mgmt to restart all DMEs on all Cisco APICs for the cluster to function correctly.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.