Field Notice: FN - 72013 - Cisco APIC-EM Root Certificate Expiration Causes All IWAN DMVPN Connections to Fail - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	18-Dec-20	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	APIC-EM Software	1	1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.6.1 (IWAN), 1.6.1 (IWAN-MIGR)), 1.6.2 (IWAN)	Versions 1.6.3 and 1.6.4 are also affected

Defect Information

Defect ID	Headline
CSCvw43646	APIC-EM Root CA Certificate expires after 5 years from installation date

Problem Description

The Public Key Infrastructure (PKI) root certificate used in Cisco Application Policy Infrastructure Controller-Enterprise Module (APIC-EM) expires five years after the product is initially installed. Once the root certificate expires it cannot be renewed and all secure connections initiated by Cisco APIC-EM fail, which includes Intelligent WAN (IWAN) Dynamic Multipoint VPN (DMVPN) connections.

Background

Cisco APIC-EM establishes a PKI private certificate authority (CA) to manage keys and issue intermediate digital certificates to participating entities such as hosts, network devices, and users within its domain. Cisco APIC-EM establishes a root certificate used to sign all intermediate certificates generated by the private CA. That root certificate is established when the product is initially installed and has a fixed five year expiration date. Subsequent software upgrades do not affect the root certificate expiration date. It remains fixed to the original installation date and does not auto-renew. Once the root certificate expires, all other intermediate certificates signed by the root are invalidated and secure trust point connections fail. Cisco IWAN deployments that use the Cisco APIC-EM IWAN application lose connections to the "spokes".

If the root certificate expires in a Cisco APIC-EM affected release it cannot be renewed. Recovery requires manual intervention on Cisco APIC-EM, its applications, and all devices that contain certificates signed by the expired root certificate.

Problem Symptom

It is critically important to identify an expiring root certificate before it expires. Once the root certificate expires, Cisco APIC-EM will no longer establish trust point connections or generate any intermediate PKI certificates. Without trust point connections, Cisco IWAN DMVPN deployments will no longer work.

Devices with expiring certificates will generate a syslog warning once the certificate is within one week of expiration.

Example:

Nov 20 21:36:47.879: %PKI-1-CERT_EXPIRY_ALERT: ID Certificate belonging to trustpoint sdn-network-infra-iwan will expire in 0 Days 23 hours 56 mins 30 secs.

There are two ways to view the Cisco APIC-EM root certificate expiration date.

Option 1. Use the CLI on One of the Routers in the IWAN Deployment to Inspect the PKI Certificate

Complete these steps in order to view the certificate information:

1. Enter the show crypto pki certificates sdn-network-infra-iwan CLI command.
2. Find the section named "CA Certificate".
3. Locate the validity date "end date". This is the date that the root certificate expires.

   Example:

   Router#show crypto pki certificates sdn-network-infra-iwan
   Certificate
     Status: Available
     Certificate Serial Number (hex): 507AAB6D1A222BD2
     Certificate Usage: General Purpose
     Issuer:
       cn=sdn-network-infra-ca
     Subject:
       Name: Router.cisco.com
       hostname=Router.cisco.com
       cn=ISR4431/K9_SERIALNUMBER_sdn-network-infra-iwan
     Validity Date:
       start date: 21:49:45 CDT Jun 23 2020
       end   date: 21:49:45 CDT Jun 23 2021
       renew date: 21:49:44 CDT Apr 11 2021
     Associated Trustpoints: sdn-network-infra-iwan
     Storage: nvram:sdn-network-#2BD2.cer
   
   CA Certificate
     Status: Available
     Certificate Serial Number (hex): 0D75068BC58DBF75
     Certificate Usage: Signature
     Issuer:
       cn=sdn-network-infra-ca
     Subject:
       cn=sdn-network-infra-ca
     Validity Date:                                  ---> Validity of the Root CA certificate
       start date: 11:23:36 CDT Jun 12 2019
       end   date: 11:23:36 CDT Jun 10 2024          ---> Root certificate expiration date
     Associated Trustpoints: sdn-network-infra-iwan
     Storage: nvram:sdn-network-#BF75CA.cer

 

Option 2. Use the CLI on the Cisco APIC-EM Console to View the Root CA Certificate Information

Complete these steps in order to view the certificate information:

1. Enter the ssh $(grape instance status | grep ejbca | awk '{print $5}') CLI command in order to attach to the Enterprise Java Beans Certificate Authority (EJBCA) service.
2. Once attached to the EJBCA service, enter the cd /opt/cisco/grapevine-services/apic-em-jboss-ejbca CLI command in order to move to the EJBCA service directory.
3. Enter the ls CLI command in order to display the contents of the directory. The contents will show the version of the EJBCA service.
4. Enter the cd CLI command in order to move to the directory named after the EJBCA service version.

   Example:

   cd 5.1.65.30024/ejbca-ejb-cli

5. Enter the java -jar ejbca-ejb-cli.jar ca listcas CLI command in order to display the Root CA information.
6. Find the subject entry named "CN=sdn-network-infra-ca".
7. Locate the "Expire time", which is beneath the "CN=sdn-network-infra-ca" subject entry. This is the date that the root certificate expires.

   Example:

   $ java -jar ejbca-ejb-cli.jar ca listcas
   CA Name: sdn-network-management-ca
   Id: 845937223
   Issuer DN: CN=sdn-network-management-ca
   Subject DN: CN=sdn-network-management-ca
   Type: 1
   Expire time: Wed Oct 17 08:02:40 UTC 2040
   Signed by: 1
   CA Name: sdn-network-infra-ca
   Id: -503229956
   Issuer DN: CN=sdn-network-infra-ca
   Subject DN: CN=sdn-network-infra-ca
   Type: 1
   Expire time: Tue Oct 21 08:03:01 UTC 2025         ---> Expiration time of the Root CA Certificate
   Signed by: 1
   CA Name: ManagementCA
   Id: 1652389506
   Issuer DN: CN=ManagementCA,O=EJBCA Sample,C=SE
   Subject DN: CN=ManagementCA,O=EJBCA Sample,C=SE
   Type: 1
   Expire time: Sun Sep 22 11:10:20 UTC 2024
   Signed by: 1
   (grapevine)

Workaround/Solution

Cisco has created a software patch for Cisco APIC-EM that fixes the root certificate renewal problem. The patch implements a root certificate rollover capability that works automatically as part of the existing PKI certificate renewal request function initiated by devices. Customers are highly recommended to upgrade their Cisco APIC-EM systems to a software version with the patch. The patch is available for Cisco APIC-EM Releases 1.6.2, 1.6.3, and 1.6.4. The software patch versions are 1.6.2.60027, 1.6.3.60027, and 1.6.4.60027 respectively. Patch software is available for download from the Cisco.com Software Download Center along with the Cisco APIC-EM release software images.

Follow the instructions in the "Install the Hotfix Patch" section of the Release Notes for Application Policy Infrastructure Controller Enterprise Module document for your release (1.6.2.60027, 1.6.3.60027, or 1.6.4.60027). Once the patch is installed, network devices managed by Cisco APIC-EM will receive new root certificate credentials from the Cisco APIC-EM CA automatically as part of their normal certificate renewal process. No additional intervention is required.

Note: The patch software cannot recover a Cisco APIC-EM deployment after the root certificate has already expired. Contact the Cisco Technical Assistance Center (TAC) for assistance with a manual recovery procedure if your root certificate has expired.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.