Field Notice: FN72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality - Software Upgrade Recommended

For affected versions of the Cisco IOS XE® software, some Secure Sockets Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly. 

The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by the Cisco IOS XE software to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates were issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.

Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.

This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.

Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.

For affected versions of Cisco IOS XE sofware, devices will be unable to connect to the Smart Licensing and Smart Call Home services hosted by Cisco. Smart licenses might fail entitlement and reflect an Out of Compliance status.

The features that use Smart Licensing will continue to function for one year after the last successful secure connection. Some Smart Licensing symptoms are:

• The device might indicate a failure to communicate with the Smart Licensing server within 30 days from the last successful connection.
• The device will show the "Authorization Expired" state if there is no communication with the Smart Licensing server within 90 days.
• The device will show the "Unregistered" state if there is no communication with the Smart Licensing server after one year and the licensed features usage become suspended.

Note: Offline licensing, such as Permanent License Reservation (PLR) and Specific License Reservation (SLR), is not affected by the certificate change on the Smart Licensing server. Wireless LAN Controllers and Wireless Access Points will continue to function even when smart licensing failures occur.

For additional information, refer to the Cisco Smart Licensing Guide, Wireless LAN Controllers Guide, Catalyst Switching Guide, and Enterprise Routing Guide for your specific version of Cisco IOS XE software.

These error logs may be observed in the affected device:

%SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Fail to send out Call Home HTTP message.

%SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco Smart Software Manager (CSSM) : Communication message send error for udi <>

%SMART_LIC-3-AGENT_REG_FAILED: Smart Agent for Licensing Registration with the Cisco Smart Software Manager (CSSM) failed: Fail to send out Call Home HTTP message.

%CALL_HOME-5-SL_MESSAGE_FAILED: Fail to send out Smart Licensing message to: https://tools.cisco.com/its/service/oddce/services/DDCEService (ERR 205 : Request Aborted)

%CALL_HOME-5-SL_MESSAGE_FAILED: Fail to send out Smart Licensing message to: https://tools.cisco.com/its/service/oddce/services/DDCEService (ERR 207 : Connection time out)

Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends these two options to add the new IdenTrust Commercial Root CA 1 certificate to the affected devices.

• Software Upgrade
• Manual Certificate Update

Software Upgrade

For Cisco IOS XE-based products, upgrade the software to any of the fixed releases - 16.12.6, 17.3.4, 17.4.2, 17.5.1, 17.6.1, 17.7.1 or later to resolve the root CA certificate issue for affected platforms.

Manual Certificate Update

In order to resolve the issue without a software upgrade, you can implement either of these workarounds.

Workaround 1

For Cisco IOS XE-based products, use the PKI Trustpool Management feature in order to resolve the issue without an upgrade to the Cisco IOS XE software.

The IdenTrust Commercial Root CA 1 certificate is included in the latest ios_core.p7b file available at http://www.cisco.com/security/pki/trs/ios_core.p7b.

Enter these commands in order to auto-import the IdenTrust Commercial Root CA 1 certificate.

• Enter this CLI command to download the IdenTrust Commercial Root CA 1 certificate file from Cisco and manually import into the product's truststore.

  Device(config)# crypto pki trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

• If the "ios_core.p7b" file was previously copied to the product's flash, enter this CLI command to import the certificates file into the truststore.

  Device(config)# crypto pki trustpool import url flash:ios_core.p7b

Workaround 2

Use this workaround to manually import the IdenTrust Commercial Root CA 1 from the text below into the product's trust store. The IdenTrust Root CA 1 shown below complies with sha1WithRSAEncryption signature algorithm requirements.

1. Use a simple text editor, such as Notepad, and copy the certificate text (without the BEGIN CERTIFICATE and END CERTIFICATE lines).

   Manual Update of Trustpool via Terminal

   -----BEGIN CERTIFICATE-----
   MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK
   MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu
   VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw
   MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw
   JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG
   SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT
   3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU
   +ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp
   S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1
   bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi
   T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL
   vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK
   Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK
   dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT
   c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv
   l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N
   iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
   /zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD
   ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH
   6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt
   LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93
   nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3
   +wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK
   W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT
   AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq
   l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG
   4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ
   mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A
   7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H
   -----END CERTIFICATE-----

2. In order to enter the configuration mode, enter the config t command.
3. Enter the crypto pki trustpool import terminal command.
4. Paste the copied PEM-formatted CA certificate and press Enter two times. The system should respond with "% PEM files import succeeded".
5. Enter the exit command.
6. In order to write to memory, enter the wr mem command.
7. Enter the show crypto pki trustpool command. The output should now contain the IdenTrust certificate.

See this example:

Device#config t
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#crypto pki trustpool import terminal
% Enter PEM-formatted CA certificate.
% End with a blank line or "quit" on a line by itself.
MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK
MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu
VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw
MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw
JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT
3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU
+ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp
S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1
bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi
T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL
vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK
Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK
dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT
c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv
l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N
iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
/zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD
ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH
6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt
LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93
nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3
+wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK
W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT
AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq
l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG
4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ
mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A
7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H
% PEM files import succeeded.
Device(config)#exit
Device#wr mem
Destination filename [startup-config]?
Building configuration...
[OK]

Device#show crypto pki trustpool
Load for five secs: 30%/2%; one minute: 25%; five minutes: 27%
Time source is NTP, 23:40:09.537 CST Sat Mar 6 2021
CA Certificate
  Status: Available
  Certificate Serial Number (hex): 0A0142800000014523C844B500000002
  Certificate Usage: Signature
  Issuer: 
    cn=IdenTrust Commercial Root CA 1
    o=IdenTrust
    c=US
  Subject: 
    cn=IdenTrust Commercial Root CA 1
    o=IdenTrust
    c=US
  Validity Date: 
    start date: 02:12:23 CST Jan 17 2014
    end   date: 02:12:23 CST Jan 17 2034
  Associated Trustpoints: Trustpool 
  Trustpool: Downloaded


CA Certificate
  Status: Available
  Certificate Serial Number (hex): 0509
  Certificate Usage: Signature
  Issuer: 
    cn=QuoVadis Root CA 2
    o=QuoVadis Limited
    c=BM
  Subject: 
    cn=QuoVadis Root CA 2
    o=QuoVadis Limited
    c=BM
  Validity Date: 
    start date: 02:27:00 CST Nov 25 2006
    end   date: 02:23:33 CST Nov 25 2031
  Associated Trustpoints: Trustpool
  Trustpool: Built-In
<<output snipped>>

Workaround 3

This workaround is only applicable to devices that run in SD WAN mode.

1. Place the certificate under bootflash:.

   Go to shell under the bootflash directory and create a file named "Test123.ca". This file HAS to be named as <Trustpoint-name>.ca. Add this certificate content to the file:

   -----BEGIN CERTIFICATE-----
   MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK
   MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu
   VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw
   MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw
   JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG
   SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT
   3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU
   +ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp
   S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1
   bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi
   T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL
   vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK
   Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK
   dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT
   c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv
   l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N
   iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
   /zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD
   ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH
   6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt
   LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93
   nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3
   +wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK
   W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT
   AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq
   l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG
   4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ
   mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A
   7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H
   -----END CERTIFICATE-----

2. Enter the config command.

   crypto pki trustpoint Test123
   	enrollment url bootflash:
   	Revocation-check none
   	Fingerprint DF717EAA4AD94EC9558499602D48DE5FBCF03A25
   	Commit

3. From exec mode, enter crypto pki authenticate Test123.

These products are affected:

• Cisco 1000 Series Integrated Services Routers
• Cisco 1100 Integrated Services Router
• Cisco 4000 Series Integrated Services Routers
• Cisco 4221 Integrated Services Router
• Cisco 4321 Integrated Services Router
• Cisco 4331 Integrated Services Router
• Cisco 4351 Integrated Services Router
• Cisco 4431 Integrated Services Router
• Cisco 4451-X Integrated Services Router
• Cisco 4461 Integrated Services Router
• Cisco ASR 1000 Series Aggregation Services Routers
• Cisco ASR 1000 Series IOS XE SD-WAN
• Cisco ASR 1000 Series Route Processor (RP2)
• Cisco ASR 1000 Series Route Processor (RP3)
• Cisco ASR 1001-HX Router
• Cisco ASR 1001-X Router
• Cisco ASR 1002-HX Router
• Cisco ASR 1002-X Router
• Cisco ASR 1004 Router
• Cisco ASR 1006 Router
• Cisco ASR 1006-X Router
• Cisco ASR 1009-X Router
• Cisco ASR 1013 Router
• Cisco Catalyst 1101 Rugged Router
• Cisco Catalyst 3650 Series Switches
• Cisco Catalyst 3850 Series Switches
• Cisco Catalyst 8000V Edge Software
• Cisco Catalyst 8200 Series Edge Platforms
• Cisco Catalyst 8300 Series Edge Platforms
• Cisco Catalyst 8500 Series Edge Platforms
• Cisco Catalyst 8500L Series Edge Platforms
• Cisco Catalyst 9200 Series Switches
• Cisco Catalyst 9200L Switch Stack
• Cisco Catalyst 9300 Series Switches
• Cisco Catalyst 9300L Series Switches
• Cisco Catalyst 9400 Series Switches
• Cisco Catalyst 9404R Switch
• Cisco Catalyst 9407R Switch
• Cisco Catalyst 9410R Switch
• Cisco Catalyst 9500 Series Switches
• Cisco Catalyst 9600 Series Switches
• Cisco Catalyst 9606R Switch
• Cisco Catalyst 9800 Wireless Controllers for Cloud
• Cisco Catalyst 9800-40 Wireless Controller
• Cisco Catalyst 9800-80 Wireless Controller
• Cisco Catalyst 9800-CL Wireless Controller for Cloud
• Cisco Catalyst 9800-L Wireless Controller
• Cisco Catalyst 9800-L-C Wireless Controller
• Cisco Catalyst 9800-L-F Wireless Controller
• Cisco Catalyst IR1100 Rugged Series Routers
• Cisco Cloud Services Router 1000V Series
• Cisco CSR 1000V Series IOS XE SD-WAN
• Cisco Embedded Wireless Controller on Catalyst 9115AX Access Points
• Cisco Embedded Wireless Controller on Catalyst 9117AX Access Points
• Cisco Embedded Wireless Controller on Catalyst 9120AX Access Points
• Cisco Embedded Wireless Controller on Catalyst 9130AX Access Points
• Cisco Embedded Wireless Controller on Catalyst Access Points
• Cisco ESR6300 Embedded Series Router
• Cisco ESR6300 Embedded Series Routers
• Cisco Integrated Services Virtual Router
• Cisco ISR 1000 Series IOS XE SD-WAN
• Cisco ISR 4000 Series IOS XE SD-WAN
• Cisco Route Processors and Route Switch Processors
• Cisco XE SD-WAN Routers