Field Notice: FN - 72298 - IE3x00 Rugged Series, Embedded Series ESS3x00, and ESS9300 Switches: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing and Smart Call Home Functionality - Software Upgrade Recommended

Available Languages

Updated:February 25, 2022

Document ID:FN72298

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.2	23-Feb-22	Updated the Problem Description, Background, Problem Symptom, and Workaround/Solution Sections
1.1	22-Dec-21	Updated the Products Affected Section
1.0	17-Dec-21	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	IOSXE	16	16.10.1, 16.10.1a, 16.10.1b, 16.10.1c, 16.10.1d, 16.10.1e, 16.10.1f, 16.10.1g, 16.10.1i, 16.10.1s, 16.11.1, 16.11.1a, 16.11.1b, 16.11.1c, 16.11.1s, 16.12.1, 16.12.1a, 16.12.1c, 16.12.1s, 16.12.1t, 16.12.1w, 16.12.1x, 16.12.1y, 16.12.1z, 16.12.1z1, 16.12.1z2, 16.9.1, 16.9.1a, 16.9.1b, 16.9.1c, 16.9.1d, 16.9.1s	ESS3x00 Cisco IOS XE Software Releases 17.6.1 or later have the fix. All previous releases need to use the suggested workaround to address the issue.
NON-IOS	IOSXE	17	17.1.1, 17.1.1a, 17.1.1s, 17.1.1t, 17.2.1, 17.2.1a, 17.2.1r, 17.2.1v, 17.3.1, 17.3.1a, 17.3.1w, 17.3.1x, 17.3.2, 17.3.2a, 17.3.3, 17.3.3a, 17.4.1, 17.4.1a, 17.4.1b, 17.5.1, 17.5.1a	ESS3x00 Cisco IOS XE Software Releases 17.6.1 or later have the fix. All previous releases need to use the suggested workaround to address the issue.
NON-IOS	IOSXE	16	16.10.1, 16.10.1a, 16.10.1b, 16.10.1c, 16.10.1d, 16.10.1e, 16.10.1f, 16.10.1g, 16.10.1i, 16.10.1s, 16.11.1, 16.11.1a, 16.11.1b, 16.11.1c, 16.11.1s, 16.12.1, 16.12.1a, 16.12.1c, 16.12.1s, 16.12.1t, 16.12.1w, 16.12.1x, 16.12.1y, 16.12.1z, 16.12.1z1, 16.12.1z2, 16.9.1, 16.9.1a, 16.9.1b, 16.9.1c, 16.9.1d, 16.9.1s	IE3x00 Cisco IOS XE Software Releases 17.6.1 or later have the fix. All previous releases need to use the suggested workaround to address the issue.
NON-IOS	IOSXE	17	17.1.1, 17.1.1a, 17.1.1s, 17.1.1t, 17.2.1, 17.2.1a, 17.2.1r, 17.2.1v, 17.3.1, 17.3.1a, 17.3.1w, 17.3.1x, 17.3.2, 17.3.2a, 17.3.3, 17.3.3a, 17.4.1, 17.4.1a, 17.4.1b, 17.5.1, 17.5.1a	IE3x00 IOS XE Software Releases 17.6.1 or later have the fix. All previous releases need to use the suggested workaround to address the issue.
NON-IOS	IOSXE	17	17.4.1, 17.4.1a, 17.4.1b, 17.4.2, 17.4.2a, 17.5.1, 17.5.1a	ESS9300 IOS XE Software Releases 17.6.1 or later have the fix. All previous releases need to use the suggested workaround to address the issue.

Defect Information

Defect ID	Headline
CSCvx00521	QuoVadis root CA decommission impacting Smart Licensing and Smart Call Home Functionality

Problem Description

For affected IOS XE versions of Cisco Catalyst (IE3x00) Rugged Series Switches, Cisco Catalyst (IE34xx ) Heavy Duty Series Switches, Cisco Embedded Services (ESS3x00) Series and Cisco Embedded Services (ESS9300) Switches, some Secure Sockets Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates (issued by QuoVadis) expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly.

Background

The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by Cisco IOS^® XE software to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.

Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.

This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.

Cisco Cloud Server	QuoVadis Certificate Expiration Date	Affected Services
tools.cisco.com	February 5, 2022	Smart Licensing Smart Call Home
smartreceiver.cisco.com	January 26, 2023	Smart Licensing

Problem Symptom

Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.

Affected Services	Symptoms for Affected Services
Smart Licensing	Failure to connect to the server (Details are provided in this section)
Smart Call Home	Failure to connect to the server and the Call-Home HTTP request fails

For Cisco IOS XE based devices, affected devices will be unable to connect to the Smart Licensing and Smart Call Home services hosted by Cisco. Smart licenses might fail entitlement and reflect an Out of Compliance status.

The features that use Smart Licensing will continue to function for one year after the last successful secure connection. Some Smart Licensing symptoms are:

The device might indicate a failure to communicate with the Smart Licensing server within 30 days from the last successful connection.
The device will show the "Authorization Expired" state if there is no communication with the Smart Licensing server within 90 days.
The device will show the "Unregistered" state if there is no communication with the Smart Licensing server after one year and the licensed features usage become suspended.

Note: Offline licensing, such as Permanent License Reservation (PLR) and Specific License Reservation (SLR), is not affected by the certificate change on the Smart Licensing server.

For additional information, refer to the Cisco Smart Licensing Guide, Smart Licensing using Policy on Catalyst Switching Platforms, and Release Notes for Cisco Catalyst IE3x00 Rugged, IE3400 Heavy Duty, ESS3300, and ESS9300 Series Switches.

Troubleshooting Tech Notes

Cisco Smart Licensing - Troubleshooting Steps and Considerations on Catalyst Platforms

If there is an error in communication with the server due to trust or other reasons, these error messages might be observed:

%CALL_HOME-5-SL_MESSAGE_FAILED: Fail to send out Smart Licensing message to:https://<;ip>/its/service/oddce/services/DDCEService (ERR 205 : Request Aborted) %SMART_LIC-3-COMM_FAILED:Communications failure with the Cisco Smart Software Manager or satellite: Fail to send out Call Home HTTP message. %SMART_LIC-3-AUTH_RENEW_FAILED:Authorization renewal with the Cisco Smart Software Manager or satellite: Communication message send error for udi PID:XXX, SN: XXX %PKI-3-CRL_FETCH_FAIL: CRL fetch for trustpoint SLA-TrustPoint failed Reason : Failed to select socket. Timeout : 5 (Connection timed out) %PKI-3-CRL_FETCH_FAIL: CRL fetch for trustpoint SLA-TrustPoint failed Reason : Failed to select socket. Timeout : 5 (Connection timed out) %SMART_LIC-3-OUT_OF_COMPLIANCE: One or more entitlements are out of compliance %SMART_LIC-3-AUTH_RENEW_FAILED: Authorization renewal with the Cisco Smart Software Manager (CSSM) : Error received from Smart Software Manager: Data and signature do not match for udi PID:IE3300,SN:XXXXXXXXXXX SAEVT_DEREGISTER_STATUS msgStatus="LS_INVALID_DATA" error="Missing Id cert serial number field; Missing signing cert serial number field; Signed data and certificate does not match %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : No detailed information given

Workaround/Solution

Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends these two options to add the new IdenTrust Commercial Root CA 1 certificate to the affected devices.

Software Upgrade
Manual Certificate Update

Software Upgrade

For Cisco IOS XE based products, upgrade the software to the fixed Release 17.6.1 or later in order to resolve the root CA certificate issue for affected platforms.

Manual Certificate Update

In order to resolve the issue without a software upgrade, either of these two workarounds can be implemented.

Workaround 1

Enter this CLI command in order to manually import the IdenTrust Commercial Root CA 1 into the product truststore:

Device(config)# crypto pki trustpool import url http://www.cisco.com/security/pki/trs/ios_core.p7b

Or if it is already copied to the device flash, enter this command:

Device(config)# crypto pki trustpool import url flash:ios_core.p7b

Workaround 2

Enter the CLI commands in the procedure in order to manually import the IdenTrust Commercial Root CA 1 into the product truststore.

The updated IdenTrust Root CA 1 is shown here and complies with sha1WithRSAEncryption signature algorithm requirements.

Manual Update of Trustpool via Terminal

Procedure:

Use a simple text editor such as Notepad and copy the certificate (without the BEGIN CERTIFICATE and END CERTIFICATE lines).
In order to enter the configuration mode, enter the config t command.
Enter the crypto pki trustpool import terminal command.
Paste the copied PEM-formatted CA certificate and press Enter two times. The system should respond with "% PEM files import succeeded".
Enter exit.
In order to write to memory, enter the wr mem command.
Enter the show crypto pki trustpool command.

See this example:

Device#config t
Enter configuration commands, one per line.  End with CNTL/Z.

Switch(config)#crypto pki trustpool import terminal

% Enter PEM-formatted CA certificate.

% End with a blank line or "quit" on a line by itself.

MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK
MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu
VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw
MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw
JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG
SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT
3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU
+ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp
S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1
bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi
T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL
vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK
Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK
dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT
c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv
l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N
iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
/zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD
ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH
6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt
LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93
nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3
+wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK
W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT
AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq
l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG
4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ
mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A
7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H

% PEM files import succeeded.
Device(config)#exit

Device#wr mem

Destination filename [startup-config]?

Building configuration...

[OK]

Device#show crypto pki trustpool
Load for five secs: 30%/2%; one minute: 25%; five minutes: 27%
Time source is NTP, 23:40:09.537 CST Sat Mar 6 2021
CA Certificate
  Status: Available
  Certificate Serial Number (hex): 0A0142800000014523C844B500000002
  Certificate Usage: Signature
  Issuer: 
    cn=IdenTrust Commercial Root CA 1
    o=IdenTrust
    c=US
  Subject: 
    cn=IdenTrust Commercial Root CA 1
    o=IdenTrust
    c=US
  Validity Date: 
    start date: 02:12:23 CST Jan 17 2014
    end   date: 02:12:23 CST Jan 17 2034
  Associated Trustpoints: Trustpool 
  Trustpool: Downloaded

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 0509
  Certificate Usage: Signature
  Issuer: 
    cn=QuoVadis Root CA 2
    o=QuoVadis Limited
    c=BM
  Subject: 
    cn=QuoVadis Root CA 2
    o=QuoVadis Limited
    c=BM
  Validity Date: 
    start date: 02:27:00 CST Nov 25 2006
    end   date: 02:23:33 CST Nov 25 2031
  Associated Trustpoints: Trustpool
  Trustpool: Built-In
<<output snipped>>

For More Information

Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.