Field Notice: FN72415 - QuoVadis Root CA 2 Decommission Might Affect AireOS Smart Licensing and Smart Call Home, and Mobility Express Software Downloads from Cisco.com - Software Upgrade Recommended

For affected versions of the AireOS software, some Secure Sockets Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Smart Licensing and Smart Call Home will fail to establish secure connections to Cisco and might not operate properly. 

Smart Licensing and Smart Call Home have replaced the QuoVadis root certificate with one from IdenTrust. Normally, the new root certificate will automatically be downloaded from tools.cisco.com by the Wireless LAN Controller (WLC). However, if the network path between the WLC and tools.cisco.com blocks access to TCP ports 80 and 443 on tools.cisco.com, the certificate download will fail and the WLC will be unable to contact Smart Licensing or Smart Call Home.

Additionally, for affected versions of Mobility Express software, direct software download from Cisco.com does not work.

The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by AireOS software to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.

Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Smart Licensing and Smart Call Home to fail to establish secure connections to Cisco cloud servers.

This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.

Additionally, the Mobility Express software download changed the API and is non-backward compatible, so connecting to Cisco.com servers gives a connection failure.

Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.

AireOS Symptoms

For affected versions of AireOS, devices will be unable to connect to the Smart Licensing services hosted by Cisco. Smart licenses might fail with these messages:

• Failure reason: "Fail to send out Call Home HTTP message."
• In the WLC uplink you will see "Unknown CA."

  

Note: For additional information, refer to the Cisco Smart Licensing Guide for your specific version of AireOS software.

In order to determine whether or not you are affected, first find out if you use Smart Licensing and/or Smart Call Home. Enter these commands:

• show call-home config-local
• show license summary

If either of these services are enabled, you are susceptible. Follow the instructions in the Workaround/Solution section.

In order to verify that you are impacted by this issue, enable these debugs and show outputs on the WLC:

• debug license core all enable
• debug license events enable
• debug license errors enable
• debug license info enable
• show license summary
• show license tech-support
• show license all

These error logs might be observed on the affected device:

*Fri Mar 18 02:06:11.597 UTC: CH-LIB-TRACE: ch_pf_curl_head_init[111], init msg header
* SSL certificate problem: self signed certificate in certificate chain
*Fri Mar 18 02:06:13.432 UTC: CH-LIB-ERROR: ch_pf_curl_send_msg[483], failed to perform, err code 60, err string "Error"
*Fri Mar 18 02:06:13.432 UTC: CH-LIB-TRACE: ch_pf_http_unlock[215], unlock http mutex.
*Fri Mar 18 02:06:13.432 UTC: CH-LIB-TRACE: ch_pf_send_http[263], send http msg, result 35

In order to check the show license status, enter the show license status command.

(Cisco Controller) >show license status

Smart Licensing is ENABLED

Utility:

  Status: DISABLED

Data Privacy:

  Sending Hostname: yes

    Callhome hostname privacy: DISABLED

    Smart Licensing hostname privacy: DISABLED

  Version privacy: DISABLED

Transport:

 Type: Callhome

Registration:

  Status: REGISTERING - REGISTRATION IN PROGRESS

  Export-Controlled Functionality: NOT ALLOWED

  Initial Registration: FAILED on Mar 02 2022 13:13:48 CET

    Failure reason: Fail to send out Call Home HTTP message.

  Next Registration Attempt: Mar 02 2022 13:30:15 CET



License Authorization:

  Status: EVAL MODE

  Evaluation Period Remaining: 89 days, 23 hours, 58 minutes, 32 seconds



Export Authorization Key:

  Features Authorized:

    <none>

Cisco Mobility Express Symptoms

Software download from Cisco.com to Mobility Express does not work. If you run Mobility Express earlier than 8.10.181.0, the download icon in the home page (a down-arrow inside a circle) is red, and hovering over the icon shows this error:

Connection failure: 60. Peer certificate cannot be authenticated with given CA certificates.

If you run Mobility Express 8.10.181.0 or later, the msglog (ctrl/msg.txt in the Support Bundle) shows this error:

*emWeb: Sep 23 18:38:57.420: check latest version failed: Error parsing response from server

Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends one of these two options to add the new IdenTrust Commercial Root CA 1 certificate to the AireOS.

• Solution - Software upgrade (preferred)
• Workaround - open access to tools.cisco.com

Solution for AireOS WLCs (Preferred)

For AireOS devices, upgrade to one of the software versions shown in the table in order to resolve the root CA certificate issue for affected platforms.

Note: WLCs not listed here are not affected by this issue.

Workaround for AireOS WLCs

Configure your network to allow your WLC to access tools.cisco.com via TCP ports 80 and 443. Enter the show network summary command to see the WLC's DNS server. Query the DNS server that is configured on your WLC to find out the IP address(es) to which tools.cisco.com resolves.

Once the WLC has access to tools.cisco.com, it will automatically download the IdenTrust Commercial Root CA 1 certificate.

When the certificate update happens, there is no reboot required.

In order to confirm that the IdenTrust Commercial Root CA 1 certificate is installed on the WLC, enter the grep include IdenTrust "show certificate all" command.

Workaround for Mobility Express

Cisco no longer supports the Cisco.com download method for Mobility Express. Download new Mobility Express software from software.cisco.com via a browser, then load that software into Mobility Express via TFTP, SFTP, or HTTP. See Cisco Mobility Express User Guide, Cisco Wireless Release 8.10 for more details.

These products are affected:

• Cisco 3504 Wireless Controller
• Cisco 5520 Wireless Controller
• Cisco 8540 Wireless Controller
• Cisco Virtual Wireless Controller
• Cisco Mobility Express

Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.