Field Notice: FN - 70527 - Expressway and VCS: QuoVadis Root CA 2 Decommission Might Affect Incident Reporting, Calls To and From Webex, Webex Edge Audio, and Hybrid Calling Functionality - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
2.0	25-Feb-22	Updated the Problem Description, Background, Problem Symptom, and Workaround/Solution Sections
1.0	01-Apr-21	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	TelePresence Software	X12	X12.5.0, X12.5.1, X12.5.2, X12.5.3, X12.5.4, X12.5.5, X12.5.6, X12.5.7, X12.5.8, X12.5.9, X12.6.0, X12.6.1, X12.6.2, X12.6.3, X12.6.4, X12.7.0, X12.7.1	All versions of Expressway and VCS up to and including X12.7.1 are affected
NON-IOS	TelePresence Software	X8	X8.1, X8.1.1, X8.1.2, X8.10.0, X8.10.1, X8.10.2, X8.10.3, X8.10.4, X8.11.0, X8.11.1, X8.11.2, X8.11.3, X8.11.4, X8.2, X8.2.1, X8.2.2, X8.5, X8.5.1, X8.5.2, X8.5.3, X8.6, X8.6.1, X8.7, X8.7.1, X8.7.2, X8.7.3, X8.8, X8.8.1, X8.8.2, X8.8.3, X8.9, X8.9.1, X8.9.2	All versions of Expressway and VCS up to and including X12.7.1 are affected
NON-IOS	TelePresence Software	X7	X7.0, X7.0.1, X7.0.2, X7.0.3, X7.1, X7.2, X7.2.1, X7.2.2, X7.2.3, X7.2.4	All versions of Expressway and VCS up to and including X12.7.1 are affected
NON-IOS	TelePresence Software	X6	X6.0, X6.1	All versions of Expressway and VCS up to and including X12.7.1 are affected
NON-IOS	TelePresence Software	X5	X5.2	All versions of Expressway and VCS up to and including X12.7.1 are affected

Defect Information

Defect ID	Headline
CSCvx00489	QuoVadis root CA decommission

Problem Description

For affected versions of the Expressway and Video Communication Server (VCS) software, some Secure Sockets Layer (SSL) certificates issued from the QuoVadis root certificate authority (CA) trust chain before March 31, 2021 cannot be renewed from this CA. Once those certificates expire on devices or are removed from the Cisco cloud servers, functions such as Incident Reporting, Calls To and From Webex, Webex Edge Audio, and Hydrid Calling will fail to establish secure connections to Cisco and might not operate properly.

Background

The QuoVadis Root CA 2 Public Key Infrastructure (PKI) used by Expressway and VCS software to issue SSL certificates is subject to an industry-wide issue that affects revocation abilities. Due to this issue, no new QuoVadis Root CA 2 certificates will be issued or renewed by Cisco after March 31, 2021. This affects certificate renewals on devices, Cisco cloud servers, and third-party services.

Certificates issued before the QuoVadis Root CA 2 was decommissioned will continue to be valid. However, the certificates will not renew when they expire on either the device or the Cisco cloud server. This will cause functions such as Incident Reporting, Calls To and From Webex, Webex Edge Audio, and Hybrid Calling to fail to establish secure connections to Cisco cloud servers.

This table shows a summary of the QuoVadis Root CA 2 certificate expiration dates for affected Cisco services.

Cisco Cloud Server	QuoVadis Certificate Expiration Date	Affected Services
cc-reports.cisco.com	February 10, 2022	Incident Reporting
Webex Calls Servers (Result of _sips._tcp.<domain>)	March 16, 2022	Calls To and From Webex (DNS Zone)
Webex Calls Servers (Result of _sips._tcp.<domain>)	July 21, 2022	Webex Edge Audio Calls (Webex Zone)
Hybrid Calling Servers (Result of _sips._tcp.callservice.webex.com)	May 7, 2022	Hybrid Calling (Webex Zone)

Problem Symptom

Expiration of the QuoVadis Root CA 2 certificates affects these services with the associated symptoms.

Affected Services	Symptoms for Affected Services
Incident Reporting	Failure to connect to the server to upload crash report
Calls To and From Webex	Failure to connect to the server to establish call
Webex Edge Audio	Failure to connect to the server to establish call
Hybrid Calling	Failure to connect to the server to establish call

 

Update: Smart Licensing is not impacted in this product.

For Expressway and VCS devices, affected devices will be unable to connect to the Incident Reporting, Calls To and From Webex, Webex Edge Audio, and Hybrid Calling services hosted by Cisco.

• For Incident Reporting, when the server experiences a crash, an alarm is raised when the connection cannot be established due to the certificate trust store not being updated on the Expressway or VCS server. The alarm is "Failed to send Crash Report" with the description "Failed to send Crash Report to Server – https://cc-reports.cisco.com/submitapplicationerror/, Server Certificate Verification Failed".

  An example alarm is shown here:

  

   

• For Calls To and From Webex services, you might observe calls being unable to connect.
• For Webex Edge Audio services, you might observe calls being unable to connect.
• For Hybrid Calling services, you might observe calls being unable to connect.

Workaround/Solution

Cisco has migrated from the QuoVadis Root CA 2 to the IdenTrust Commercial Root CA 1 for SSL certificates. Cisco recommends these two options to add the new IdenTrust Commercial Root CA 1 certificate to the Expressway and VCS.

• Software Upgrade
• Manual Certificate Update

Software Upgrade

Upgrade to Release X14.0 of Expressway or VCS or later in order to resolve the root CA certificate issue for affected platforms.

 

Manual Certificate Update

To resolve the issue without a software upgrade, complete these steps:

1. Create a local file that contains the IdenTrust Commercial Root CA 1 certificate. To do so, copy and paste the IdenTrust Commercial Root CA 1 certificate shown here into a text file, using an application such as Notepad, and save the file as identrust_RootCA1.pem or identrust_RootCA1.cer.

   Note: The certificate must include and begin with “-----BEGIN CERTIFICATE-----” and end with “-----END CERTIFICATE-----“, without any extra space inserted.

   The updated IdenTrust Commercial Root CA 1 certificate is shown here and complies with sha1WithRSAEncryption signature algorithm requirements. Alternately, the IdenTrust Commercial Root CA 1 certificate can be downloaded from the IdenTrust web site.

   -----BEGIN CERTIFICATE-----
   MIIFYDCCA0igAwIBAgIQCgFCgAAAAUUjyES1AAAAAjANBgkqhkiG9w0BAQsFADBK
   MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVu
   VHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTQwMTE2MTgxMjIzWhcNMzQw
   MTE2MTgxMjIzWjBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScw
   JQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEwggIiMA0GCSqG
   SIb3DQEBAQUAA4ICDwAwggIKAoICAQCnUBneP5k91DNG8W9RYYKyqU+PZ4ldhNlT
   3Qwo2dfw/66VQ3KZ+bVdfIrBQuExUHTRgQ18zZshq0PirK1ehm7zCYofWjK9ouuU
   +ehcCuz/mNKvcbO0U59Oh++SvL3sTzIwiEsXXlfEU8L2ApeN2WIrvyQfYo3fw7gp
   S0l4PJNgiCL8mdo2yMKi1CxUAGc1bnO/AljwpN3lsKImesrgNqUZFvX9t++uP0D1
   bVoE/c40yiTcdCMbXTMTEl3EASX2MN0CXZ/g1Ue9tOsbobtJSdifWwLziuQkkORi
   T0/Br4sOdBeo0XKIanoBScy0RnnGF7HamB4HWfp1IYVl3ZBWzvurpWCdxJ35UrCL
   vYf5jysjCiN2O/cz4ckA82n5S6LgTrx+kzmEB/dEcH7+B1rlsazRGMzyNeVJSQjK
   Vsk9+w8YfYs7wRPCTY/JTw436R+hDmrfYi7LNQZReSzIJTj0+kuniVyc0uMNOYZK
   dHzVWYfCP04MXFL0PfdSgvHqo6z9STQaKPNBiDoT7uje/5kdX7rL6B7yuVBgwDHT
   c+XvvqDtMwt0viAgxGds8AgDelWAf0ZOlqf0Hj7h9tgJ4TNkK2PXMl6f+cB7D3hv
   l7yTmvmcEpB4eoCHFddydJxVdHixuuFucAS6T6C6aMN7/zHwcz09lCqxC0EOoP5N
   iGVreTO01wIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
   /zAdBgNVHQ4EFgQU7UQZwNPwBovupHu+QucmVMiONnYwDQYJKoZIhvcNAQELBQAD
   ggIBAA2ukDL2pkt8RHYZYR4nKM1eVO8lvOMIkPkp165oCOGUAFjvLi5+U1KMtlwH
   6oi6mYtQlNeCgN9hCQCTrQ0U5s7B8jeUeLBfnLOic7iPBZM4zY0+sLj7wM+x8uwt
   LRvM7Kqas6pgghstO8OEPVeKlh6cdbjTMM1gCIOQ045U8U1mwF10A0Cj7oV+wh93
   nAbowacYXVKV7cndJZ5t+qntozo00Fl72u1Q8zW/7esUTTHHYPTa8Yec4kjixsU3
   +wYQ+nVZZjFHKdp2mhzpgq7vmrlR94gjmmmVYjzlVYA211QC//G5Xc7UI2/YRYRK
   W2XviQzdFKcgyxilJbQN+QHwotL0AMh0jqEqSI5l2xPE4iUXfeu+h1sXIFRRk0pT
   AwvsXcoz7WL9RccvW9xYoIA55vrX/hMUpu09lEpCdNTDd1lzzY9GvlU47/rokTLq
   l1gEIt44w8y8bckzOmoKaT+gyOpyj4xjhiO9bTyWnpXgSUyqorkqG5w2gXjtw+hG
   4iZZRHUe2XWJUc0QhJ1hYMtd+ZciTY6Y5uN/9lu7rs3KSoFrXgvzUeF0K+l+J6fZ
   mUlO+KWA2yUPHGNiiskzZ2s8EIPGrd6ozRaOjfAHN3Gf8qv8QfXBi+wAN10J5U6A
   7/qxXDgGpRtK4dw4LTzcqx+QGtVKnO7RcGzM7vRX+Bi6hG6H
   -----END CERTIFICATE-----

2. From the Expressway administrator web user interface, choose Maintenance > Security > Trusted CA certificate.

   

    

3. Click Choose File and then choose the local file that contains the IdenTrust Commercial Root CA 1 certificate.

4. Click Append CA certificate.

   

    

   The IdenTrust certificate will appear in the list of trusted CA certificates. No restart is necessary.

   

    

For more information, see the March 2021 Cisco Webex Root CA Certificate update for Expressway video.

For More Information

Cisco has created a web page to provide customers and partners with additional information on this issue. Consult the QuoVadis Root CA 2 Decommission page for a full list of products affected, associated Field Notices, and frequently asked questions.

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.