Upgrade of Video Communication Server (VCS) / Expressway X14.x - Guide & FAQ

Important Notes About Upgrades To X14.2:
1. Expressway X14.2 only supports Smart Licensing.
2. Expressway X14.2 is capped at 2500 encrypted signaling sessions to endpoints and includes changes in the trafficserver behavior related to Cisco bug ID CSCwc69661 that can lead to MRA failures - Please read the Release Notes and Admin Guide before an upgrade to X14.2. Also refer to note 4 in the Pre-upgrade Actions section of this document for more information.
3. If you have a custom MTU size configured on the Expressway, it does get changed to the default of 1500 after the upgrade which causes problems in connections and media, this is tracked under the Cisco bug ID CSCwc74590 , so after the upgrade you must change the MTU size back to the previous MTU size that was configured before the upgrade.

Introduction

This document describes the Expressway upgrade process and is designed to guide you and answer the most commonly asked questions.

Background information

The information in this document applies to both Expressway and Video Communication Server (VCS). The document references Expressway but this can be interchanged with VCS.

Note: While this document is designed to help you with the upgrade, it does not replace the Expressway release notes. They are the source of truth.
Always refer to the Release notes for the destination version before you go ahead with the upgrade.

Important information for all deployments

1. The upgrade steps can be found in the Release notes here for both standalone systems and clustered systems.
2. You can directly upgrade to X14.x from version X8.11.4 and higher, no intermediate version is required. Upgrades from any version before X8.11.4, require an intermediate upgrade to X8.11.4.
3. No release key is required for an upgrade of Expressway to X12.5.4 or higher. It is required for Cisco VCS systems though.
4. In case of a cluster, start the upgrade on the “Primary” server in the cluster first. Once the "Primary" is upgraded, you can upgrade the "Subordinate(s)" nodes one at a time. This avoids the risk of configuration data loss and to maintain service continuity.
   

   Note: You can find the "Primary" on System > Clustering menu. The “Configuration primary” number points to the “Primary peer” in the list of peers in the same page.

   
5. You can upgrade Expressway-E and Expressway-C "Primary" at the same time. Or you can upgrade Expressway-E cluster first ("Primary" then "Subordinate(s)"), then upgrade Expressway-C cluster ("Primary" then "Subordinate(s)"), but ensure that at the end of the upgrade window, all servers (Expressway-C and Expressway-E) are on the same version.
6. If you use Cisco Meeting Server (CMS) WebRTC Proxy over Expressway feature and the Expressway-E as TURN server, you must ensure that you run a CMS version 2.9.3 or 3.0 and later for WebRTC to continue functionality after the upgrade. Previous releases of CMS do not work due to TURN service incompatibility related to Cisco bug ID CSCvv01243.
7. If you have push notifications enabled for Mobile and Remote Access (MRA), you must ensure that you run at minimum Cisco Unified Communications Manager (CUCM)/ Instant Messaging and Presence (IMP) version 11.5.1.18900-97 or 12.5.1.13900-152 or 14.0.1.10000-20 or later before the upgrade of your Expressways.
   

   Note: You can check if you have push notifications enabled from CUCM > Advanced features > Cisco Cloud Onboarding, and see if Enable push notifications is checked (enabled).

Pre-upgrade Actions

1. In order for MRA functionality to continue after the upgrade, you need to upload the root and intermediate certificates that signed Expessway-C certifiicate to the CUCM publisher as “tomcat-trust” and as “callmanager-trust”.

After the upload of the certificates, restart the “Cisco Tomcat” service, “Cisco callmanager” service and “Cisco TFTP” service on all relevant CUCM nodes (Cisco HAProxy service is restarted automatically with the Tomcat service restart).
This is needed due to changes made in Cisco bug ID CSCvz20720 . This is needed even if you use non-security phone profiles and " TLS verify mode" is disabled for the CUCM cluster added on Expressway-C.

For more information on the exact steps needed to achieve that, please refer to the document Upload the Root and Intermediate Certificates of Expressway-Core onto CUCM. 

Note: You can only restart the “Cisco Tomcat” service from the command line with the command utils service restart Cisco Tomcat .

2. From X14.2 onwards, even if TLS verify is set to Off on Unified Communications servers (CUCM, IM&P, CUC and CMS), the CA certificates (both Root and any intermediate CAs) for those servers must be added to Expressway-C trust store. Failure to do so, can cause MRA login problems after an upgrade to X14.2 or higher.

In addition to that, the FQDN of the Unified Communication servers Expressway-C connects to, need to be in the SAN list in the certificate of those servers.

This change was added part of a security enhancements to Expressway, tracked in Cisco bug ID CSCwc69661. Further information on this can be found in X14.2 release notes.

Further more, you can also refer to the document Troubleshoot Expressway Traffic Server Certificate Verification for MRA Services Introduced by CSCwc69661

3. From X14.2 onwards, Smart Licensing is the only available license mode with Expressway, traditonal PAK (option key) based license model has been removed.

Normally, if you only use the Expressways for MRA, no license is needed and this change does not impact your system. However, if you use B2B calls or you register endpoints to Expressway (or any other features that require a license), then you must make sure that Expressway-C and Expressway-C can access Cisco Smart Software Manager on cloud directly or via a proxy or connect to Cisco Smart Software Manager On-Prem.

Smart licensing is enabled by default after an upgrade to X14.2, but you have to make sure that the connection to the CSSM (Cloud or on-prem) is successful.

Further information on this can be found in the X14.2 release notes.

4. From X14.2 onwards, Expressway is limited to the 2500 crypto sessions limit (2500 sessions here is a sum of all MRA sessions + calls + endpoint registrations to Expressways), a single MRA session with one client could consume two crypto sessions or more, the same with dual regisration endpoints (H.323 and SIP), each of those endpoints would consume 2 crypto sessions.

Normally this does not impact small sized Expressways which are only used for MRA, however this does impact a Medium or large sized Expressway used for MRA.

Pre X14.2, a large sized Expressway could normally handle up to 3500 MRA sessions, but with X14.2 it is limited to 2500.

This means that the Expressway capacity could be halved. For example if you have 2500 Jabber users (with phone and IM&P services), after an upgrade to X14.2, this is seen by Expressway as 5000 encrypted signal sessions, and sessions over the 2500 mark are rejected, which affects MRA calls and registrations.

This limit cannot be removed in X14.2.

Further information on this can be found in the X14.2 release notes.

5. If you have an Expressway cluster, ensure you have no cluster alarms (From Status > Alarms).

Note: If you only have the alarm number “40049” about “Cluster TLS permissive -  Cluster TLS verification mode permits invalid certificates”, you can ignore this alarm and continue with the upgrade, but any other cluster alarms need to be dealt with before the upgrade.

6. If you have an Expressway cluster, connect to the Expressway server that you are about to upgrade through SSH and use the “root” user and run the command:
cd / && ./sbin/verify-syskey

Note: This command must not provide any output. If you get an “error” as a result from this command, open a TAC case to fix the errors before you proceed with the upgrade.

7. Finally, take a backup before the upgrade (From Maintenance > Backup and Restore). Do this on each server.

Upgrade Directions

1. Download the upgrade file (name ends with ".tar.gz") from Expressway software downloads (For example download  “s42700x14_0_6.tar.gz” for X14.0.6).
2. Upload the upgrade file (For example “s42700x14_0_6.tar.gz”) to the Expressway (From Maintenance > Upgrade, then click Browse to find the upgrade file on your PC, and finally click Upgrade)

   Note: The upgrade file is uploaded to Expressway when you click Upgrade. After the upload is done, press Continue to proceed with the upgrade. The server installs the software and it asks you at the end to Reboot in order to switch to the new software.

Post-upgrade actions

After an Expressway upgrade, you must Refresh the Unified Communications nodes from the primary Expressway-C server:

- Navigate to Configuration > Unified communication > Unified CM servers.  Select all CUCM clusters and select Refresh.

- Navigate to Configuration > Unified communication > IM and presence service nodes. Select all IM&P clusters and select Refresh.

- Navigate to Configuration > Unified communication > Unity Connection Servers. Select all CUC clusters and select Refresh.

FAQ

Licenses

1. Do I need a release key in order to upgrade ?

A. A release key is not required to upgrade an Expressway to version X12.5.4 or higher (release keys are still used for Cisco VCS systems).

2. Do I need to migrate my licenses?

A. Licenses that are installed on the Expressway prior to the upgrade are migrated automatically to the new version.

3. What licenses do I need to upgrade?

A. If you plan to upgrade from X8.11.4 or higher to a later version on the same server, no additional licenses are required, your current licenses are migrated automatically to the new version (VCS systems still require a release key).

These licenses are not required from version X12.5.4 onwards:

LIC-SW-EXP-K9 Release Key (From X12.5.4, this is provided by default on an upgrade of Expressway Systems. It is still required for VCS systems.)

LIC-EXP-TURN TURN relay licenses (Provided by default)

LIC-EXP-GW Interworking gateway (Provided by default)

LIC-EXP-AN Advanced networking (Provided by default)

These licenses are not required from version X12.6 onwards:

LIC-EXP-SERIES Expressway Series (You can now change this from the UI through the Service setup Wizard from Status > Overview)

LIC-EXP-E Traversal server license (You can now change this from the UI through the Service setup Wizard from Status > Overview)

4. Do I need to enable Smart licensing?

A. Smart licensing is mandatory from X14.2 onwards. Any version lower than X14.2 can still use option key license model.

Smart licensing is enabled by default after an upgrade to X14.2, but you have to make sure that the connection to the CSSM (Cloud or on-prem) is successfull.

Compatibility

1. Can I upgrade directly to X14.x ?

A. You can directly upgrade to X14.x (or to X12.x) Expressway release from version X8.11.4 and higher. Any version lower than X8.11.4 requires two-stage upgrade. Further information is available in the release notes.

2. Which Cisco Unified Communications Manager and IM&Presence versions are compatible with Expressway ?

A. If you use Push Notification for Jabber over MRA, the minimum versions are 11.5.1.18900-97, 12.5.1.13900-152 or 14.0.1.10000-20.

You can check if Push Notifications are enabled under CUCM admin page Advanced features > Cisco Cloud Onboarding. Verify if Enable push notifications is checked (enabled).

3. Which CMS version is compatible with Expressway 12.X and 14.X ?

A. If you use CMS WebRTC Proxy over Expressway, ensure that you run CMS version 2.9.3, 3.0 or later.
Previous releases do not work due to TURN service incompatibility related to Cisco bug ID CSCvv01243

Post upgrade

1. Are there any additional tasks that I need to perform after the upgrade?

A. The Unified Communications nodes must be refreshed from Expressway-C primary peer:

- Navigate to Configuration > Unified communication > Unified CM servers.  Select all CUCM clusters and select Refresh .

- Navigate to Configuration > Unified communication > IM and presence service nodes. Select all IM&P clusters and select Refresh .

- Navigate to Configuration > Unified communication > Unity Connection Servers. Select all CUC clusters and select Refresh .

2. How can I validate that the upgrade is successful?

A. There are couple things that can be checked:

- Check if the the cluster is stable (from System > Clustering) and confirm that there are no cluster alarms Status > Alarms.

- Ensure that the zone with the type “Unified communication traversal” shows as “Active” for “SIP status” on Expressway-C and on Expressway-E. It is normal to see the Auto-created CE (tcp/tls/OAuth) Zones (from Configuration > Zones) show as “Address resolvable” instead of “Active”. 

- Perform live tests by MRA log in, test calls, and so on.

3. I see a new alarm about "Unsupported Hardware" or "Unsuitable hardware warning" on my Virtual Expressway server after a successful upgrade?

A. Expressway version X14.x now verifies the Virtual Machine (VM) CPU clock speed and makes sure that it matches the clock speed required for the VM of the same size as it is mentioned in the virtualization guide for Expressway. The exact alarms shows up as: "Unsuitable hardware warning - Your current hardware does not meet supported VM configuration requirements for this version of Expressway.".

If you see this alarm, verify that the VM resources match the resources mentioned in the virtualization guide for Expressway. If they are lower than what is mentioned in the guide, then you need to rebuild the server to meet the minimum requirements for the size you have selected and then restore a backup.

Important note for X14.0.7

If you have a Medium deployment (seen from Status > System > Information) AND your VM has a clockspeed higher than 3.19GHz AND your VCS/Expressway version is exactly X14.0.7, then you can ignore the alarm. This alarm gets incorrectly triggered due to Cisco bug ID CSCwc09399.

Mobile Remote Access (MRA)

1. Does the upgrade require configuration changes on Cisco Unified Communications Manager (CUCM) ?

A. If you use MRA, due to security enhancement Cisco bug ID CSCvz20720, the root and intermediate certificates of the Certificate Authorities that signed Expressway-C certificate must be uploaded as “tomcat-trust” and “callmanager-trust” to the CUCM publisher server (it replicates them to the subscribers). This is needed even if you use non-security phone profiles and " TLS verify mode" is disabled for the CUCM cluster added on Expressway-C. Restart the “Cisco Tomcat”, “Cisco CallManager” and “Cisco TFTP” services on each server so the changes take effect.
The “Cisco Tomcat” service can be restarted only from command line with the command “utils service restart Cisco Tomcat".

For more information on the exact steps needed to achieve that, please refer to the document Upload the Root and Intermediate Certificates of Expressway-Core onto CUCM. 

2. Do I need to change my Expressway-C certificate to upgrade?

A. No need to change the Expressway-C certificate if it is still valid. However, the root and intermediate certificates of the Certificate Authorities that signed Expressway-C certificate must be uploaded as “tomcat-trust” and “callmanager-trust” to the CUCM publisher server. See point 1 in Pre-upgrade actions section for more information.

Pre-upgrade

1. What must I check prior the upgrade ?

A. If you have a clustered Expressway system, verify that you do not have cluster alarms from Status > Alarms.
Note: Alarm "40049” with message “Cluster TLS permissive - Cluster TLS verification mode permits invalid certificates” does not impact the upgrade process. All other occurrences must be resolved prior to the upgrade.
Also, run the comand cd / && ./sbin/verify-syskey from command line through root user. This command must not give any output. In case it does, we recommend you to open up a TAC case to get this investigated and corrected.

Upgrade process

1. What is the upgrade sequence in a clustered system?

A. Start the upgrade from the "Configuration primary" peer in the cluster. You can see which one it is under menu System > Clustering. The “Configuration primary” number displays which one it is among the peers.
After the upgrade of the primary peer is complete, you can continue with the subordinate peers (one at a time).

2. Can I upgrade Expressway-C and Expressway-E at the same time?

A. Yes, you can do that however it is recommended to upgrade the Expressway-E server(s) first and then the Expressway-C servers so that the traversal zone is set up correctly first on the E server. And if you have a cluster, ensure that you start the upgrade with the "Primary" servers. Once the upgrade on the "Primary" is done, you can upgrade the "Subordinate" peers.

3. Where can I download the Expressway upgrade image?

A. You can find all Expressway upgrade images in the link here. Download the file with extension "tar.gz" for the version you wish to upgrade to:

https://software.cisco.com/download/home/286255326/type/280886992/

4. How do I start the upgrade?

A. Navigate to Maintenance > Upgrade > Browse, select the upgrade file and click "Upgrade". First the file is transferred. After that, you get a "Continue" button to start the actual upgrade process.

5. How long does the upgrade process take ?

A. Most of the time the upgrade process takes up to 10 minutes after the upgrade file was transferred to the system and you have selected "Continue".
However, it is highly recommended to schedule a maintenance window from 4 to 48 hours to accomodate for post upgrade tests.

6. What access is required to perform the upgrade?

A. The upgrade is performed over the Web interface however in case you run into any issues after the upgrade, console access could be required.
It is good to check prior the upgrade that VMware or CIMC console access is available.

Backup and restore

1. Do I need to take a backup before the upgrade?

A. A backup is recommended before the upgrade of Expressway. In case of a cluster, take a backup from all servers.
You can do it on each server from Maintenance > Backup and Restore .

2. Can I take a snapshot of the Expressway before the upgrade?

A. VMware Snapshots are not supported on Expressway.

3. Can I Rollback/Revert to the previous system I had before the upgrade?

A. Expressway keeps two sets of partitions after an upgrade. One is with the upgraded version and one is with the previous version. 
You can switch between those partitions with the command "selectsw <1 or 2>" from root user shell.
You can verify the current active partition with command "selectsw".
For example, if you get "1" after you ran the "selectsw" command, then the active version is "1" and the inactive version is "2". To switch to the inactive partition, execute a command "selectsw 2". A  reboot is required to boot from a newly selected partition system.

Physical Appliance servers

1. Can I upgrade to this version on my Physical Appliance server?

A. For all Physical Appliance servers (CE500, CE1000, CE1100, CE1200), please refer to "Table 2" in "Supported platforms" section of the release notes for your destination version to verify if you can upgrade to the destination version.

2. I have a CE1100, can I upgrade it to X14.0.x and X14.2.x?

A. For Physical Applicance server CE1100, you can upgrade to X14.0.x and X14.2.x to mitigate vulnerabilites and you can ignore the "Unsupported Hardware" alarm. This is mentioned in the release notes of X14.0.6. Cisco has extended the End of Vulnerability/Security Support from November 14, 2021 (as per the original End-of-Life announcement) to November 30, 2023, in line with the last date of support, for those customers with a valid service contract. Note that this only applies for vulnerability fixes and not for new features.

Virtual servers and ESXi

1. Which ESXi version is supported with this Expressway version?

A. You can find the ESXi support information in the installation guide (under System Requirements > ESXi Requirements) for the destination version of your Expressways.