To generate the Artifactory API Key, follow the steps below:

Note	It is mandatory for the CCO ID used to generate API keys to have necessary software upgrade entitlements. The CCO ID used by the partner or customer should have a valid SWSS (service contract) or Flex subscription in order to have the necessary entitlement.

• Login to https://devhub-download.cisco.com/console/ using your CCO Username and Password.

• Navigate to 'Manage Download Key page.

• Click Generate Key option to Generate the API key. Option to View and Revoke Key is available in Manage Download Key page.

• Click on the Copy option to copy the API key to the clipboard.

  Note	You must log into https://devhub-download.cisco.com/console once every six months to extend the validity of the API key.

Note

Cisco recommends not to use the same Artifactory API key generated by a single CCO ID across multiple deployments. For multiple deployments such as test, pre-production, production, and so on, generate the Artifactory API key for each deployment using different CCO IDs. Artifactory API key generated by a single CCO ID can be used in both publisher and subscriber of Cloud Connect in a single deployment.

Cisco hosts all the software artifacts in a cloud-based artifactory. The Cloud Connect server uses this artifactory to download and notify new updates.

Configure the Cloud Connect server with Cisco-hosted software Artifactory URL, Repository Name, and API Key. Run the command utils image-repository set. Refer to the Set Command table.

To view the configured Artifactory URL, Repository Name, and API Key in the Cloud Connect server, run the command utils image-repository show. Refer to the Show Command table.

Note	You can run the utils image-repository set command only in the publisher node of the Cloud Connect server. The replication of image repository configuration occurs automatically from the publisher node to the subscriber node when you run this command with successful results on the publisher node.

Note	Before running the command utils image-repository set on the CLI, access the link https://software.cisco.com/download/eula and accept the End User License Agreement (EULA)

Note	Use the command utils image-repository set to change export-restricted or unrestricted software in the deployment. Use the CLI utils initiate software-download to enforce the cleanup and download the restricted vs unrestricted software.

Note

On the successful configuration of artifactory details, artifacts are downloaded locally to the Cloud Connect server at the scheduled or default time. Orchestration operations such as patch install, rollback, or upgrade can be performed only after the artifacts are downloaded. If you need to download the artifacts immediately after the configuration, use the utils initiate software-download CLI. Usage of orchestration-related CLI is blocked during download, and this duration depends on the number of artifacts to be downloaded.

Note

Before you configure the bandwidth using the utils set software-download bandwidth command, make sure the software is downloaded locally for the first time after the artifactory is successfully configured using the utils image-repository set command. To download the artifacts immediately after the configuration, use the utils initiate software-download command.

The onboarding process helps to establish a password-less connection between the Cloud Connect node and the VOS nodes.

Prerequisites:

• Ensure that the Cloud Connect server and target nodes maintain the minimum software versions that are required as outlined in System Requirements.

• If you are using self-signed certificates, import the self-signed Tomcat certificate of the Cloud Connect server into the VOS nodes which you have to onboard. Ensure to import both Cloud Connect publisher and subscriber node certificates on all VOS publisher and subscriber nodes. For details, see Self-Signed Certificate.

To onboard Finesse, CUIC, VVB, IDS, LD to a Cloud Connect server, run the utils system onboard initiate command from the publisher node of the respective VOS cluster that you wish to onboard. The publisher node of the Cloud Connect server must be up and running when onboarding is initiated from VOS node. When the onboarding is initiated from VOS node, FQDN of the Cloud Connect server must be used.

Note	If the system (cluster) onboards to the Cloud Connect server with partial error, check the reason for the error and correct it. Then, run the utils system onboard update command instead of running the utils system onboard initiate command.

Note	Onboarding is allowed only when all the publisher and subscriber nodes in the Cloud Connect server are reachable.

Note	If the Cloud Connect server is corrupted and redeployed by doing fresh install, the administrator has to run utils system onboard remove from the VOS node and then run utils system onboard initiate to onboard the VOS nodes again.

To validate the onboarding of VOS and Windows nodes, and to check whether the Orchestration feature is ready to be used, run the utils deployment test-connection command.

If an email notification is configured, the Cloud Connect server checks the Cisco-hosted artifact repository periodically at scheduled times and sends email notifications along with the release notes when new software updates are available. Administrators can decide when to apply a patch or perform an upgrade. Email notifications are not triggered if no new software updates are available.

Note	The SMTP server referred to in this section is the mail server that is used within the customer organization for their internal email communication.

Perform the following procedures in the same sequence as given here.

Microsoft Windows update configuration needs to be done on the target Windows node. Microsoft Windows updates can be downloaded in one of following ways on the target Windows node:

• by directly connecting to the Microsoft server;

• from the Windows update server configured. To deploy or configure Windows server update services, refer to https://docs.microsoft.com/en-us/windows-server/administration/windows-server-update-services/deploy/deploy-windows-server-update-services.

To check the currently installed software version and patches on a node or group of nodes or all nodes in either Windows or VOS systems, run the utils deployment show status command.

To get a list of available patches for a specific node or group of nodes in the inventory, run the utils patch-manager list command.

To install patch to a specific node or group of nodes, run the utils patch-manager install command.

Note	To start Unified ICM services, post the successful completion of patch install with reboot on Unified ICM nodes. See Start ICM Services.

Note	You can check the status of the patch install which is currently in-progress. For more information, see Check Status.

Note	Maintenance mode for IDS co-resident in 2000 Agents Deployment model is not supported

To roll back a previously installed patch on a specific node or a group of nodes, run the utils patch-manager rollback command.

Note	To start Unified ICM services, post the successful completion of patch rollback with reboot on Unified ICM nodes. See Start ICM Services

Note	You can check the status of patch rollback which is currently in-progress. For more information, see Check Status.

To install Windows updates to a node or group of nodes or all Windows nodes, run the utils patch-manager ms-patches install command.

Note

Before running this command, refer to the recommended guidelines in the Microsoft Security Updates section of the Security Guide for Cisco Unified Contact Center Enterprise at:https://www.cisco.com/c/en/us/support/customer-collaboration/unified-contact-center-enterprise/products-installation-and-configuration-guides-list.html.

Microsoft Windows updates are NOT hosted on Cisco-hosted Software Artifactory. You must configure the target Windows node to fetch the Microsoft Windows updates, either by directly connecting to the Microsoft Server via Internet or from the Windows Update Server. For more details, refer to the Configure Windows Server for Updates section. The utils patch-manager ms-patches install command will not list the available Windows updates for the administrator to choose for the target node. Instead, it will check the available updates for the below listed Windows update categories and install all the available updates:

• Application

• Connectors

• DefinitionUpdates

• DeveloperKits

• FeaturePacks

• Guidance

• ServicePacks

• Tools

• UpdateRollups

• CriticalUpdates

• SecurityUpdates

• Updates

The administrator can control the installation of Windows updates using Windows Update Server, instead of directly connecting to the Microsoft Server via Internet. Ansible log, generated during the running of utils patch-manager ms-patches install CLI, captures the details of the Windows updates, along with the Knowledge Base (KB) number of the updates that were installed on the target node. Refer to the Serviceability section for the command to retrieve the Ansible log.

Note	The utils patch-manager ms-patches install operation may require a considerable amount of time to complete, depending on the number of Windows updates available on the target node.

To roll back Windows update from a specific node or group of nodes or all Windows nodes, run the utils patch-manager ms-patches rollback command.

Note

Before running this command, refer to the recommended guidelines in the Microsoft Security Updates section of the Security Guide for Cisco Unified Contact Center Enterprise at: https://www.cisco.com/c/en/us/support/customer-collaboration/unified-contact-center-enterprise/products-installation-and-configuration-guides-list.html Listing of Windows updates available for rollback is not supported.

You can enable or disable compatibility enforcement. When the compatibility enforcement is enabled, it ensures that the upgrade, rollback, or switch forward is as per the compatibility matrix published by Cisco for reference design-based deployment. To enable or disable compatibility enforcement, run the utils deployment compatibility-check command.

Note	By default, the compatibility enforcement is enabled. When the compatibility enforcement is disabled, the Orchestration framework does not enforce upgrade, rollback, or switch forward as per the compatibility matrix published by Cisco.

Note	You can run this command only from the publisher node of the Cloud Connect server. The compatibility configuration replicates automatically from the publisher node to the subscriber node when the utils deployment compatibility-check command is run with successful results on the publisher node.

Initiating maintenance mode allows the components to failover gracefully or shutdown the services gracefully (depending on the selected components) without interrupting the active traffic or causing outage to new traffic. This ensures that the system can be taken down for maintenance activity such as installing new software updates, restarting services etc. The maintenance mode is supported for PG, CVP server, IdS, Finesse, Router, and Rogger.

Note	Ensure that not all CVP servers are put into maintenance mode at same time, so that incoming call traffic can be distributed.

Pre-requisites for Maintenance Mode Support

• Maintenance mode is supported for PG, CVP server, IdS, and Finesse if the target nodes are on 12.6(1) and above.

• Maintenance Mode is supported for Router and Rogger if the target nodes are on version 15.0(1) and above, or if the target nodes are installed with 12.6(2) Router Component ES that is ES68 or higher.

Note	To enable Maintenance Mode support for Router and Rogger in Orchestration, install Cloud Connect mandatory 15.0(1) ES202508 or above.

To initiate maintenance mode for a specific node in the inventory, run the utils system maintenance initiate command.

Note	If either the Publisher or Subscriber or the active/inactive node is already in maintenance mode in any of the components, the other server cannot be initiated for maintenance.

Note	You can check the status of system maintenance initiate which is currently in-progress. For more information, see Check Status.

Note	Maintenance mode for IDS co-resident in 2000 Agents Deployment model is not supported

To get a list of available upgrade options for VOS and Windows nodes individually or for group of nodes or for all nodes in the inventory, run the utils upgrade-manager list command.

To perform software version upgrades on VOS or Windows nodes , run the utils upgrade-manager upgrade command from the Cloud Connect server. It’s recommended to run this command during a maintenance window as the procedure involves a system restart that causes a service outage.

To upgrade the selected VOS or Windows component, a compatibility check is performed in the background based on the configured deployment type. This ensures that all the associated components are onboarded. If the components are onboarded, the upgrade procedure begins either the required dependent components are in the same target upgrade version or backward compatible version. However, if the components are not onboarded, you have to onboard them first or if the versions are not compatible, upgrade them to the required version.

For example, if you select to upgrade the Rogger nodes to the 12.6(2) version, the intercomponent compatibility check is run for the Rogger dependent components such as Finesse, Cisco Unified Customer Voice Portal, VVB, and CUIC. These must already be in the 12.6(2) version and PG must be in the backward compatible version, that is, 12.5(2).

Note

The subcomponents sequence dependencies aren’t validated as part of the upgrade compatibility. Refer to the upgrade guides of the respective components for the correct sequence. For example, in Cisco Unified Customer Voice Portal, we have subcomponents such as Operations Console, Unified Cisco Unified Customer Voice Portal Reporting Server, and Unified Cisco Unified Customer Voice Portal Server. These must be upgraded in the required sequence. Orchestration ensures 15.0(1) ICM and CVP are deployed on the supported Windows Operating System. Before upgrading ICM or CVP to 15.0(1), ensure that the Windows Operating System is upgraded to a supported version. Else, Orchestration will not allow the upgrade to 15.0(1). Refer to Unified CCE 15.0(1) Compatibility Matrix for the details on supported Windows Operating System for ICM and CVP. CVP Reporting Server upgrade to 15.0(1) is not supported through Orchestration. For details on manually upgrading CVP Reporting Server to 15.0(1), refer to the Installation and Upgrade Guide for Cisco Unified Customer Voice Portal.

For the VOS node/cluster, switch forward is optional at the end of the upgrade. If administrators opt for switch forward, the target node is restarted and the active/inactive partition is switched. If they decide not to switch forward, the upgraded version remains in the inactive partition of the target node. Switch forward for these nodes can be performed later. For details, see Perform Switch Forward on Specific VOS Node or Group of Nodes.

For the VOS cluster, the upgrade or the switch forward procedure is performed first on the publisher and then on the subscriber nodes. If a switch forward is performed immediately after an upgrade, the overall procedure takes a significant amount of time; hence plan the maintenance window accordingly.

Prerequisites

• Before upgrading the Unified CCE and its associated components, refer to the installation and upgrade guide for each component. This should be part of your deployment upgrade planning to address component-specific prerequisites, such as modifying resources based on OVA requirements, following Windows updates and antivirus recommendations, performing in-place Windows/SQL upgrades, identifying unsupported features, meeting backup requirements, and ensuring that components are deployed with valid, unexpired certificates.

• The following prerequisites must be completed on Unified ICM and Unified CVP components before initiating the upgrade to a major release (for example, 15.0(1)) remotely through Orchestration. If the prerequisites aren’t completed, the Unified ICM and Unified CVP ISO upgrade fails. After the prerequisites are completed, you must manually restart the nodes before initiating the upgrade from Orchestration.

  Unified ICM and Unified CVP components

  • Username configured in the Orchestration inventory must not conflict with any local or domain user. For example, a domain administrator user may have conflicting local administrator user on the node.

  • Disable the Windows Defender Real-Time Protection under Local Group Policy Editor → Computer Configuration → Administrative Templates → Windows Components → Windows Defender Antivirus → Real-time Protection in the Local Group Policy Editor.

  • Windows should be up-to-date with the latest updates.

  • Disable the Windows update service.

  • To enable auto-logon, update the following registry keys. Use the username configured in the Cloud Connect inventory for this Virtual Machine when updating the below registry key value:

    Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon

    Update the following keys:

    1. AutoAdminLogon - Set to value 1

    2. DefaultDomainName - Provide Domain Name, if applicable

    3. DefaultUserName - Provide the username which is configured in the Cloud Connect Inventory

    4. DefaultPassword - Provide the password

  Unified ICM components

  • All the User Account Control (UAC) group policies must be configured with the default value. However, you must update the following policies with the given value under Local Security Policy → Local Policies → Security Options in the Local Security Policy:

    • Behavior of the elevation prompt for administrators in Admin Approval Mode—Elevate without prompting.

    • Switch to the secure desktop when prompting for elevation—Disabled.

  • Security setting shouldn’t be marked as Not Applicable for any policy under Local Security Policy → Local Policies → Security Options in the Local Security Policy.

Note

For faster upgrades, the Cloud Connect server downloads locally all the new software updates from the Cisco hosted repository at a predefined time. To start the Unified ICM services, post the successful completion of upgrade with reboot on Unified ICM nodes. See Start ICM Services. The option to upgrade “All nodes” in the deployment (VOS and Windows nodes together) to CCE 15.0(1) via Orchestration is not supported. The administrator can upgrade the individual components to 15.0(1) by selecting the respective VOS or Windows nodes. All nodes option is currently supported only for upgrading to CCE 12.6(x)

Note	You can check the status of the upgrade that is currently in-progress. For more information, see Check Status.

Administrators can perform switch forward on target VOS nodes independently. When the active partition is on lower version and the inactive partition is on higher version, run the utils upgrade-manager switch-forward command to perform a switch forward. It is recommended to run this command during a maintenance window as the procedure involves system restart that will cause service outage.

Note	You can check the status of switch forward which is currently in-progress. For more information, see Check Status.

To roll back an upgrade on VOS or Windows nodes, run the utils upgrade-manager rollback command from the Cloud Connect server. It’s recommended to run this command during a maintenance window as the procedure involves the system restart that causes a service outage.

For the selected VOS or Windows component for rollback, a compatibility check is performed in the background to ensure that all the associated components are onboarded and the versions are compatible. If the components are onboarded and the versions are compatible with each other, the rollback procedure begins. However, if the components aren't onboarded, you have to onboard them first or if the versions aren't compatible, roll them back to the required version.

For VOS nodes/cluster, the rollback (switch backward) must be initiated from an active higher version to an inactive lower version of the node. Also, the publisher node of the managed cluster must be rolled back before the subscriber node of the cluster.

Note	Orchestration does not allow rollback from major release versions, for example, 15.0(1), for Unified ICM and Unified CVP components. However, CVP components can be manually rolled back from major release versions on the target node(s).

Note	To start Unified ICM services, post the successful completion of rollback upgrade with reboot on Unified ICM nodes. See Start ICM Services.

Note	You can check the status of the rollback which is currently in-progress. For more information, see Check Status.

To check the current status of patch manager install, patch manager rollback, upgrade manager upgrade, upgrade manager rollback, switch-forward, or system maintenance initiate, run the utils deployment show in-progress command. You can run this command if connectivity to CLI is lost after initiating any of above procedures.

To check the last known orchestration operation status (last completed state or last known state when the operation is in progress or when the remote node is not reachable) on the remote node, run the utils deployment show progress-HA command. This command is applicable for patch manager install, patch manager rollback, upgrade manager upgrade, upgrade manager rollback, switch-forward, and system maintenance initiate.

This command can be used only in Cloud Connect High Availability setup.

Note	Last known orchestration operation status will not be synchronized to remote node, in case of communication loss to remote node after initiating the orchestration operation and operation being completed before re-establishing the communication.

To start Unified ICM services from Cloud Connect server, run the utils system icm-services start command.

To update VOS based nodes that have been onboarded, run the utils system onboard update command from the publisher node in the VOS node/cluster that you want to update.

To remove any existing VOS-based node or cluster, run the utils system onboard remove command from the publisher node in the VOS node/cluster that you want to remove.

The update procedure is similar to the onboarding procedure described in Onboard Windows nodes to orchestration control node.

Note	If SSH connection is already established, skip Step 1 in the above procedure.

The procedure to validate updated nodes that have been onboarded is the same as described in Validate Onboarded Nodes for Orchestration.

You can check your email configuration details by running the respective commands as described below:

• Get the IP address or hostname of the SMTP server by running the show smtp-host command.

  Command	show smtp-host
  Description	This command is used to get the IP address or hostname of the SMTP server.
  Expected Inputs	NA
  Expected Outcome	Shows the configured IP address or host name of the SMTP server. If the smtp host details are not configured, a message is displayed indicating it.

• Get the email address from which the emails are triggered by running the show smtp-from-email command.

  Command	show smtp-from-email
  Description	This command is used to get the email address from which the emails are triggered. This email address is not monitored and therefore not used for replying to any emails.
  Expected Inputs	NA
  Expected Outcome	Shows the email address from which the emails are triggered. If the smtp from-email is not configured, a message is displayed indicating it.

• See if SMTP authentication is enabled or not by running the show smtp-use-auth command.

  Command	show smtp-use-auth
  Description	This command is used to know if SMTP authentication is enabled or not.
  Expected Inputs	NA
  Expected Outcome	SMTP authentication : <enable/disable>

• Get the username for SMTP server connection by running the show smtp-user command.

  Command	show smtp-user
  Description	This command is used to show the user name to be used for SMTP server connection.
  Expected Inputs	NA
  Expected Outcome	Shows the SMTP username. If the smtp user details are not configured, a message is displayed indicating it.

• See if the SMTP password is set or not by running the show smtp-pswd command.

  Command	show smtp-pswd
  Description	This command is used to know if the SMTP password is set or not. To reset the password, run the set smtp-pswd command.
  Expected Inputs	NA
  Expected Outcome	Shows whether the SMTP password is set or not. If the smtp password details are not configured, a message is displayed indicating it.

• See the email addresses subscribed for notification by running the utils smtp show subscriptions command.

  Command	utils smtp show subscriptions
  Description	This command is used to get a list of all the email addresses subscribed for email notification.
  Expected Inputs	NA
  Expected Outcome	Shows the email addresses that are subscribed for email notification. If there is no email address subscribed, a message is displayed indicating it.

To remove the configuration for email notifications, run the utils smtp remove-config command.

To unsubscribe from email notifications, run the utils smtp unsubscribe command.

utils smtp unsubscribe <emailaddress1,emailaddress2,.....emailaddressesN>

To export inventory to an SFTP server, run the utils system inventory export command.

To import inventory to Cloud Connect server, run the utils system inventory import command.

Note	For information on adding deployment type and deployment name in the inventory file, see Add Deployment Type and Deployment Name.

Available patches for nodes in the deployment can be obtained in either of the following ways:

• Email Notification

• Using the utils patch-manager list command.

Current patch levels can be exported in text file format using the utils patch-manager export status command.

Audit Logs

Audit trial for administrative operation that is initiated from Orchestration CLI on Cloud Connect is captured in Orchestration Audit logs. Audit trial captures the user, action and date/time details of the CLI operation.

• file get activelog orchestration-audit/audit.log*

CLI Logs

Run the following command on the Cloud Connect node to retrieve CLI logs:

• file get activelog platform/log/cli*.log

Ansible Logs

Run the following commands on the Cloud Connect node to retrieve ansible-related logs:

• Current transaction logs: file get activelog ansible/ansible.log

• Historical logs: file get activelog ansible/ansible_history.log

Operation Status HA Synchronization Logs

Run the following command on the Cloud Connect node to retrieve synchronization-related logs:

• file get activelog ansible/sync_ansible_log_to_remote_cc.log

Email Notification-related Logs

Run the following commands on the Cloud Connect node to retrieve email-related logs:

• Current transaction logs:file get activelog ansible/ansible_email_cron.log

Software Download Logs

Run the following commands on the Cloud Connect node to retrieve software download-related logs:

• Current transaction logs:file get activelog ansible/software_download_ansible.log

• Historical logs:file get activelog ansible/software_download_ansible_history.log

• Process logs:file get activelog ansible/software_download_process.log

Note	Software is downloaded separately on Cloud Connect publisher and subscriber.

Orchestration Logs in RTMT

You can also view the below-mentioned logs using the Real-Time Monitoring Tool (RTMT):

• Audit logs by selecting 'Orchestration Audit' as the Cloud Connect service

• Ansible logs by selecting 'Ansible Controller' as the Cloud Connect service

To download RTMT from Cloud Connect, access https://%3Cfqdnoripaddress%3E:8443/plugins/CiscoRTMTPlugin.zip.

For more information, refer to the Cisco Unified Real-Time Monitoring Tool Administration Guide at:https://www.cisco.com/c/en/us/support/unified-communications/unified-communications-manager-callmanager/products-maintenance-guides-list.html.

For logs on individual components, refer to the Serviceability Guide for Cisco Unified Contact Center Enterprise available at https://www.cisco.com/c/en/us/support/customer-collaboration/unified-contact-center-enterprise/products-installation-and-configuration-guides-list.html.

To enable and view open SSH logs, do the following:

• Make sure the sshd_config file %programdata%\ssh\sshd_config has the value as 'LogLevel DEBUG' and uncomment the line.

• Restart the service (select service name OpenSSH SSH Server).

• In the Windows Event Viewer, select option Show Analytic and Debug Logs from View on the top menu bar.

• Select Debug channel from OpenSSH folder.

• On the right hand side, under Actions from Debug channel, select Enable log.

To turn on file-based logging, do the following:

• In the sshd_config file %programdata%\ssh\sshd_config, set the value as "SyslogFacility LOCAL0" and uncomment the line.

• Restart the service (select service name OpenSSH SSH Server).

• The file based logs are collected at location %programdata%\ssh\logs.