Troubleshoot Issue with CCXToCUICEngineSyncFailed Alert

Introduction

This document describes how to troubleshoot the issue with CCXToCUICEngineSyncFailed RTMT Alerts due to the inactive users on Cisco Unified Contact Center Express (UCCX).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Unified Intelligence Center (CUIC)
• Cisco Unified Contact Center Express (UCCX)
• Cisco Unified Communications Manager (CUCM)

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Problem

Generally when an agent/supervisor leaves the organization or moves to a new role such that they are no longer a part of the Contact Center.

In such a scenario, you can delete the local user from CUCM or the AD (for LDAP integrations) with the assumption that the user gets deleted from CCX.

However, note that the user still exists on CUIC. When you attempt to delete the user from CUIC, it fails in case the user has ownership of one or more entities on CUIC such as dashboard, reports, custom reports, Valuelists, and so on.

Quick Check

Log in to Cisco Unified Intelligence Center > Configure > Users and search for the deleted users.

In such a scenario, an alert gets triggered as sync failed between CCX and CUIC.

RTR: User with userID:anie failed to be synced to CUIC. Please run the sync CLI command to fix the issue.

AppID: Cisco Unified CCX Engine

ClusterID: 

NodeID: uccx-sr 

TimeStamp: Wed Apr 27 08:59:08 IST 2022. The alarm is generated on Wed Apr 27 08:59:08 IST 2022

Syslog Alert:

At Wed Apr 27 08:59:35 IST 2022 on node uccx-sr; the following SyslogSeverityMatchFound events 
generated: SeverityMatch : Alert MatchedEvent : Apr 27 08:59:08 uccx-sr local7 1 : 53: 
uccx-sr.cisco.local: Apr 27 2022 03:29:08.290 UTC : %UC_REST_CLIENT-1-UCCX_TO_CUIC_SYNC_FAILED: 
%[RTR=User with userID:anie failed to be synced to CUIC. Please run sync CLI command to fix the issue.]
[AppID=Cisco Unified CCX Engine][ClusterID=][NodeID=uccx-sr]: Notification to CUIC failed AppID : 
Cisco Syslog Agent ClusterID : NodeID : uccx-sr TimeStamp : Wed Apr 27 08:59:08 IST 2022

Verify

From Logs

Enable traces.

Navigate to Cisco Unified CCX Serviceability > Trace > Configuration and select Cisco Unified CCX Engine and set the traces.

REST_CLIENT - Debug, Xdebug1

Save the changes after you enable the traces.

Collect Cisco Unified CCX Engine logs from RTMT

Snippet

1440456: Apr 27 08:59:08.286 IST %MIVR-REST_CLIENT-7-UNK: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.uccx.rest.client.CuicUserMgmtRestClient CuicUserMgmtRestClient.DELETE succeded, result=DELETE https://uccx-sr.cisco.local:8444/cuic/rest/cuicusers returned a response status of 200 OK
1440457: Apr 27 08:59:08.287 IST %MIVR-REST_CLIENT-7-UNK: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.uccx.rest.util.cuic.CuicUserMgmtRestUtils raising alert since deleteUsers API returned some failed users
1440458: Apr 27 08:59:08.287 IST %MIVR-REST_CLIENT-7-UNK: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.uccx.rest.util.cuic.CuicUserMgmtRestUtils Failed users list size1Failed users list : [anie]
1440459: Apr 27 08:59:08.287 IST %MIVR-REST_CLIENT-7-UNK: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.uccx.rest.util.cuic.CuicUserMgmtRestUtils parseFailedUsers ::anie
1440460: Apr 27 08:59:08.291 IST %MIVR-REST_CLIENT-1-UCCX_TO_CUIC_SYNC_FAILED: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.uccx.rest.util.cuic.CuicUserMgmtRestUtils Notification to CUIC failed: UCCX TO CUIC SYNC failed for these resources:::=User with userID:anie failed to be synced to CUIC. Please run sync CLI command to fix the issue.
1440461: Apr 27 08:59:08.291 IST %MIVR-REST_CLIENT-7-UNK: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.uccx.rest.util.cuic.CuicUserMgmtRestUtils Failed users list size1Failed users list : [anie]
1440462: Apr 27 08:59:08.291 IST %MIVR-REST_CLIENT-7-UNK: [MIVR_SS_RM_CONFIG-94-0-CONFIG] com.cisco.wf.admin.iaq.DBIAQConfiguration raiseRTMTAlertOnNotifcationFailure :: Sync API call failed for the userID ::anie

From CLI

1. On CLI, run the SQL query to verify if the user id exists in the CUIC database (DB).

Run SQL, select ID, name, description, from cuic_data:cuicuser where name like '%anie%'.

2. Log In to the UCCX Admin page. Navigate to User management > User View and notice that the USERID doesn't exist.

Check if you can delete the user from CUIC. The user cannot be deleted from CUIC with the error "Cannot delete the user. This user owns one or more entities."

Solution

As observed, if the user has access to one or more entities you cannot delete the user directly from CUIC with the error

"Cannot delete the user. This user owns one or more entities."

Identify the Entities Owned by the User

Identify the entities owned by the user with two methods.

Method 1. Use of SQLs against the DB directly.

Method 2. Use the Resource Ownership and Access report under the Reports > Stock > Intelligence Center Admin.

Method 1.

Identify the user ID  for which the alert was triggered, with the SQL query:

admin:run sql select id, name, description from cuic_data:cuicuser where name like '%anie%'
id name description
================================ ======== ===========
6902FD2E100001800000015A0A7FFCD4 CCX\anie

Note down the user id from the output and replace it with <userId> in the list of SQL queries in order to get the ownership of entities owned by the user.

run sql select Id, name from cuic_data:cuicCategory where Owner = '<userId>'

run sql select Id, name from cuic_data:cuicDashboard where Owner = '<userId>'

run sql select Id, name from cuic_data:cuicReportDefinition where Owner = '<userId>'

run sql select Id, name from cuic_data:cuicReport where Owner = '<userId>'

run sql select Id, name from cuic_data:cuicDatasource where Owner = '<userId>'

run sql select Id, name from cuic_data:cuicValueList where Owner = '<userId>'

run sql select Id, name from cuic_data:cuicCollection where Owner = '<userId>' and ValueListId is not null

run sql select Id, ScheduledreportName from cuic_data:cuicScheduledReports where UserId = '<userId>'

From the outputs of SQL queries, it is clear that the user has ownership of just 1 entity which is a Custom report by the name ‘Agent CSQ Custom Report’.

Method 2.

Use the ‘Resource Ownership and Access’ report under the Reports > Stock > Intelligence Center Admin > Resource Ownership and Access.

Note: Resource Ownership and Access Report - describes view access rights of users and the ownership status of users who use Unified Intelligence Center resources such as Dashboards, Reports, and other entities. The Resouce Ownership and Access report of a particular version is available as a part of the Admin Security template from the Software page; https://software.cisco.com/download/home/282163829/type/284697222/release/11.5%25281%2529

Import the report on to the CUIC. Run the report against the user to identify the entity owned by the user on the CUIC.

Delete the User from CUIC

With the two methods, you found the entity owned by the User. The next step is to delete the user, which can be achieved as follows

Option 1. Log In As to simulate the log in of the user anie, and navigate to the report owned by the user and delete it. However, you do not know if anyone else uses this report. But, if this report or entity is not important and currently not used, you can proceed to delete this.

Option 2. Use SQL on CLI to update the ownership and make super admin the owner of this entity.

Option 1.

Log In to CUIC with Super Admin and navigate to the Users section. Search for the user and select the Run As option from the Action Menu as shown in the image.

You can notice the page redirected to as super admin is logged in and simulate the log in for the user anie. This does not require any credentials, but CUIC simulates it. The direct login for this user does not work because this user does not exist.

Navigate to reports, search for the report name identified and delete it.

Once the report is deleted, go ahead and stop the Run As.

With CUIC logged back in with super admin, navigate to the Users section and you must be able to successfully delete the user anie.

Option 2.

This option can be helpful when you don't want to delete the Custom Report and continue to use it. Log in to CUIC and use the Run As option to simulate the missing user account. Then navigate to reports, search for reports, and change the permissions to include the group for admins or reporting users:

Log in to CLI and verify the owner ID of the user that was deleted:

Update the ownership of any reports created by this user to the super admin.

run sql update cuic_data:cuicReport set owner = '1111111111111111111111111111AAAA' where owner = '6902FD2E100001800000015A0A7FFCD4' as shown in the image.

Log back into the CUIC as super admin and notice the report.

Proceed to delete the user on CUIC now. Since there are no other entities owned by the user. Navigate to Configure > User > Actions Menu > Delete as shown in the image.

On the bottom right side of the page, a pop-up appears as the user deleted successfully.

This confirms that the user anie got deleted successfully from CUIC.