IPCC: Troubleshoot Mutex Lock Errors

Introduction

In a Cisco Unified Contact Center Express environment, a user cannot change configurations in the trigger information section of the Java Telephony Application Programming Interface (JTAPI) Triggers on the Cisco Customer Response Solution (CRS) Admin. In the attempt to change the application in the trigger information section of the JTAPI Triggers, this error message appears in the MADM log:

java.lang.InterruptedException: User (CRSuser) attempt to acquire mutex lock for the
purpose of (Cluster Mutex acquired by JTAPI Provider - Update.),
but could not acquirelock within (3000) milisecond. 
Please try after few minutes 

This document describes how to troubleshoot these mutex lock errors.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco CRS

• Cisco Unified Contact Center Express

• DC Directory Administration

• Active Directory

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Problem

When a user attempts to update the JTAPI Triggers/upload prompts or scripts using the Cisco CRS Application Admin, this error message appears:

java.lang.InterruptedException: User (CRSuser) 
attempt to acquire mutex lock for the
purpose of (Cluster Mutex acquired by JTAPI Provider - Update.),
but could not acquirelock within (3000) milisecond. 
Please try after few minutes 

This is a known defect when a lock entry is missing in the Lightweight Directory Access Protocol (LDAP). This issue is documented by Cisco bug ID CSCsd13553 (registered customers only) .

Solution 1 - For the DC Directory Environment

If this is a DC Directory environment, use this solution to solve the problem.

Note: You need to log into the DC Directory Manager as a Directory Manager in order to make the necessary changes.

1. In the DC Directory LDAP, choose CCN Apps > clusters > [profile] > Locks > Locks.00000000 and confirm that these mutex lock entries are named as this list shows:

   lockApplicant?empty
   lockOwner?empty
   lockUsage?empty,
   lockUserInfo?empty
   lockUserTimestamp?empty

2. If any of the entries in step 1 are missing the ?empty suffix in their name, then they need to be renamed to exactly match the list in step 1.

   Note: You can ignore the lockExpiration entry. It does not need the ?empty suffix in the name.

3. If any of the lock____?empty entries are missing completely, you must add them manually. In order to add the entry, complete these steps:

   Note: The lockApplicant?empty value is used for illustration purposes only.

   1. Right-click on Locks.00000000 and select New > ciscoCCNocConfigInfoCES.

   2. Enter the name as lockApplicant?empty and press Enter.

   3. In the next window, click Add and enter x in the Enter String value box. Then click OK.

   4. Click OK again.

4. Once you have confirmed that all of these entries are named properly, confirm that these entries have the value configured as x (lowercase x):

   lockApplicant?empty
   lockOwner?empty
   lockUsage?empty,
   lockUserInfo?empty
   lockUserTimestamp?empty

   Note: Ignore the lockExpiration entry in this step. Its value should not be x.

   If any of these lock entry values is not configured as x, then configure them as x.

Solution 2 - For the Active Directory Environment

If you have an Active Directory (AD) integration, you need to use ADSI Edit in order to change the lock parameters. Complete these steps in order to resolve the issue in an AD environment:

1. On the AD server, you can browse your directory schema when you open the Active Directory Services Interface (ADSI) edit utility. Then drill down to dc=xxxxx, dc=com, ou=Cisco, ou=CCNApps, ou=clusters, ou= <profilename>, ou=Locks, ou=Locks.000000000.

2. Check that the lock entries are named as this list shows:

   lockApplicant?empty
   lockOwner?empty
   lockUsage?empty,
   lockUserInfo?empty
   lockUserTimestamp?empty

3. If any of the entries in step 2 are missing the ?empty suffix in their name, then they need to be renamed to exactly match the list in step 2.

4. If any of the lock____?empty entries are missing completely, then you need to add them manually. Complete these steps in order to add the entry:

   Note: The lockApplicant?empty value is used for illustration purposes only.

   1. Right-click on Locks.00000000 and select New > Object > ciscoCCNocConfigInfoCES.

   2. Enter the name as lockApplicant?empty and press Next.

   3. In the next window, click More Attributes.

   4. From the Select a Property to View pull-down menu, select ciscoCCNatConfigInfoCESValue.

   5. In the Edit Attribute: box, enter x and click Add.

   6. Click OK.

   7. Click Finish.

5. Once you have confirmed that all entries are named properly, confirm that these entries have the value configured as x (lowercase x):

   lockApplicant?empty
   lockOwner?empty
   lockUsage?empty,
   lockUserInfo?empty
   lockUserTimestamp?empty

   Note: Ignore the lockExpiration entry in this step. Its value should not be x.

   If any of these lock entry values are not configured as x, then complete these steps in order to configure them as x:

   1. Right-click on lockApplicant?empty and choose Properties.

      Note: The lockApplicant?empty value is used for illustration purposes only.

   2. From the Attributes: box, select ciscoCCNatConfigInfoCESValue and click Edit.

   3. Highlight the existing entry in the Values: box and click Remove (skip if none are present).

   4. In the Value to add: box, type x and click Add. Then click OK.

   5. Click Apply and then OK.

Error: Cannot acquire ClusterMutex

When the user sets up call wrap up time for the agents in the Customer Response Solutions Administration application, this error message appears:

Can not acquire ClusterMutex; nested exception is: com.cisco.config.ConfigException:
UnmarshalException; nested exception is: javax.xml.bind.UnmarshalException: Content is not
allowed in prolog. - with linked exception: [org.xml.sax.SAXParseException: Content is not
allowed in prolog.]

Solution

Complete these steps in order to resolve this issue:

1. Go to the C:\program files\wfavvid\ClusterData\Default\ folder on the CRS server.

2. Rename the com.cisco.crs.cluster.config.LockConfig folder to com.cisco.crs.cluster.config.LockConfig.bak.

3. Restart Node Manager.

If you do not wish to restart Node Manager, here is another way of clearing MutexLocks:

1. Click Start and type CET.

2. Choose No on popup message.

3. Find and click com.cisco.crs.cluster.config.LockConfig in the list located on the left.

4. Double-click the one record located on the right.

5. Select the com.cisco.crs.cluster.config.LockConfig tab located at the top.

6. Clear any fields that are not empty.

UCCX Mutex Lock Error

Problem 1

When you try to change the skills of a resource, this error is received:

Error: can not acquire ClusterMutex; nested exception is:
com.cisco.config.ConfigException: Store config record – error: config 
request timed out.

Solution

This error can occur due to one of these issues:

• The backup process did not clear the Lock from the DB, but the locks and the Archive are clean on both servers.

• The lock config file may be having problem. Specifically, the server cannot read from it or the XML file inside it became corrupted.

Complete these steps in order to fix this issue:

1. Verify from the CET that the locks and the Archive are clean on both servers.

2. Verify the NIC order and that cliconfg is set correctly.

3. Go to the C:\program files\wfavvid\ClusterData\Default\ folder on the CRS server.

4. Rename the com.cisco.crs.cluster.config.LockConfig folder to com.cisco.crs.cluster.config.LockConfig.bak.

5. Reboot the Cluster.

Complete these steps in order to verify the Mutex lock setting on the DB:

1. Go to Start > Run, type in cet, and press Enter.

2. Click No when the window pops up.

3. In the left-hand pane, double-click on this Configuration Object Type: com.cisco.crs.cluster.config.ClusterSpecificConfig.

4. In the right-hand pane, double-click on the row returned for your node.

5. In the new window, click the com.cisco.crs.cluster.config.ClusterSpecificConfig tab.

6. Click the Archive tab.

   • If anything exists in double quotes regarding Archive ID, Archive Request Info, Archive User Info, or Archive Client, delete the content, but leave the double quote.

     

     Click Apply.

7. Click OK in order for the changes to take effect.

8. Select the com.cisco.crs.cluster.config.LockConfig tab located at the top.

   • If anything exists in double quotes regarding Lock Owner, Lock Usage, or Lock User Info, delete the content, but leave the double quotes.

     

     Click Apply.

9. Click OK in order for the changes to take effect.

10. Perform the same procedure in the second node if you have two UCCX servers.

Problem 2

When trying to update existing configuration, this error is received:

User (lawr) attempt to acquire mutex lock for the purpose of (Cluster Mutex 
acquired by ICD - CSD RG Update.), but could not acquire lock within (3000) 
milisecond. Please try after few minutes

If you reboot and restart Node Manager, the RMCM subsystem gets stuck in the Initializing state. When trying to release the lock, you have to delete some of the attributes and create new ones. As a result, LDAP sometimes throws an error. This causes that attribute not to be created. From this point onwards, any Appadmin operation will result in a ClusterMutex error, and a restart of the engine will cause RmCm to be stuck in the Initializing state because it cannot obtain the ClusterMutex lock.

Solution

Complete these steps in order to add the lockApplicant entry:

1. Right-click the Locks.xxxxxxx, and choose New > ciscoCCNocConfigInfoCES.

2. Enter the name as lockApplicant?empty, and press Enter.

3. In the next window, click Add, and in the Enter String value box, enter x.

4. Click Ok.

This is documented in Cisco Bug ID CSCsd13553 (registered customers only) .

Related Information

• CRAThere has been an error while reading or saving to the database. Please contact Admin error message appears when you update CSQ, skills, or resource information in CRS 3.x
• Cisco Unified Contact Center Express 5.0/7.0: "Error While Handling The Input Request" Error Message
• Voice and Unified Communications Product Support
• Cisco Unified Contact Center Express Support
• Technical Support & Documentation - Cisco Systems