Guest

Cisco Unified Intelligent Contact Management Hosted

Field Notice: FN - 63371 - Unable to Make Config Changes - UpdateAW Process Failure

Field Notice: FN - 63371 - Unable to Make Config Changes -- UpdateAW Process Failure

Revised January 28, 2011

October 22, 2010


NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision Date Comment
1.5
28-JAN-2011
Added defect CSCtk01278 to DDTS section; updated Workaround/Solution section.
1.4
12-NOV-2010
Added defect CSCtj96993 to DDTS section; added Related Issues section.
1.3
1-NOV-2010
Added ICMs to Workaround/Solution section.
1.2
29-OCT-2010
Removed Related Issues section; updated Workaround/Solution section; added statements at end of Workaround/Solution section; under Problem Symptoms and DDTS sections, changed the word "crash" to "unexpectedly exit".
1.1
27-OCT-2010
Added defect CSCtj67119 to the DDTS section; added to Problem Description/Symptom 1; added links under Workaround; updated Engineering Special links; added Related Issues section.
1.0
22-OCT-2010
Initial Public Release

Products Affected

Products Affected Comments
CCMP - 7.x, 8.x
NOTE: If the central controller is restarted there is a very high likelihood of hitting this issue.
ICME/CCE AW - 4.x, 5.x, 6.x, 7.x, 8.x
NOTE: If the central controller is restarted there is a very high likelihood of hitting this issue.

Problem Description

Problem/Issue

Cisco Unified Contact Center Enterprise (UCCE/H and ICME/H) system administrators are unable to make configuration changes from an Admin Workstation (AW) using Unified ICM/CCE Configuration Manager, Script Editor tool, Internet Script Editor (ISE), Contact Center Management Portal (CCMP), or via Automated Administrator for Symposium (AAS).

Symptom

UpdateAW.exe process restarts on AW when making configuration changes. See defect CSCtj51228 for added detail.

Another symptom is that ICM/CCE Router starts to reject any configuration updates, which again prevents system administrators from making any configuration changes. See defect CSCtj65060 for added detail.

The Internet Script Editor asserts randomly a few times a day. See CSCtj67119 for added detail.

Impact

Administrator is unable to make configuration or change/update any routing or admin scripts for Unified ICM/CCE.

Likelihood

  • The problem can occur under normal operation.
  • The probability increases if both sides of Central Controller have been restarted recently.
  • Exposure to the problem increases when using the ICMDBA tool to back up and restore the database with the "Truncate Config Message Log" option checked.

Background

Issue

Multiple occurrences of this issue have been observed since October 14, 2010 and on all Unified ICM/CCE releases. This is not a regression defect. The issue occurs when a Microsoft SQL Server query operation is performed to compare the 'lastUpdateKeys' on AW and Central Controller/Logger, during the incremental copying of configuration changes to the local AW. In addition, using the ICMDBA tool to backup and restore the database with "Truncate Config Message Log" option checked can lead to a similar issue that causes the ICM/CCE Router to reject configuration updates. No UpdateAW restart is observed in this situation.

Symptom

UpdateAW.exe process restarts on AW when making configuration changes. See defect CSCtj51228 for added detail.

Another symptom is that ICM/CCE Router starts to reject any configuration updates, which again prevents system administrators from making any configuration changes. See CSCtj65060 for details.

Impact

Administrator is unable to make configuration or change/update any routing or admin scripts for Unified ICM/CCE.

Likelihood

  • The problem can occur under normal operation.
  • The probability increases if both sides of Central Controller have been restarted recently.
  • Exposure to the problem increases when using the ICMDBA tool to back up and restore the database with the "Truncate Config Message Log" option checked.

Root Cause

The root cause for this issue is in an internal algorithm using a base time reference to generate values for 'lastUpdateKeys'. The key computation can cause the value to exceed the threshold such that it cannot always be represented in 64 bit IEEE-754 format as an integer without losing precision. The loss of precision causes the (above-mentioned) floating point comparison of 'lastUpdateKeys' to fail, in turn causing the configuration updates to be blocked.

Problem Symptoms

When configuration changes are saved, the updateAW.exe process (uaw) on the Real-Time Distributor machine restarts, and prior to restarting, it produces log messages similar to this one:

14:10:37:436 dis-uaw Trace: Starting incremental copy operation.
14:10:37:436 dis-uaw Trace: Recovery keys: 576559966035.0 (in memory), 576559966035.0
(local AWControl), 576559966039.0 (router AWControl).
14:10:37:436 dis-uaw Trace: Recovery keys: 576559966035.0 (in memory), 576559966035.0
(local AWControl), 576559966039.0 (router AWControl).
14:10:37:436 dis-uaw Trace: Central log was truncated. Must initialize local database.

If ICMDBA has been used to backup and restore the configuration database with "Truncate Config Message Log" option checked, the Router may start preventing any further configuration updates. In this scenario, no updateAW.exe unexpectedly exit is observed. The updateAW process on the real-time distributor logs a message similar to this one:

"Admin Workstation configuration data is out of date with the central controller. Please update your local ICM database"

The router process logs a message similar to this one:

"UpdateCC from Administrator on RBBH-AW rejected because wrong key used. Expected 576864042017, got 576864042017."

Workaround/Solution

There is no workaround for the UpdateAW.exe restart issue. Cisco has made an Engineering Special available for this defect:

CSCtj51228
The Engineering Special will be merged forward to the subsequent maintenance releases for fielded versions under active support.

ICM 6.0(0)SR12_ET10
Installer for Service Release 12

ICM 7.1(2)ES31
Installer for Engineering Special 31

ICM 7.1(3)ES56
Installer for Engineering Special 56

ICM 7.1(4)ES50
Installer for Engineering Special 50

ICM 7.1(5)ES155
Installer for Engineering Special 155

ICM 7.2(2)ES12
Installer for Engineering Special 12

ICM 7.2(3)ES9
Installer for Engineering Special 9

ICM 7.2(4)ES47
Installer for Engineering Special 47

ICM 7.2(5)ES23
Installer for Engineering Special 23

ICM 7.2(6)ES23
Installer for Engineering Special 23

ICM 7.2(7)ES139
Installer for Engineering Special 139

ICM 7.5(2)ES6
Installer for Engineering Special 6

ICM 7.5(3)ES16
Installer for Engineering Special 16

ICM 7.5(4)ES23
Installer for Engineering Special 23

ICM 7.5(5)ES13
Installer for Engineering Special 13

ICM 7.5(6)ES73
Installer for Engineering Special 73

ICM 7.5(7)ES10
Installer for Engineering Special 10

ICM 7.5(8)ES15
Installer for Engineering Special 15

ICM 7.5(9)ES9
Installer for Engineering Special 9

ICM 8.0(2)ES5
Installer for Engineering Special 5

ICM 8.0(3)
Engineering Special not required. Fix has been included in the Maintenance Release.

PLEASE NOTE: We are providing a fix for some End of life (EoL) and End of Sale(EoS) maintenance releases as an exception.
End-of-Life and End-of-Sale Notices

MR Download page
MR Download page

Regarding defect CSCtj65060:
Currently no software resolution is available for this defect. As a safeguard, it is strongly recommended that ICM/UCCE administrators uncheck the "Truncate Config Message Log" option while using the ICMDBA tool to export the configuration database from the Logger. However if you are experiencing symptoms described above, please see the defect CSCtj65060 for workaround.

If you are experiencing the symptom mentioned below, perform these steps noted to workaround the issue:

  1. Stop RouterA, RouterB, LoggerA, Logger B.
  2. Stop AWs.
  3. Take the back-up of the Logger database.
  4. Delete the rows from Config_Message_Log table from LoggerA, except the first row with RecoveryKey=0. Sample SQL Delete Query:
    delete Config_Message_Log where RecoveryKey >= 1
  5. Restart the LoggerA, RouterA and One AW which is connecting to the A side.
  6. Perform configuration updates and verify they are successful.
  7. Now Sync the logger database from A->B using ICMDBA.
  8. Start LoggerB and RouterB, followed by any other AWs.

Regarding defect CSCtk01278:
The CIM Config Deletion is a side effect of Customers NOT having ES for CSCtj51228 applied on their AW Distributor Servers.
There are only two specific conditions in which the UAW process would do the Re-initialization of the Configuration Data from Central Controller:

  1. When the Initialize Local Database (AWInit.exe) tool is used, to MANUALLY force the AW to delete its current configuration and fetch a copy of the latest Configuration from the Central Controller. This is generally a one-time operation, which is done when we set up the AW Server for the first time.
  2. When the UAW process detects that the local Configuration Data may be bad or inconsistent with the Central Controller Database, and it needs to fetch a fresh copy of the latest Configuration from Central Controller. The Recovery Keys are vital for UAW process to make this determination, and one of the most common points of failure has been the floating point comparison of the Recovery Keys which fails, as noted in defect CSCtj51228.
CSCtj51228 in point b above, causes the UAW to delete and re-sync configuration from Central Controller DB, each time a Configuration change is made by an Administrator. This operation of delete and copy of Configuration data, may leave an approximate 8-15 second window, in which if CIM does a AW_DB dip directly using an ODBC Connection, it will encounter NULL entries in the Config Tables, causing it to trigger a Delete of Configuration in CIM DB.

Pre-emptive application of the ES for CSCtj51228 (FLOATING POINT RECOVERY KEY COMPARISON) to all Admin Workstations safely avoids triggering CSCtk01278 - inadvertent deletion of CIM configuration.

Cisco is continuing to research these issues and will provide an update to this field notice shortly.

Additional Information
To access the Bug Toolkit, log onto http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs (registered customers only)

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered customer and you must be logged in.

DDTS Description
CSCtj51228 (registered customers only) UpdateAW process restarted whenever a configure change is made.
CSCtj65060 (registered customers only) Configuration updates are rejected in router.
CSCtj67119 (registered customers only) Internet Script editor unexpectedly exits intermittently while synchronizing data.
CSCtj96993 (registered customers only) Recovery Key issue after Logger DB import to AW during post-upgrade.
CSCtk01278 (registered customers only) CIM lost intergated Queues after AW reboot

Related Issues

CDETS CSCtj67119 - Internet Script editor crashes

CDETS CSCtj96993 - Recovery Key issue after Logger DB import to AW during post-upgrade

CDETS CSCtk01278 - CIM lost intergated Queues after AW reboot

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.