Introduction
This document describes the steps to troubleshoot when the UCCE Logger A and B are stuck in an initializing state.
Contributed by Pratham Prakash, Cisco Software Engineer.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Cisco UCCE
- Micrsoft Structured Query Language (SQL)
Components Used
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Problem
Log analysis revealed that UCCE Logger A and B are stuck in an initialization state. Loggers on both sides will not become active and the loggers keep crashing with an exception bcp connection exhausted. An example of error message for this condition can be found in the log files.
14:09:45:286 la-rcv Trace: SQL Server User Error: 2627, State 1, Severity: 14, Message:
Violation of PRIMARY KEY constraint 'XPKPeripheral_Interval'. Cannot insert duplicate key
in object 'dbo.t_Peripheral_Interval'. The duplicate key value is (Jul 3 2015 12:30PM,
5002, 300, 1).
14:09:45:335 la-rcv Trace: Duplicate key ignored because the record already exist in the
database.
14:09:45:335 la-rcv Trace: bcp_done failed
This is ocurring because there are duplicate keys found in the t_Persistent_Variable table. Neither logger A and B is able to complete initialization.
Solution
This condition can occur when using Persistent Variables on UCCE release 10.x ThedDefect "CSCuw02024 t_Persistent_Variable table deleting and re-adding records".
Perform following workaround
Step 1. Set the following registry key on ogger Side A and Logger Side B from value 1 to 0
HKEY_LOCAL_MACHINE\Software\Geotel\ICR\Customerinstance\LoggerB\Logger\HistoricalData\Persistent
Step 2. Bring one side down
1) truncate the Persistent_VariableTmp1, Persistent_VariableTmp2 and t_Persistent_Variable tables on down side.
2) truncate the Persistent_VariableTmp1, Persistent_VariableTmp2 and t_Persistent_Variable tables on active side.
Step 3 Restart Logger Service on Both Side A and Side B
Step 4 Perform test to make sure users are able to make configuration changes.
Step 5 Place test call into the system to verify calls are working.
Step 6 It may still be neeccesary to Execute exit_router, It was found that the system is up and running, and both sides of routers completed the state transfer by taking configuration from side A logger. Though the contact center system is running and working, Side B logger db still in initializing state. This occured when Side B logger recovery key is lagging Side A logger by huge amount.
Step 7 Performing manual config db from A --> B
Performed manual export/import config data from A --> B
Though lastUpdatekey is matched between side A and B, Logger B clgr complained of a checksum error. Perform manual logger config db sync through ICMDBA to prevent checksum error.
Later performed below steps to resolve checksum issue
1. Stopped config change by having changed the DBMaintnenance registry key to 1
2. Backed up the entire logger A database on MSSQL. And transferred the db backup to Logger B server.
3. Dropped logger B database, and recreated the Logger B database.
4. Restored the logger db on Logger B from the db backup from logger A.
5. Powered logger B service backup.
6. Reset the DBMaintenance registry key to 0
Verified
1. Router rttest has successfully established MDS connection with Logger B processes, including CLGR, HLGR, RCV etcs.
2. Logger B does not drop out from MDS due to Data Checksum error.
3. Since Logger B has been in shutdown state for a few days, the system is now actively syncing Historical data with HDS.
4. Config change is still working
Feedback