This document describes how to troubleshoot an issue with the Cisco Unified Contact Center Enterprise (UCCE) Version 10.x when data is not written to the Historical Data Server (HDS).
After a purge event (between 12:30 AM and 12:35 AM server time), the recovery table becomes blank and data no longer populates on the HDS from the logger.
Note: The instance name that is used for the examples throughout this document is lab, which can be interchanged with <instance>.
For the initial troubleshooting steps, you should verify the database health via ICM DBA on the loggers and distributors that encounter this problem. You should also verify the basic registry keys for purges. In order to verify the database size and usage, navigate to the ICM DBA, right-click <instance>_<component>, and click Properties. Verify that this is not at or above 80%. Check this information on the logger, HDS, and Administrative Workstation (AW) as well.
Next, you can check the Space Use Summary on the ICM DBA:
Navigate to Data > Space Used Summary from the menu near the top of the page.
Uncheck the Display Empty Tables and Display Temporary Tables check boxes.
If you view the HDS that does not contain the current data, the last time that the data was received on most of the tables indicates the time that the issue occurred.
Check the basic purge and replication registry settings on the logger:
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\ICM\lab\LoggerA\ Recovery\CurrentVersion\Purge\Schedule\Schedule Class Name: <NO CLASS> Last Write Time: 12/8/2014 - 1:41 PM Value 0 Name: Schedule Type: REG_SZ Data: 00:30 M,T,W,Th,F,S,Su
At this point, if the data is verified and correct, the next step is to pull the replication logs from the distributor and the logger. The trace value is set to 3 on Diagnostic Framework Portico, which pulls the logs soon after. The logs should show data similar to the next two examples.
Here are the logger replication logs:
14:40:55:861 la-rpl Trace: No MATCHING entry for table t_Termination_Call_Detail, FromRecoveryKey = 7369086520649.0 and ToRecoveryKey = 7369085626000.0
Here are the distributor replication logs:
14:29:52:607 dis-rpl Trace: Sent Replicated request to the Server for table t_Termination_Call_Detail, FromRecoveryKey = 7369086520649.0 and ToRecoveryKey = 7369085626000.0
In order to better understand the replication keys and the No MATCHING entry that appears in the logs, Structured Query Language (SQL) queries are run for more information about all of the loggers and distributors.
Here are the SQL queries:
select max(RecoveryKey) from t_Termination_Call_Detail select max(DateTime) from t_Termination_Call_Detail
Notice that the recovery keys for Logger A and Distributor A are significantly further apart than is expected and than that on side B. Furthermore, the most recent data on Distributor A approximately matches the location of the missing data reports that were first observed. Also note the discrepancy in the maximum RecoveryKey distance between the table from the queries and the logs.
You can view the recovery table in order to determine the keys that are stored.
Here is the SQL query:
select max(RecoveryKey) from Recovery
Here are the SQL query results:
HDSA - 7369090330048 RogA - EMPTY
In the output, you can see that Logger A appears EMPTY, with no recovery key. Since this data is unexpected, you should then verify the data in the Logger A recovery table.
Here is the SQL query:
select * from Recovery
When this query is run, the results are blank, which indicates that there is no data in the recovery table. At this point, you should reevaluate the registry entries. When the two sides are exported and a comparison is made, the discrepancy is noticed.
Here are the registry entries for Logger A:
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\ICM\lab\ LoggerA\Recovery\CurrentVersion\Purge\Retain\System\Recovery Class Name: <NO CLASS> Last Write Time: 12/8/2014 - 1:39 PM Value 0 Name: Days Type: REG_DWORD Data: 0X0000001e
Here are the registry entries for Logger B:
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\ICM\lab\ LoggerB\Recovery\CurrentVersion\Purge\Retain\System\Recovery Class Name: <NO CLASS> Last Write Time: 12/8/2014 - 1:39 PM Value 0 Name: Days Type: REG_DWORD Data: 0X00000e42
As shown on Logger A, the Recovery Days value is set for 0x1e, which is the hexadecimal (HEX) value for 30. On Logger B, the Days value is set for 0xe42, which is the HEX value for 3,650. Upon further review with the default registry entries in the lab, this seems to be the issue. This also matches the symptom of the issue, which occurs approximately once a month.
The cause of this behavior has been reproduced in our labs in two scenarios.
Scenario 1 - When you create a new logger, if a user has enabled the check box for "Display Database Purge Configuration Steps", it shows the recovery retention value as 30 days. This in turn updates the registry settings after saving.
Scenario 2 - When you delete a logger and recreate it, if a user has enabled the check box for "Display Database Purge Configuration Steps", it shows the recovery retention value as 30 days. This in turn updates the registry settings after saving.
Note: This issue is can be tracked by Cisco bug ID CSCuu26777.
Note: Cisco recommends that you perform the actions that are described in this section during a maintenance window.
In order to resolve this issue, you must update the registry key to 0xe42 (3,650), which is the default key:
Set the registry keys to 0xE42 (3,650).
Restart the service for Logger A.
Restart the service for Distributor A.
Restart the services for Logger and Distributor B, if applicable.