Guest

Cisco Unified Intelligent Contact Management Hosted

Field Notice: FN - 62882 - Router Out-of-Sync Condition in Cisco Unified Contact Center and Unified Intelligent Contact Management Enterprise and Hosted Products


August 27, 2007

NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.


Products Affected

Product

Products Affected

Cisco Unified Contact Center Enterprise and Hosted - 7.x(x) and 6.x(x), all SRs/MRs

Cisco Unified Intelligent Contact Management Enterprise and Hosted - 7.x(x) and 6.x(x), all SRs/MRs

Cisco Unitifed System Contact Center Enterprise - 7.x(x), all SRs/MRs

Problem Description

In a duplexed ICM environment, the ICM Router may, under rare circumstances, become out-of-sync with its peer. The main trigger for this event is a failed data checksum comparison between the RouterA and RouterB. An out-of-sync condition is a very rare instance and may cause Router to process messages differently between the two sides. When the Router detects that it is out-of-sync with its partner, the Router will generate a Standard Object Dump (SOD) file which is a memory dump of the Router at the point of failure in the following directories:

:\icm\\ra
:\icm\\rb

The Router will also log an event. See Problem Detection and Serviceability information in the Problem Symptom section of this field notice.

This notice is intended to alert users of known root causes and caveats related to Router out-of-sync conditions and to provide necessary software updates to avoid experiencing such problems.

Background

In a duplexed operation, the Router works in synchronous mode. The duplex Routers regularly synchronize their configuration and state data and handle all operations simultaneously. If one of the Routers fails due to any error, the other side will stay online and handle all requests with no interruption in service. When the inactive side comes back online, it will receive the current state of configuration and call status from the active side and they will run in sync again.

The Router sync check is automatically performed every 15 seconds. During this check, Router performs a checksum of certain variables like Calltype data, lables etc. and determines if the Routers are in sync. If the Routers are not in Sync, the Router generates a SOD file, logs a message, and, by default continues operating.

The Router can optionally be configured to automatically initiate a state transfer to synchronize with its partner. This can be achieved with the following registry keys at registry patch:

HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems,Inc.\ICM\\\Router\CurrentVersion\Configuration\SyncCheck

ExitAOnSyncCheckFailure: If enabled, the RouterA will exit if it detects an out of sync failure.

ExitBOnSyncCheckFailure: If enabled, the RouterB will exit if it detects an out of sync failure.

Refer to the Release notes of the bug CSCma27904 (registered customers only) for detailed information.

Problem Symptom

When an out-of-sync occurs, calls may appear to randomly fail or to be incorrectly routed. In addition:

  1. Database values may go out of sync - The database reports on the tables CallTypeHalfHour, TCD, and RCD between sideA and sideB may go out of sync.

  2. ATR configuration changes may fail - When Router goes out-of-sync, Agent Targeting Rule (ATR) configuration update may fail.

Problem Detection and Serviceability

Only the Router can detect that it is out of sync with its partner. When it does, it responds with:

  1. Router Log traces

    The process log prints the information in the Router logs. The following is a sample of this information:

    RouterA side:

    07:00:07 ra-rtr The router has detected that it is no longer synchronized with its partner. 
    07:00:07 ra-rtr Trace: RunningSyncCheck failure: Side A reported 648442A8, B reported 84F5B217 
    07:00:09 ra-rtr Trace: Wrote 318807 records to sync364.sod, total length = 61497766 bytes. 
    07:00:09 ra-rtr Trace: Router dump created in sync364.sod
    

    RouterB side:

    07:00:08 rb-rtr The router has detected that it is no longer synchronized with its partner. 
    07:00:08 rb-rtr Trace: RunningSyncCheck failure: Side A reported 648442A8, B reported 84F5B217
    07:00:10 rb-rtr Trace: Wrote 318807 records to sync364.sod, total length = 61497766 bytes. 
    07:00:10 rb-rtr Trace: Router dump created in sync364.sod
    
  2. .SOD Files

    The .SOD files are generated by Router and will be stored in ICM Directories in both side of the router, :\icm\\ra and in the :\icm\\rb.

  3. Windows Application Log Event

    Windows event is generated to indicate Router out-of sync. Message details:

    Event ID: 572 
    Source: Call Router 
    Event Description: The router has detected that it is no longer synchronized with it's partner?
    
  4. SNMP Trap

    Router out-of sync generates a SNMP trap.

Note:  Some of the above serviceability items may not appear. See CSCsj88170 (registered customers only) and CSCsh33685 (registered customers only) for exceptions and fixes.

Workaround/Solution

The workaround for this condition is to restart both Routers. A restart is accomplished by restarting the service through the ICM service window.

Note: A system reboot is not required.

Note: You can also choose to restart only one side of the duplexed Routers. By cycling only one Router, it will force a state transfer from the active side and as a result both Routers should then be in sync.

Note: In very rare instances the cycling of one CallRouter does not resolve the issue. When this occurs, run in simplex mode and then schedule a maintenance window to shut down the Central Controller nodes (Routers and Loggers) at the same time and bring them back up in the following order:

Note: LoggerA, RouterA, RouterB, LoggerB.

Note: Cisco strongly recommends that a maintenance window be scheduled to perform these recovery actions as to lessen the impact to production call routing.

Note: Cisco has identified several caveats related to out-of-sync problems and has made changes to address them in ICM 7.1(5) ES19, ICM 7.2(1), ICM 6.0 SR11 and all later maintenance releases (MRs). Cisco strongly recommends upgrading to these versions to avoid experiencing out-or-sync related problems. Please refer to the DDTS section of this notice for details.

Note: ICM 7.1.5 ES19 can be downloaded from the ICM7.1.5_ES19 software download link.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

DDTS

Description

CSCsh67876 (registered customers only)

ATR causes router out of sync after state transfer

CSCsh63301 (registered customers only)

Requery fails if call is sent to VRU after TransRouted to type2 VRU

CSCsi24395 (registered customers only)

Router may go out of sync when there are calls queued during state transfer

CSCsh33685 (registered customers only)

Serviceability improvement - Router may not generate sod file when out of sync occurs

CSCma23597 (registered customers only)

Serviceability improvement - rttest Should Show When Last Out of Sync Router Dump Occurred

CSCsj88170 (registered customers only)

Serviceability improvement - ICM Router Out of sync is not properly reflected in RTTest

CSCma27904 (registered customers only)

Serviceability improvement - Routers do not Automatically Restart When Out Of Sync

Revision History

Revision

Date

Comment

1.0

27-AUG-2007

Initial Public Release

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.