September 20, 2002
Products Affected
-
Cisco Intelligent Contact Manager (ICM) Version 2.5
-
ICM Version 3.0 ICM Version 4.1.x
-
ICM Version 4.5.x
-
ICM Version 4.6.x
-
IP Contact Center (IPCC) Version 4.1.x
-
IPCC Version 4.5.x
-
IPCC Version 4.6.x
-
Cisco Network Applications Manager (NAM) Version 2.5
-
NAM Version 3.0
-
NAM Version 4.0.x
-
NAM Version 4.1.x
-
NAM Version 4.5.x
-
NAM Version 4.6.x
Problem Description
Recently, an ICM customer with an extremely large configuration experienced a system failure while manipulating Call Router state transfer data in a duplexed central controller environment. The specific incident occurred after an attempt to upgrade to ICM version 4.6.2 from version 4.0, but the root cause of the underlying problem is the configuration size and is independent of the upgrade itself. Since the underlying configuration element scope is larger in ICM version 4.6.2 than in ICM version 4.0, the upgrade triggered the failure event.
Background
In preparation for the joining of a duplexed Call Router pair, the running side builds an internal structure of "standard objects" that conveys dynamic ICM state to the pairing (joining) side. These objects are derived from the complete set of configuration elements including routes, services, skills, trunks, call types, and additional data, and the resulting size of the object structure is relative to the quantity of configuration elements across each specific category. The number of objects required for a given configuration element differs by element type. For example, a service may require 35 objects while a route requires 13. The sum total of all objects across configuration elements is limited in the current ICM to approximately one million.
ICM Engineering is working to derive precise calculations that customers can apply to their specific configurations to accurately determine the magnitude of their configuration object size and calculate the degree of margin available before they approach the point of a failure.
Problem Symptoms
The specific evidence of this failure is a Call Router assertion, which can be identified in the corresponding Call Router EMS logfile. An example appears as follows:
13:26:23 rb-rtr Fail: Assertion failed: info->count <= instanceNumberMax. File: S:\icm\ICRLib\StdObj\TRANBLD.CPP. Line 751
It should be stressed that, while of obvious concern, this issue applies only to extremely large ICM configurations. As noted above, ICM upgrades can also exacerbate the problem due to incremental increases in overall configuration object size. While the specific formulas to quantify size are not yet available, a representative configuration that would raise concern is as follows:
Trunk Groups 750 Services 18000 Routes 32000 Script Cross References 50000 Peripherals 100
The ICM Engineering team is actively working to rectify the issue, and a corresponding hotfix will be made available at a near future date. Cisco recommends that pending ICM migrations for production level environments be postponed until this hotfix is created.
Workaround/Solution
In the unlikely event that you encounter this problem, the immediate workaround is to intentionally avoid pairing the Call Router machines by shutting down one side with ICM Service Control. Full call routing functions are, of course, available in simplex mode - albeit with the lowered risk of a system component failure impact.
This Field Notice and the associated DDTS will be updated as further information becomes available. Again, Cisco recommends that pending ICM migrations for production level environments be postponed until this hotfix is created.
DDTS
To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.
For More Information
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
Receive Email Notification For New Field Notices
Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.