Guest

Cisco BTS 10200 Softswitch

Field Notice: FN - 61861 - BTS 10200 Softswitch - MGA Process Heap Memory Can Cause Failover - Proactive Monitor of Heap Memory


Revised May 9, 2007

October 04, 2004

NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.


Products Affected

  • BTS 10200 Softswitch - All versions

Problem Description

BTS systems that have a large number of Media Gateway Control Protocol (MGCP) terminations, greater than 20,000, and that have peak call rates exceeding 20 calls per second may experience growth of MGA heap memory beyond the limit allowed by the BTS platform configuration file. The limit is currently set at 500MB. If the MGA process attempts to allocate heap memory beyond this limit the Call Agent application will fail over to the standby side.

Background

The MGA process uses heap memory primarily for temporary storage of recent MGCP commands and responses in case retransmission is necessary due to network loss.

The BTS platform configuration file limits the amount of heap memory that each process may use to 500MB per process. At run time processes allocate heap memory incrementally on an as needed basis and use only the amount required for execution. Each process can only allocate up to its 500MB limit as defined in the platform configuration file.

Problem Symptoms

Systems with a large number of MGCP terminations that have peak call rates greater than 20 calls per second, may suddenly fail over to the secondary side after running for several weeks.

Workaround/Solution

The BTS engineering team recommends monitoring MGA heap memory usage on systems that have a large number of MGCP terminations and experience peak call rates of 20 calls per second or greater. The monitoring should be done once per day at a time when call volume is lowest, typically 3:00 AM to 5:00 AM for most deployments. A tool named show_heap has been created to facilitate the monitoring.

Recommendations for monitoring are as follows:

  1. Identify the time of day when call volume is lowest.

  2. Create a shell script that executes the commands date and show_heap <pid of mga> redirecting the output of both commands to a file in the /opt directory.

  3. Add an entry in root's crontab to call the shell script in step 2 at the time identified in step 1.

  4. Check the output file daily and observe the amount of heap memory used by the MGA process.

  5. If MGA heap usage reaches 400MB (80 percent of the limit), a proactive switchover should be done as soon as possible in the next maintenance window. The daily heap growth rates indicated in the output file will provide guidance on the urgency of a proactive switchover.

Customers should contact their account team to obtain the show_heap tool and if they need additional clarification or assistance with the monitoring recommendations.

The BTS engineering team is actively investigating the appropriate upper limit of heap memory usage required by the MGA process. Due to the nature of the issue and the slow incremental growth of heap, each study can take several weeks to complete. An update to this Field Notice will be published when more information is available.

A patch (P22) that slows the heap growth by a factor of 10 was issued for this problem in June 2005 on release 3.5.4 V01. The patch incorporates a fix for CSCsb03819. MGA should use mtmalloc heap memory allocator.

In addition, to reduce the interval between the maintenance switchovers, the heap memory size for MGA processes has been increased from 500 MB to 1 GB. Hence, the recommended failover should be modified to 80 percent of the new 1 GB size.

The fix is also incorporated into later BTS releases including 3.5.5, 4.4 and 4.5.

Further investigation of the problem is pending availability of new Solaris tools that come with Solaris 10 on release 4.5.

Revision History

Revision

Date

Comment

1.5

09-MAY-2007

Updated the Workaround/Solution section to address the increase in heap memory size for MGA processes.

1.4

09-FEB-2006

Updated title

1.3

08-FEB-2006

Added patch info to Workaround

1.2

18-JAN-2005

Maint. Update

1.1

27-OCT-2004

Maint. Update

1.0

04-OCT-2004

Initial Public Release

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.