Guest

Cisco Emergency Responder

Field Notice: FN - 63173 - Cisco Emergency Responder - Memory Leak May Cause 911 Service Outage - Upgrade Required

Field Notice: FN - 63173 - Cisco Emergency Responder - Memory Leak May Cause 911 Service Outage - Upgrade Required

November 19, 2008


NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision Date Comment
1.0
19-NOV-2008
Initial Public Release

Products Affected

Products Affected Comments
CER - 1.1(1)
CER - 1.1(2)
CER - 1.1(3)
CER - 1.1(4)
CER - 1.2(1)
Including 1.2(1)_SR1
CER - 1.2(2)
Including 1.2(2)_SR1
CER - 1.2(3a)
Including 1.2(3a)_SR1 and 1.2(3a)_SR2
CER - 1.3(1a)
CER - 1.3(2)

Problem Description

There are two issues related to memory leakage in Cisco Emergency Responder (CER) versions 1.3(2) and earlier:

1. There is a possibility that all the system memory will be consumed due to a memory leak in the Cisco Discovery Protocol (CDP) Driver.

2. There is a possibility of the CERServer service running out of memory when CER is left operational for a long period of time. It is not possible to determine exactly how long it takes for memory resources to be exhausted.

Background

This vulnerablity will manifest itself in various circumstances and is intermittent in nature. There is not one specific trigger that will cause this event to occur.

Problem Symptoms

The symptoms for CERServer service memory leak are as follows:
1. Once memory space is consumed it will result in the following error while trying to perform administrative task when logged into CER System Administration:"Failed to contact Cisco Emergency Responder"
2. Another symptom of this condition is that the E911 Route Point configured in Cisco Unified Communications Manager (Cisco CallManager) will show as "Unregistered".
3. Routing of emergency (911) calls may fail or may result in a fast busy signal, since CER will not be able to handle the call.
4. The stacktrace.log file located at C:\Program Files\Cisco Systems\CiscoER\logs will show a "java.lang.OutOfMemory" error.
Note: Rebooting the system in this state will result in the stacktrace.log file getting cleared.

The symptom for the CDP driver memory leak is as follows:
1. After the Cisco Emergency Responder server has been running without a reboot for several months, the server will become sluggish and ultimately become unresponsive causing 911 calls to fail. After the server is rebooted, the following error may be found in the system event log with the description "The server was unable to allocate from the system nonpaged pool because the pool was empty."

Workaround/Solution

Customers may perform one or both of the following to address these defects.

Workaround
The workaround to this problem is to reboot the CER server. This frees the system memory and will restore functionality.

Note: Rebooting is a temporary workaround. The system will continue to have memory leaks and has the potential for service outage. It is not possible to predict how long it will take for memory resources to become exhausted. It is strongly recommended that customers follow the Solution section and upgrade accordingly.

Solution
Customers must upgrade to CER 1.3(2) SR1 to address the memory leaks. Before performing this upgrade, customers may perform a reboot to free memory and restore normal CER operations, but a reboot is not a prerequisite for performing the upgrade.

Customers running CER 1.3(2) can upgrade directly to CER 1.3(2) SR1.

Customers running earlier versions of CER must upgrade to CER 1.3(2) and then to CER 1.3(2) SR1.

Cisco Emergency Responder (CER) 1.3(2) SR1 can be downloaded following the link below:
Download Cisco Emergency Responder(CER) 1.3(2) SR1 (registered customers only)

Note:

This update MUST be applied to Publisher AND Subscriber servers

Cisco Security Agent (CSA) software MUST be disabled before installing CER 1.3(2) SR1

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered customer and you must be logged in.

DDTS Description
CSCsa80104 (registered customers only) Cisco Emergnecy Responder server runs out of memory
CSCsj58794 (registered customers only) Non-Paged Memory Leak is caused by CDP

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.