Document ID: 44366 |
Introduction
This document provides information about a Cisco CallManager crash and how to identify the known bugs.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software or hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Cisco CallManager Crash Description
When the Cisco CallManager service crashes, this message appears in the System Event log:
The Cisco CallManager service terminated unexpectedly. It has done this 1 time. The following corrective action will be taken in 60000 ms. Restart the service.
At this time, devices such as Cisco IP phones and gateways that are registered to the Cisco CallManager are unregistered. The Cisco CallManager service can crash due to one of these reasons:
-
An unexpected event occurs in the Cisco CallManager service. This crash generates a Dr.Watson log and a User.dmp file in the C:\Documents and Settings\All Users\Documents\DrWatson folder.
-
The Cisco CallManager service does not have enough resources, such as CPU or memory, to function. Generally the CPU utilization in the server is at 100 percent, at that time.
This document discusses only those situations in which the crash occurs due to an unexpected event.
Reading the Dr.Watson Log
Whenever there is an application crash, the Dr.Watson log is appended. Open the Dr. Watson log in Notepad, scroll to the bottom of the file, and search for Application exception occurred. This shows the latest crash:
Application exception occurred:
App: (pid=680)
When: 3/8/2003 @ 14:01:06.978
Exception number: e06d7363
Compare the date and time with the Event log message in order to ensure that the crash mentioned has the same time. The previous example output indicates that the application that crashed had a process identifier (PID) of 680. This trace lists all of the of the PIDs:
PID PROCESS 8 System.exe 212 SMSS.exe 240 CSRSS.exe 264 WINLOGON.exe 292 SERVICES.exe 304 LSASS.exe 424 termsrv.exe 520 svchost.exe 560 msdtc.exe 696 DLLHOST.exe 736 Ipvmsapp.exe 752 DLLHOST.exe 824 AudioTranslator.exe 848 RisDC.exe 860 LogoutService.E.exe 884 DCX500.exe 936 svchost.exe 980 LLSSRV.exe 1028 sqlservr.exe 1112 ntpd.exe 1140 rcmdsvc.exe 1172 regsvc.exe 1176 mstask.exe 1204 SNMP.exe 1244 WinMgmt.exe 1260 cpqnimgt.exe 1284 cqmgserv.exe 1296 cqmgstor.exe 1308 sysdown.exe 1372 cqmghost.exe 1524 aupair.exe 1552 sqlagent.exe 276 svchost.exe 2400 inetinfo.exe 2412 explorer.exe 2752 sqlmangr.exe 2700 taskmgr.exe 2704 mmc.exe 680 ccm.exe 868 DRWTSN32.exe
The PID (680) is ccm.exe, which is the Cisco CallManager service. After you verify the date and time in the Event Viewer and confirm that the crash is caused by ccm.exe, search for the word FAULT in the Dr.Watson log. That shows the location that actually caused the crash:
function: RaiseException
77eab2d4 85c9 test ecx,ecx
77eab2d6 740e jz GetVolumePathNameA+0x7e (77eb3fe6)
77eab2d8 8d4801 lea ecx,[eax+0x1] ds:0751c41a=????????
77eab2db 8d7dc4 lea edi,[ebp+0xc4] ss:0751c46a=????????
77eab2de f3a5 rep movsd ds:06cfeed8=06cfeef4 es:06cfee68=00000000
77eab2e0 eb04 jmp SetVolumeMountPointA+0x172 (77eb35e6)
77eab2e2 8365c000 and dword ptr [ebp+0xc0],0x0 ss:0751c46a=????????
77eab2e6 8d45b0 lea eax,[ebp+0xb0] ss:0751c46a=????????
77eab2e9 50 push eax
77eab2ea ff156414e877 call dword ptr [77e81464] ds:77e81464=77fb1130
FAULT ->77eab2f0 5f pop edi
77eab2f1 5e pop esi
77eab2f2 c9 leave
77eab2f3 c21000 ret 0x10
The FAULT is unique for different types of crashes. The first column is the memory location, which can vary. In this example, the fault is in 77eab2f0. However, the rest of the line, 5f pop edi, is always the same for this type of crash.
Cisco CallManager Publisher Server Cannot Start Services :DBL errors
The Cisco CallManager publisher server cannot start services since the database cannot be accessed. The Database Layer Monitor service also cannot access the database.
DBL Error Solution
The Database Layer Monitor accesses the DB through a series of DLL files. In order to resolve this problem, unregister and then re-register the database access DLLs from the Microsoft Windows operating system. This allows the core applications to make database calls again through the Cisco provided DLLs.
Cisco CallManager Services Do not Start after a Power Outage
Cisco CallManager services sometimes do not start after a server reboot or power outage when there are two Network Interface Cards (NICs) enabled and therefore two IP addresses assigned. Ensure you only have one NIC enabled on the server at a time. Dual NICs are not supported. The recommendation is to have two NICs and use one as fault tolerance, but only one is operational at a time. Failure to disable the second NIC can result in two IP addresses that are assigned to the Cisco CallManager server. When two IP addresses are assigned to the Cisco CallManager server, it can cause a loss of service. You must have only one NIC enabled (the one which is configured). Disable the one which is not used in order to resolve the issue.
List of Known Crashes and Fixes
This section lists known crashes, with FAULT codes and available fixes. If a fix is available in an Engineering Special (ES), open a case with Cisco Technical Support with the TAC Service Request Tool (registered customers only) in order to obtain a patch.
Cisco Bug ID CSCdx42096
Cisco bug ID CSCdx42096 (registered customers only) involves a Cisco CallManager crash due to badly formatted Media Gateway Control Protocol (MGCP) messages from MGCP gateways.
This shows the fault in the Dr.Watson log:
77eab2e9 50 push eax
77eab2ea ff156414e877 call dword ptr [77e81464] ds:77e81464=77fb1130
FAULT ->77eab2f0 5f pop edi
77eab2f1 5e pop esi
77eab2f2 c9 leave
This issue is fixed in these Cisco CallManager versions:
-
3.3(2)SpC
-
3.2(2c)ES64
Cisco Bug ID CSCdx32456
Cisco bug ID CSCdx32456 (registered customers only) involves Cisco CallManager crashes while an H.323 call is processed.
This shows the four possible faults in the Dr.Watson log that can cause the crash:
FAULT ->005783e7 f3a5 FAULT ->005777ea 8b00 FAULT ->0057784a 8b00 FAULT ->005790c7 8b5004
This issue is fixed in these Cisco CallManager versions:
-
3.2(2c)
-
3.3(2)
Cisco Bug ID CSCdz69051
Cisco bug ID CSCdz69051 (registered customers only) involves a Cisco CallManager crash because the array is out of bounds.
This shows the fault in the Dr. Watson log:
77e989ca 50 push eax
77e989cb ff156414e877 call dword ptr [77e81464] ds:77e81464=77fb0f18
FAULT ->77e989d1 e978f80100 jmp SetThreadContext+0x46 (77eb824e)
77e989d6 8b4510 mov eax,[ebp+0x10] ss:06629f32=????????
77e989d9 83f80f cmp eax,0xf
This issue is fixed in these Cisco CallManager versions:
-
3.2(2c)ES47
-
3.3(2)SpB
Cisco Bug ID CSCea45057
Cisco bug ID CSCea45057 (registered customers only) involves a Cisco CallManager restart on an unexpected H.225 signal.
This shows the fault in the Dr. Watson log:
00b7d363 8b45fc mov eax,[ebp+0xfc] ss:06d8839e=????????
00b7d366 8b4d08 mov ecx,[ebp+0x8] ss:06d8839e=????????
FAULT ->00b7d369 894810 mov [eax+0x10],ecx ds:0081d5d2=208d8b52
00b7d36c 8be5 mov esp,ebp
00b7d36e 5d pop ebp
This issue is fixed in these Cisco CallManager versions:
-
3.2(2c)ES66
-
3.2(3)ES01
-
3.3(2)SpC
Cisco Bug ID CSCdz25416
Cisco bug ID CSCdz25416 (registered customers only) involves a Cisco CallManager crash because the internal tables are not cleaned properly.
This shows the fault in the Dr. Watson log:
00b598b6 8b45fc mov eax,[ebp+0xfc] ss:0576ca9e=00000000
00b598b9 8b4d08 mov ecx,[ebp+0x8] ss:0576ca9e=00000000
FAULT ->00b598bc 8b5004 mov edx,[eax+0x4] ds:0081d5d6=fe808d8d
00b598bf 3b5104 cmp edx,[ecx+0x4] ds:0576cb12=00000000
00b598c2 753f jnz 00b62403
This issue is fixed in these Cisco CallManager versions:
-
3.1(4b)SpD
-
3.2(2c)SpH
-
3.3(2)
Cisco Bug ID CSCea52097
Cisco bug ID CSCea52097 (registered customers only) involves a Cisco CallManager crash that occurs when the Unexpected field in the Gatekeeper disengages.
This shows the fault in the Dr. Watson log:
00b53dd7 b916000000 mov ecx,0x16
00b53ddc 8d7530 lea esi,[ebp+0x30] ss:0656bece=????????
FAULT ->00b53ddf f3a5 rep movsd ds:05d4e92c=00000008 es:00000010=????????
00b53de1 8b8d88000000 mov ecx,[ebp+0x88] ss:05d4e984=00000002
00b53de7 51 push ecx
This issue is fixed in these Cisco CallManager versions:
-
3.2(2c)ES67
-
3.3(2)SpC
Cisco Bug ID CSCdy19452
Cisco bug ID CSCdy19452 (registered customers only) involves a Cisco CallManager restart due to an array exception in StationOutputSetRinger.
This shows the fault in the Dr. Watson log:
77e989ca 50 push eax
77e989cb ff156414e877 call dword ptr [77e81464] ds:77e81464=77fb0f18
FAULT ->77e989d1 e978f80100 jmp SetThreadContext+0x46 (77eb824e)
77e989d6 8b4510 mov eax,[ebp+0x10] ss:0576bfba=????????
77e989d9 83f80f cmp eax,0xf
This issue is fixed in these Cisco CallManager versions:
-
3.1(4b)SpA
-
3.2(2c)SpC
-
3.3(2)
Cisco Bug ID CSCtg41510
A Cisco Unified Communications Manager server can crash due to kernel panic. This error isobserved on the console.
<0>Fatal exception: panic in 5 seconds
This problem can affect CUCM version 7.1.3 and CUCM version 8.0.
Try these workarounds:
-
Disable the Fixed MOH Audio Source. This allows the IPVMS services to operate, but of course fixed MOH is not selectable as an audio source.
-
Plug in USB MOH devices to each server in the cluster, which has Fixed MOH Audio Source enabled.
-
Turn off the MOH run flag for the MOH servers that do not have the fixed USB MOH device. This allows the other IPVMS services such as MTP, CFB and ANN to run as desired whereas MOH only runs on the server with the fixed USB MOH device.
New Crash
If a crash is encountered and it does not match any of the previously described faults, open a case with Cisco Technical Support with the TAC Service Request Tool (registered customers only) Be sure to provide this information:
-
Cisco CallManager traces from 15 minutes before and after the crash.
You can find these traces in C:\Program Files\cisco\trace\ccm.
-
Signal distribution layer (SDL) traces from 15 minutes before and after the crash.
You can find these traces in C:\Program Files\cisco\trace\sdl\ccm.
-
System and Application Event log files.
You can find these at Start > Programs > Administrative Tools > Event Viewer.
-
The Dr. Watson log.
You can find this log at C:\Documents and Settings\All Users\Documents\DrWatson\Drwtsn32.log.
-
The User.dmp file.
You can find this file in C:\Documents and Settings\All Users\Documents\DrWatson.
Cisco Support Community - Featured Conversations
Related Information
- Voice Technology Support
- Voice and Unified Communications Product Support
- Troubleshooting Cisco IP Telephony
- Technical Support & Documentation - Cisco Systems
| Updated: Sep 30, 2011 | Document ID: 44366 |