Guest

Cisco Unified Communications Manager (CallManager)

Troubleshooting Cisco CallManager Crashes

Document ID: 44366

Updated: Jan 31, 2013

   Print

Introduction

This document provides information about a Cisco CallManager crash and how to identify the known bugs.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software or hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Cisco CallManager Crash Description

When the Cisco CallManager service crashes, this message appears in the System Event log:

The Cisco CallManager service terminated unexpectedly.
It has done this 1 time. The following corrective action
will be taken in 60000 ms. Restart the service.

At this time, devices such as Cisco IP phones and gateways that are registered to the Cisco CallManager are unregistered. The Cisco CallManager service can crash due to one of these reasons:

  • An unexpected event occurs in the Cisco CallManager service. This crash generates a Dr.Watson log and a User.dmp file in the C:\Documents and Settings\All Users\Documents\DrWatson folder.

  • The Cisco CallManager service does not have enough resources, such as CPU or memory, to function. Generally the CPU utilization in the server is at 100 percent, at that time.

This document discusses only those situations in which the crash occurs due to an unexpected event.

Reading the Dr.Watson Log

Whenever there is an application crash, the Dr.Watson log is appended. Open the Dr. Watson log in Notepad, scroll to the bottom of the file, and search for Application exception occurred. This shows the latest crash:

Application exception occurred:
        App:  (pid=680)
        When: 3/8/2003 @ 14:01:06.978
        Exception number: e06d7363

Compare the date and time with the Event log message in order to ensure that the crash mentioned has the same time. The previous example output indicates that the application that crashed had a process identifier (PID) of 680. This trace lists all of the of the PIDs:

PID  PROCESS
   8 System.exe
 212 SMSS.exe
 240 CSRSS.exe
 264 WINLOGON.exe
 292 SERVICES.exe
 304 LSASS.exe
 424 termsrv.exe
 520 svchost.exe
 560 msdtc.exe
 696 DLLHOST.exe
 736 Ipvmsapp.exe
 752 DLLHOST.exe
 824 AudioTranslator.exe
 848 RisDC.exe
 860 LogoutService.E.exe
 884 DCX500.exe
 936 svchost.exe
 980 LLSSRV.exe
1028 sqlservr.exe
1112 ntpd.exe
1140 rcmdsvc.exe
1172 regsvc.exe
1176 mstask.exe
1204 SNMP.exe
1244 WinMgmt.exe
1260 cpqnimgt.exe
1284 cqmgserv.exe
1296 cqmgstor.exe
1308 sysdown.exe
1372 cqmghost.exe
1524 aupair.exe
1552 sqlagent.exe
 276 svchost.exe
2400 inetinfo.exe
2412 explorer.exe
2752 sqlmangr.exe
2700 taskmgr.exe
2704 mmc.exe
 680 ccm.exe
 868 DRWTSN32.exe

The PID (680) is ccm.exe, which is the Cisco CallManager service. After you verify the date and time in the Event Viewer and confirm that the crash is caused by ccm.exe, search for the word FAULT in the Dr.Watson log. That shows the location that actually caused the crash:

function: RaiseException
        77eab2d4 85c9          test    ecx,ecx
        77eab2d6 740e          jz      GetVolumePathNameA+0x7e (77eb3fe6)
        77eab2d8 8d4801        lea     ecx,[eax+0x1]         ds:0751c41a=????????
        77eab2db 8d7dc4        lea     edi,[ebp+0xc4]        ss:0751c46a=????????
        77eab2de f3a5          rep  movsd ds:06cfeed8=06cfeef4 es:06cfee68=00000000
        77eab2e0 eb04          jmp     SetVolumeMountPointA+0x172 (77eb35e6)
        77eab2e2 8365c000      and  dword ptr [ebp+0xc0],0x0 ss:0751c46a=????????
        77eab2e6 8d45b0        lea     eax,[ebp+0xb0]        ss:0751c46a=????????
        77eab2e9 50            push    eax
        77eab2ea ff156414e877  call    dword ptr [77e81464]  ds:77e81464=77fb1130
FAULT ->77eab2f0 5f            pop     edi
        77eab2f1 5e            pop     esi
        77eab2f2 c9            leave
        77eab2f3 c21000        ret     0x10

The FAULT is unique for different types of crashes. The first column is the memory location, which can vary. In this example, the fault is in 77eab2f0. However, the rest of the line, 5f pop edi, is always the same for this type of crash.

Cisco CallManager Publisher Server Cannot Start Services :DBL errors

The Cisco CallManager publisher server cannot start services since the database cannot be accessed. The Database Layer Monitor service also cannot access the database.

DBL Error Solution

The Database Layer Monitor accesses the DB through a series of DLL files. In order to resolve this problem, unregister and then re-register the database access DLLs from the Microsoft Windows operating system. This allows the core applications to make database calls again through the Cisco provided DLLs.

Cisco CallManager Services Do not Start after a Power Outage

Cisco CallManager services sometimes do not start after a server reboot or power outage when there are two Network Interface Cards (NICs) enabled and therefore two IP addresses assigned. Ensure you only have one NIC enabled on the server at a time. Dual NICs are not supported. The recommendation is to have two NICs and use one as fault tolerance, but only one is operational at a time. Failure to disable the second NIC can result in two IP addresses that are assigned to the Cisco CallManager server. When two IP addresses are assigned to the Cisco CallManager server, it can cause a loss of service. You must have only one NIC enabled (the one which is configured). Disable the one which is not used in order to resolve the issue.

List of Known Crashes and Fixes

This section lists known crashes, with FAULT codes and available fixes. If a fix is available in an Engineering Special (ES), open a case with Cisco Technical Support with the TAC Service Request Tool (registered customers only) in order to obtain a patch.

Cisco Bug ID CSCdx42096

Cisco bug ID CSCdx42096 (registered customers only) involves a Cisco CallManager crash due to badly formatted Media Gateway Control Protocol (MGCP) messages from MGCP gateways.

This shows the fault in the Dr.Watson log:

77eab2e9 50               push   eax
        77eab2ea ff156414e877    call    dword ptr [77e81464] ds:77e81464=77fb1130
FAULT ->77eab2f0 5f              pop     edi
        77eab2f1 5e              pop     esi
        77eab2f2 c9              leave

This issue is fixed in these Cisco CallManager versions:

  • 3.3(2)SpC

  • 3.2(2c)ES64

Cisco Bug ID CSCdx32456

Cisco bug ID CSCdx32456 (registered customers only) involves Cisco CallManager crashes while an H.323 call is processed.

This shows the four possible faults in the Dr.Watson log that can cause the crash:

FAULT ->005783e7 f3a5
FAULT ->005777ea 8b00
FAULT ->0057784a 8b00
FAULT ->005790c7 8b5004

This issue is fixed in these Cisco CallManager versions:

  • 3.2(2c)

  • 3.3(2)

Cisco Bug ID CSCdz69051

Cisco bug ID CSCdz69051 (registered customers only) involves a Cisco CallManager crash because the array is out of bounds.

This shows the fault in the Dr. Watson log:

77e989ca 50               push  eax
        77e989cb ff156414e877   call    dword ptr [77e81464]   ds:77e81464=77fb0f18
FAULT ->77e989d1 e978f80100     jmp     SetThreadContext+0x46 (77eb824e)
        77e989d6 8b4510         mov     eax,[ebp+0x10]         ss:06629f32=????????
        77e989d9 83f80f         cmp     eax,0xf

This issue is fixed in these Cisco CallManager versions:

  • 3.2(2c)ES47

  • 3.3(2)SpB

Cisco Bug ID CSCea45057

Cisco bug ID CSCea45057 (registered customers only) involves a Cisco CallManager restart on an unexpected H.225 signal.

This shows the fault in the Dr. Watson log:

00b7d363 8b45fc           mov   eax,[ebp+0xfc]                 ss:06d8839e=????????
        00b7d366 8b4d08         mov     ecx,[ebp+0x8]          ss:06d8839e=????????
FAULT ->00b7d369 894810         mov     [eax+0x10],ecx         ds:0081d5d2=208d8b52
        00b7d36c 8be5           mov     esp,ebp
        00b7d36e 5d             pop     ebp

This issue is fixed in these Cisco CallManager versions:

  • 3.2(2c)ES66

  • 3.2(3)ES01

  • 3.3(2)SpC

Cisco Bug ID CSCdz25416

Cisco bug ID CSCdz25416 (registered customers only) involves a Cisco CallManager crash because the internal tables are not cleaned properly.

This shows the fault in the Dr. Watson log:

00b598b6 8b45fc           mov   eax,[ebp+0xfc]                 ss:0576ca9e=00000000
        00b598b9 8b4d08         mov     ecx,[ebp+0x8]          ss:0576ca9e=00000000
FAULT ->00b598bc 8b5004         mov     edx,[eax+0x4]          ds:0081d5d6=fe808d8d
        00b598bf 3b5104         cmp     edx,[ecx+0x4]          ds:0576cb12=00000000
        00b598c2 753f           jnz     00b62403

This issue is fixed in these Cisco CallManager versions:

  • 3.1(4b)SpD

  • 3.2(2c)SpH

  • 3.3(2)

Cisco Bug ID CSCea52097

Cisco bug ID CSCea52097 (registered customers only) involves a Cisco CallManager crash that occurs when the Unexpected field in the Gatekeeper disengages.

This shows the fault in the Dr. Watson log:

00b53dd7 b916000000       mov   ecx,0x16
        00b53ddc 8d7530         lea     esi,[ebp+0x30]         ss:0656bece=????????
FAULT ->00b53ddf f3a5           rep movsd ds:05d4e92c=00000008 es:00000010=????????
        00b53de1 8b8d88000000   mov     ecx,[ebp+0x88]         ss:05d4e984=00000002
        00b53de7 51             push    ecx

This issue is fixed in these Cisco CallManager versions:

  • 3.2(2c)ES67

  • 3.3(2)SpC

Cisco Bug ID CSCdy19452

Cisco bug ID CSCdy19452 (registered customers only) involves a Cisco CallManager restart due to an array exception in StationOutputSetRinger.

This shows the fault in the Dr. Watson log:

77e989ca 50               push  eax
        77e989cb ff156414e877   call    dword ptr [77e81464]   ds:77e81464=77fb0f18
FAULT ->77e989d1 e978f80100     jmp     SetThreadContext+0x46 (77eb824e)
        77e989d6 8b4510         mov     eax,[ebp+0x10]         ss:0576bfba=????????
        77e989d9 83f80f         cmp     eax,0xf

This issue is fixed in these Cisco CallManager versions:

  • 3.1(4b)SpA

  • 3.2(2c)SpC

  • 3.3(2)

Cisco Bug ID CSCtg41510

A Cisco Unified Communications Manager server can crash due to kernel panic. This error is observed on the console.

<0>Fatal exception: panic in 5 seconds

This problem can affect CUCM version 7.1.3 and CUCM version 8.0.

Try these workarounds:

  • Disable the Fixed MOH Audio Source. This allows the IPVMS services to operate, but of course fixed MOH is not selectable as an audio source.

  • Plug in USB MOH devices to each server in the cluster, which has Fixed MOH Audio Source enabled.

  • Turn off the MOH run flag for the MOH servers that do not have the fixed USB MOH device. This allows the other IPVMS services such as MTP, CFB and ANN to run as desired whereas MOH only runs on the server with the fixed USB MOH device.

Cisco Bug ID CSCts29293

HuntListCdrc code enters an infinite loop which leads to SDL Router thread failure and eventual CCM core.

This line might be printed in the trace file for some period that leads up to the core:

12:29:49.199 |HuntListCdrc::SendCcNotifyReq with 
   transactioId=84180720|5,100,49,1.130009640

Note:  The transactioId does not increase as it leads to the infinite loop state.

If the server runs on a UCS Platform, disable the LRO and update VMware tools. However, the problem has been observed on CUCM systems that have LRO disabled. Hence, no confirmed workaround is available.

On the MCS platform, there is no workaround.

New Crash

If a crash is encountered and it does not match any of the previously described faults, open a case with Cisco Technical Support with the TAC Service Request Tool (registered customers only) Be sure to provide this information:

  1. Cisco CallManager traces from 15 minutes before and after the crash.

    You can find these traces in C:\Program Files\cisco\trace\ccm.

  2. Signal distribution layer (SDL) traces from 15 minutes before and after the crash.

    You can find these traces in C:\Program Files\cisco\trace\sdl\ccm.

  3. System and Application Event log files.

    You can find these at Start > Programs > Administrative Tools > Event Viewer.

  4. The Dr. Watson log.

    You can find this log at C:\Documents and Settings\All Users\Documents\DrWatson\Drwtsn32.log.

  5. The User.dmp file.

    You can find this file in C:\Documents and Settings\All Users\Documents\DrWatson.

Related Information

Updated: Jan 31, 2013
Document ID: 44366