Guest

Cisco Unity

Troubleshooting Cisco Unity System Crashes

Cisco - Troubleshooting Cisco Unity System Crashes

Document ID: 22382

Updated: Jan 15, 2007

   Print

Introduction

This document provides thorough information-gathering and troubleshooting steps in order to identify the problem within a system that lands in one of these states:

  • Ports locking

  • System slows down

  • System crashes

The information described in this document is required for Engineering review. Failure to provide this information increases the amount of time to resolution.

Note: The information in this document is based on Cisco Unity for Exchange.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Note: Services that are enabled, but not in use, can cause additional logging to be generated on Cisco Unity systems. In general, if you see errors or logs for a process or feature that you do not use, verify that the service is not needed and disable it. If you have non-used services that run, this can also cause abnormal behavior on the Cisco Unity system.

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco Unity for Exchange 2.4(6)

  • Cisco Unity 3.x

  • Cisco Unity 4.0(x)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Identify the Problem

The System Experiences a System Crash

The system most likely experiences a system crash if any of these are true:

  • The system is unresponsive.

  • Restarting the Cisco Unity service does not resolve the problem.

  • One or all of the Cisco Unity services are no longer running.

  • The SA or Status Monitor cannot be accessed.

  • CPU utilization is 100% or near 100%.

Provide Basic Details

It is important that you provide Cisco Technical Assistance Center (TAC) with these basic details:

  • Version and build information for Cisco Unity.

  • Phone system type and version.

  • Type of Cisco Unity installation—voice mail or Unified messaging.

  • The number of mail users.

  • If Unified messaging, the number of mail users homed on each remote Exchange server.

  • The platform on which Cisco Unity is installed, along with voice boards if applicable.

  • Cisco Unity features that are enabled, such as fax, TTS, and so on.

  • Frequency of failures.

  • Version of TSP.

Preparing to Gather Data for the Next Failure

For Ports Locking

Refer to Troubleshooting Port Lockups in Cisco Unity if the problem is ports locking.

For System Crashes

The first time the system crashes, you can gather the application logs for initial investigation.

  1. Format the logs first, as they do not provide needed data when unformatted.

    1. From the Start menu, choose Programs > Administrative Tools > Event Viewer.

    2. From the Event Viewer window, highlight the log from the left pane, then choose Action > Save Log File As.

    3. From the Save System Log As window, in the Save as Type field, choose CSV (Comma delimited) (*.csv), then click Save.

      Note: Do not save the raw Event log data as a *.evt file.

  2. If the system is in a crashed state, but memory dumps are not installed, you can use the Microsoft Memory.dmp tool leavingcisco.com in order to gather data.

  3. Make sure your system runs, or is updated to, the latest Cisco Unity version [3.1(x) or 4.0(x)] in order to eliminate the risk of a known issue.

  4. In order to be prepared to gather data for another failure:

    1. Contact Cisco TAC to copy the memory dump utility to the system and collect user dumps the next time it crashes while in the crashed state and before you reboot.

    2. Include dumps for the AvCsMgr.exe service as well as all svchost.exe processes. (There will be multiple versions of the svchost.exe. Make sure you get all of them.)

    3. Include the Application event log.

    4. Enable system performance monitoring with the perfAnalysis.vbs script as described in the Monitoring Cisco Unity Performance white paper, and provide the generated logs (found by default under C:\PerfLogs).

System Crash Known Problems

These known problems exist in Cisco Unity software. Refer to the Bug Toolkit (registered customers only) in order to view the Cisco bug IDs mentioned in this section.

Speed Control

Cisco bug ID CSCdw88890 (registered customers only) exists in:

  • Cisco Unity 3.1(x)

The workaround for this issue is in:

  • Cisco Unity 3.1(3) and 3.1(2)c

Problem Description

The use of playback speed controls during message retrieval results in various symptoms, which depends on system configuration. Cisco Unity might drop the call immediately, disable the port after the call has ended, or not answer any calls to any of the Cisco Unity ports.

Workaround

Refer to the Cisco Unity Playback Speed Controls May Drop Callers or Lock Up All Unity Ports field notice.

RPC Hang

Cisco bug ID CSCdw22615 (registered customers only) exists in:

  • All versions of Cisco Unity.

Problem Description

Cisco Unity stops answering calls but appears to be running. Opening the System Administrator, Status Monitor, and other Cisco Unity tools does not work. Cisco Unity will actually answer a call on each port but it will be dead air. After it has done this for each port, it will not answer any more calls.

Symptoms

Several of these behaviors are exhibited:

  • The AvCsMgr process still runs.

  • Cisco Unity System Administration pages are not accessible.

  • Subscribers might receive silence after entering their passwords while logging on to their mail boxes by using the telephone user interface (TUI).

  • Callers might experience Ring-No-Answer when calling Cisco Unity.

Note: These behaviors are much like the SQL Server Deadlock, but without events in the Application event log. Before you apply the workaround, confirm that you do not see that defect. It is believed that this defect can occur on any version of Cisco Unity that uses Telephony Record and Playback (TRaP). The defect is related to a bug in the Windows 9x RPC layer. This affects Cisco Unity because Cisco Unity communicates with the Windows 9x clients during TRaP calls. Windows Me is also affected because it is part of the Windows 9x operating system line.

Workaround

Refer to the Unity Stops Responding Because of RPC Hang field notice. Microsoft has released a fix for this issue and it is described in MSFT KB article Q315575.

Idle Systems

Cisco bug ID CSCdw68215 (registered customers only) exists in:

  • Cisco Unity 3.x.

This issue is fixed in:

  • Cisco Unity 3.1(2)c and 3.1(3).

Problem Description

The Cisco Unity system stops responding after sitting idle for one week.

Symptom

Intermittently, on failover or idle systems in Cisco Unity 3.1 and 3.0, Cisco Unity stops responding on the primary and secondary servers. Typically, this occurs after failover systems have been sitting idle for approximately one week.

These errors appear multiple times in the application log:

Event Type: Error
Event Source: COM+
Event Category: Executive
Event ID: 4199
Date: 2/4/2002
Time: 4:36:57 PM
User: N/A
Computer: UPRIM1
Description:
The COM+ Services DLL (comsvcs.dll) was unable to 
load because allocation of thread local storage failed.

Process Name: AvCsMgr.exe
Error Code = 0x80070008 : Not enough storage is 
available to process this command.
COM+ Services Internals Information:
File: .\comsvcs.cpp, Line: 289

Event Type: Error
Event Source: AvRdbSvr_MC
Event Category: Startup
Event ID: 21000
Date: 2/4/2002
Time: 4:36:57 PM
User: N/A
Computer: UPRIM1
Description:
Last Category Failed connecting to underlying database using:
provider=SQLOLEDB;driver={SQL Server};Data Source=UPRIM1;
Initial Catalog=UnityDb;Integrated
Security=SSPI: 80004005

Event Type: Error
Event Source: AvCs_MC
Event Category: None
Event ID: 100
Date: 2/4/2002

Time: 4:36:57 PM

User: N/A
Computer: UPRIM1
Description:
Exception occurred and handled in File:
e:\views\amis3.1.1.13\un_Doh1\Dal\DalDb\AvRdbSvr\AvRdbRowsetImpl.h
at Line: 340 - Error: 80004005H Call stack:
0x5E014E7D AvRdbSvr.dll: <unknown symbol>
0xFEDCBA98 <unknown module>: : <unknown symbol>
0x89ABCDEF <unknown module>: : <unknown symbol>
0x77D5298F RPCRT4.DLL: RpcRevertToSelf + 858 bytes

Solution

Upgrade to Cisco Unity 3.1(2)c or later.

SQL Server Deadlock

Cisco bug ID CSCdw13964 (registered customers only) exists in:

  • Cisco Unity 3.x.

This issue is fixed in:

  • SQL Hot Fix (see the field notice in the Workaround section for details).

Problem Description

Cisco Unity Version 3.x stops processing calls due to a Microsoft SQL server lock.

Symptom

Several of these behaviors are exhibited when the Cisco Unity server experiences a Microsoft SQL server lock up:

  • The AvCsMgr process still runS.

  • Cisco Unity System Administration pages are not accessible.

  • Subscribers might receive silence after their passwords are entered while logging on to their mailboxes by using the telephone user interface (TUI).

  • Callers might experience Ring-No-Answer when calling Cisco Unity.

These symptoms are similar to the RPC Hang issue, but have specific entries in the event log, which are listed in Cisco Unity Unified Messaging Release 3.x Stops Processing Calls Due to Microsoft SQL Server Lock Up.

Workaround

Refer to the Cisco Unity Unified Messaging Release 3.x Stops Processing Calls Due to Microsoft SQL Server Lock Up field notice. You can also refer to the Microsoft support site leavingcisco.com.

Logging Memory Leak

Cisco bug ID CSCdv86261 (registered customers only) exists in:

  • Cisco Unity 3.x.

This issue is fixed in:

  • Cisco Unity 3.0(4) and 3.1(2).

Problem Description

The Unity server stops responding.

Symptom

When you try to access the system, this error appears:

Windows was unable to save all the data for the file 
\Device\Harddisk\Volume2\WINNT\system32\config\SysEvent.Evt. 
The data has been lost. This error may be by a failure of your 
computer hardware or network connection. Please try to save 
this file elsewhere.

This problem occurs because new log files are not opened each night. Therefore, paged pool and non-paged pool for all Cisco Unity processes appear to leak.

Solution

Upgrade to Cisco Unity 3.0(4) or 3.1(2).

Related Information

Updated: Jan 15, 2007
Document ID: 22382