Troubleshooting Guide for Cisco CallManager, Release 4.1(3)
Troubleshooting Tools

Table Of Contents

Troubleshooting Tools

Sniffer Traces

Debugs

Cisco CallManager Troubleshooting Tools

Cisco Secure Telnet

Command Line Tools

Show Command

Cisco CallManager System Performance Monitoring

Path Analysis Operation

System Log Management Process

Simple Network Management Protocol Support

CiscoWorks2000

Cisco Discovery Protocol Support

SQL Query Analyzer

Troubleshooting Tips

Where to Find More Information


Troubleshooting Tools


This section addresses the tools and utilities that you use to configure, monitor, and troubleshoot Cisco CallManager 4.0 and provides general guidelines for collecting information to avoid repetitive testing and recollection of identical data.


Note To access some of the URL sites listed this document, you must be a registered user and you must be logged in.


This section contains the following topics:

Sniffer Traces

Debugs

Cisco CallManager Troubleshooting Tools

Troubleshooting Tips

Where to Find More Information

Sniffer Traces

Typically, you collect sniffer traces by connecting a laptop or other sniffer-equipped device on a Catalyst port that is configured to span the VLAN or port(s) (CatOS, Cat6K-IOS, XL-IOS) that contains the trouble information. If no free port is available, connect the sniffer-equipped device on a hub that is inserted between the switch and the device.


Tip To help facilitate reading and interpreting of the traces by the TAC engineer, Cisco recommends using Sniffer Pro software because it is widely used within the TAC.


Have available the IP/MAC addresses of all equipment that is involved, such as IP phones, gateways, Cisco CallManagers, and so on.

Debugs

The output from debug privileged EXEC commands provides diagnostic information about a variety of internetworking events relating to protocol status and network activity in general.

Set up your terminal emulator software (such as HyperTerminal), so it can capture the debug output to a file. In HyperTerminal, click Transfer; then, click Capture Text and choose the appropriate options.

Before running any IOS voice gateway debugs, make sure that service timestamps debug datetime msec is globally configured on the gateway.


Note Avoid collecting debugs in a live environment during operation hours.


Preferably, collect debugs during non-working hours. If debugs must be collected in a live environment, configure no logging console and logging buffered. To collect the debugs, use show log.

Some debugs can be lengthy, so collect them directly on the console port (default logging console) or on the buffer (logging buffer). Collecting debugs over a Telnet session may have an impact on the device performance, and the result could be incomplete debugs, which requires that you re-collect them.

To stop a debug, use the no debug all or undebug all commands. Verify that the debugs have been turned off by using the command show debug.

Cisco CallManager Troubleshooting Tools

Refer to the Cisco CallManager Serviceability Administration Guide and the Cisco CallManager Serviceability System Guide for detailed information of the following different types of tools that Cisco CallManager Serviceability provides to monitor and analyze the various Cisco CallManager systems.

Table 2-1 Serviceability Tools 

Term
Definition

Real-Time Monitoring Tool

This term identifies a program in Serviceability that provides real-time information about Cisco CallManager devices and performance counters.

Alarm

Administrators use alarms to obtain run-time status and state of the Cisco CallManager system. Alarms contain information about system problems such as explanation and recommended action.

Alarm Catalog

This term refers to a file containing all the Alarm definitions for Cisco CallManager services. Serviceability supports multiple alarm catalogs that are specific to the alarm type.

Alarm Definition

Administrators search the alarm definitions database for alarm information. The alarm definition contains a description of the alarm and a recommended action.

Alarm Event Levels

Administrators determine the level of information that an alarm will contain. Levels range from general information about the system to information for debugging purposes only.

Alarm Filters

Administrators determine the level of information an alarm contains and where the alarm information gets saved.

Alarm Monitors

Cisco CallManager Serviceability allows alarms to be sent to different destinations called monitors: Windows 2000 Event Viewer, CCM trace, SDL trace, SNMP trap, and SysLog.

Alert Notify

Administrators configure alert notifications for performance counters and gateway ports/channels by using the Real-Time Monitoring Tool. Real-time monitoring sends alerts to the administrator by e-mail or in a system notification (popup) window.

Category Tabs

Administrators configure specific monitoring windows in real-time monitoring for troubleshooting purposes. The administrator creates these specific windows by using Category tabs.

Chart View

The Performance Monitoring Window displays performance counters in chart view by default. Chart view graphically shows the counter information.

Cisco CallManager service

Cisco CallManager supports many services in the form of software that performs a specific function, such as TFTP, CTI, or music on hold.

Control Center

Control Center tool in Serviceability allows administrators to view the status of and to start and stop Cisco CallManager services.

Debug Trace Levels

Administrators determine the level of information that a trace will contain. Levels range from general errors to detailed errors for debugging purposes only.

Device Monitoring

Real-time monitoring displays real-time information about Cisco CallManager devices such as phones and gateways.

Device Monitoring Window

The right side of the Real-Time Monitoring Tool window displays device performance information when the tool is monitoring device performance.

Device Name Based Trace Monitoring

Administrators obtain trace information about selected devices by configuring trace parameters for Cisco CallManager and Cisco CTIManager services.

Monitoring Objects Window

The left side of the Real-Time Monitoring Tool window displays Cisco CallManager-related objects and counters or devices for a cluster. The information that displays depends on which tab is active in the window.

Objects and counters

Windows 2000 provides performance data that contains information about various objects and counters. Objects are the logical groupings of like counters for a specific device or feature, such as Cisco IP Phones or Cisco CallManager System Performance. Counters measure various aspects of system performance. Counters measure statistics such as the number of registered phones, calls attempted, and calls in progress. The Real-Time Monitoring Tool monitors the real-time statistics generated by these counters.

Performance Monitoring

The Real-Time Monitoring Tool displays real-time information about a performance counter. Performance counters can be system specific or Cisco CallManager specific.

Performance Monitoring Window

The right side of the Real-Time Monitoring Tool window displays counter statistics when the tool is monitoring counters.

CCM Trace log file (formerly SDI Trace)

Every Cisco CallManager service includes a default trace log file. The system traces system diagnostic interface (SDI) information from the services and logs run-time events and traces to a log file.

SDL Trace log file

This file contains call-processing information from services such as Cisco CallManager and Cisco CTIManager. The system traces the signal distribution layer (SDL) of the call and logs state transitions into a log file.


Note In most cases, you will only gather SDL traces when Cisco Technical Assistance Center (TAC) requests it of you.


Service status icons

Control Center displays three icons that represent the status of a service on a server:

Square represents a stopped service.

Arrow represents a service that is running.

Question mark represents a service that is in an unknown state.

Trace

Administrators and Cisco engineers use trace files to obtain specific information about Cisco CallManager service problems.

Trace Analysis

This program presents trace information in a format that allows you to filter the results.

Trace log file

Cisco CallManager Serviceability sends configured trace information to this file. Two types of trace log files exist: CCM and SDL.

Window Status Bar

The bottom, right corner of the Real-Time Monitoring Tool window displays the window status bar. The status bar displays five icons: Preferences, Cluster Information, Resource Usage, About, and Help.

Quality Report Tool

This term designates voice quality and general problem-reporting utility in Cisco CallManager Serviceability.


Cisco Secure Telnet

Cisco Secure Telnet allows Cisco Service Engineers (CSE) transparent firewall access to the Cisco CallManager node on your site. Using strong encryption, Cisco Secure Telnet enables a special Telnet client from Cisco Systems to connect to a Telnet daemon behind your firewall. This secure connection allows remote monitoring and troubleshooting of your Cisco CallManager nodes, without requiring firewall modifications.


Note Cisco provides this service only with your permission. You must ensure that a network administrator is available at your site to help initiate the process.


Command Line Tools

Command Line Tools prove useful in troubleshooting. The following list gives the available command line tools:

showDisplays the Cisco CallManager database content, the .ini config file, memory statistics, and Windows diagnostic information and runs from a DOS shell or from a Telnet session into the Cisco CallManager.

nslookup hostnameChecks for a host-name-to-IP-address resolution.

netstat - a | moreChecks for socket listens on the correct port number.

ping hostnameChecks that the machine can be reached via an IP.

net startChecks whether services are running.

Show Command

Use the Show command line tool to display the contents of the system memory statistics and the Windows diagnostic information. You can run the Show command from a DOS shell or from a Telnet session if Telnet server software is enabled. You can display the output data on the console or save it as a text file.


Note Because the show command uses a temporary file in the \Temp directory for the output, check to ensure that you have enough disk space available to receive it. The amount that you will need varies depending on a number of factors; for example, the number of users and devices being used and the size of the database being used by the system.


Alternatively, you can run show.exe from a Telnet session if Telnet server software is enabled.

The following syntax applies for the show command:

show [-f <filename>] [-c <column width>] [-w <console width>] [-v] [command]

Table 2-2 lists options that the show command supports.

Table 2-2 Show Command Options 

Command
Description

-f <filename>

Name of file to print the report

-c <col width>

Width of each column in the database report (default 15)

-w <con width>

Width of the database report area (default 80)

-v

Verbose mode


Use the following parameters with the show command:

?—Show help message.

db—Show configuration database.

db tables—Show database table names.

db t <tablename>—Show content of the database table.

inst [apps | elem | all]Show information about installed applications and elements.

isdn [cluster | local | specific]—Show D channel status on gateway.

ps—Show all processes running on the local system.

win—Report windows diagnostics. The win parameter includes, but is not limited to, information such as system statistics, storage information, software environment, and summary statistics.


Note Show win consumes a significant part of CPU resources to get the windows system information and takes a long time to display. Execute it only when Cisco CallManager is not busy.


tech | (none)—Report database and Windows system information.


Note Show tech reports the same multireport output as show command without a parameter


Example:

	show -f output.txt -v -w480 db
	show tech
	show db t ProcessNode

Refer to the Cisco CallManager Serviceability Administration Guide for more information on the show command.

Cisco CallManager System Performance Monitoring

Use Windows 2000 Performance to collect and display system and device statistics for any local or remote Cisco CallManager installation. This administrative tool allows you to gain a full understanding of a system without studying the operation of each of its components.It reports both general and specific information in real time.

After adding the Cisco CallManager parameters, you can define the terms under which Cisco CallManager will display statistics generated by the system.

For more information about Performance, refer to the Microsoft Windows 2000 documentation.

Path Analysis Operation

Path Analysis, a diagnostic application, traces connectivity between two specified points on a network. It analyzes both physical and logical paths (Layer 2 and Layer 3) that are taken by packets that flow between those points.

After a call completes, PathTool traces the route of audio packets by specifying the directory number of the calling and called parties. This applies to calls among any of the following endpoints: Cisco IP Phones, analog devices that are connected to a station gateway, or trunk gateways (analog or digital).

Refer to the Cisco CallManager Serviceability Administration Guide for more information.

System Log Management Process

Although it can be adapted to other network management systems, Cisco Syslog Analysis, which is packaged with CiscoWorks2000 Resource Manager Essentials, provides the best method to manage Syslog messages from Cisco devices.

Cisco Syslog Analyzer serves as the component of Cisco Syslog Analysis that provides a common storage and analysis of the system log for multiple applications. The other major component, Syslog Analyzer Collector, gathers log messages from Cisco CallManager servers.

These two Cisco applications work together to provide a centralized system logging service for Cisco IP Telephony Solutions.

Refer to the Cisco CallManager Serviceability Administration Guide for more information.

Simple Network Management Protocol Support

Network management systems (NMS) use SNMP, an industry-standard interface, to exchange management information between network devices. A part of the TCP/IP protocol suite, SNMP enables administrators to remotely manage network performance, find and solve network problems, and plan for network growth.

An SNMP-managed network comprises three key components: managed devices, agents, and network management systems.

A managed device designates a network node that contains an SNMP agent and resides on a managed network. Managed devices collect and store management information and make it available by using SNMP.

An agent, as network management software, resides on a managed device. An agent contains local knowledge of management information and translates it into a form that is compatible with SNMP.

A network management system comprises an SNMP management application together with the computer on which it runs. An NMS executes applications that monitor and control managed devices. An NMS provides the bulk of the processing and memory resources that are required for network management. The following NMSs share compatibility with Cisco CallManager:

CiscoWorks2000

HP OpenView

Third-party applications that support SNMP and Cisco CallManager SNMP interfaces

For detailed information, refer to the Cisco CallManager Serviceability Administration Guide and the Cisco CallManager Serviceability System Guide.

CiscoWorks2000

CiscoWorks2000 serves as the network management system of choice for all Cisco devices including Cisco CallManager. Because CiscoWorks2000 is not bundled with Cisco CallManager, you must purchase it separately. Use the following tools with CiscoWorks2000 for remote serviceability:

System Log

Path Analysis

Cisco Discovery Protocol

Simple Network Management Protocol

Refer to the Cisco CallManager Serviceability Administration Guide and the CiscoWorks2000 documentation for more information on CiscoWorks2000 at the following URL:

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/index.htm

Cisco Discovery Protocol Support

The Cisco Discovery Protocol Support enables discovery of Cisco CallManager servers and management of those servers by CiscoWorks2000.

Refer to the Cisco CallManager Serviceability Administration Guide and the CiscoWorks2000 documentation for more information on CiscoWorks2000 at the following URL:

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/index.htm

SQL Query Analyzer

The SQL Query Analyzer to find devices, such as endpoints, IP phones, and gateways, that are associated with a location. To use the SQL Query, perform the following steps.


Note If the value None appears in the Location field in the Cisco CallManager Configuration window, the device has not been assigned to a specific location. Therefore, devices not assigned to a location do not get returned by the SQL Query.


Procedure


Step 1 Choose Start > Programs > Microsoft SQL Server > Query Analyzer on the Windows 2000 server console of the Cisco CallManager server to run the SQL Server Query Analyzer application.

The Connect to SQL Server window appears. (The SQL Query Analyzer window is dimmed in the background.)

Step 2 In the SQL Server field, type a period character.

Step 3 Uncheck the Start SQL Server if it is Stopped option.

Step 4 Click the Windows Authentication button.

Step 5 Click OK.

Step 6 The Query (local) window appears. (The SQL Query Analyzer window gets dimmed in the background.)

Step 7 From the DB field, click the drop-down arrow and choose the highest numbered Cisco CallManager database.

Cisco CallManager databases are labeled with the following format: CCM03xx (where xx is the number of the database).

Step 8 Enter the following SQL query in the body of the Query (local) window.

SELECT Device.name, Device.description

FROM Device, Location

WHERE Device.fkLocation=Location.pkid

AND Location.name="enter location name between these quotes"

Step 9 Choose Query > Execute from the main Query Analyzer window to run the query.

You can also click the green arrow from the toolbar or press F5 to run the query.

Step 10 When the results are done, close the SQL Server Query Analyzer window.


Troubleshooting Tips

The following tips may help you when troubleshooting the Cisco CallManager.


Tip Check the release notes for Cisco CallManager for known problems.


The release notes provide descriptions and workaround solutions for known problems.


Tip Know where your devices are registered.


Each Cisco CallManager log traces files locally. If a phone or gateway is registered to a particular Cisco CallManager, then the call processing gets done on that Cisco CallManager if the call is initiated there. You will need to capture traces on that Cisco CallManager to debug a problem.

A common mistake involves having devices registered on a subscriber server, but capturing traces on the publisher server. These trace files will be nearly empty (and most definitely will not have the call in them).

Another common problem involves having Device 1 registered to CM1 and Device 2 registered to CM2. If Device 1 calls Device 2, the call trace occurs in CM1 and if Device 2 calls Device 1 the trace occurs in CM2. If you are troubleshooting a two-way calling issue, you need both traces from both Cisco CallManagers to obtain all the information needed to troubleshoot.


Tip Know the approximate time of the problem.


Multiple calls may have been made, so knowing the approximate time of the call helps TAC quickly locate the trouble.

You can obtain phone statistics on a Cisco IP Phone 79xx by pressing the i button twice during an active call.

When you are running a test to reproduce the issue and produce information, know the following data that is crucial to understanding the issue:

Calling number/called number

Any other number that is involved in the specific scenario

Time of call


Note Remember that time synchronization of all equipment is important for troubleshooting.


If you are reproducing a problem, make sure to choose the file for the timeframe by looking at the modification date and the timestamps in the file. The best way to collect the right trace is to reproduce a problem and then quickly locate the most recent file and copy it from the Cisco CallManager server.


Tip Save the log files to prevent them from being overwritten.


Files will get overwritten after some time. The only way to know which file is being logged to is to choose View >Refresh on the menu bar and look at the dates and times on the files.


Tip Verify that the Cisco CallManager services are running.


Use the following procedure to verify that the Cisco CallManager service is active on a server.

Procedure


Step 1 From Cisco CallManager Administration, choose Application > Cisco CallManager Serviceability.

The Cisco CallManager Serviceability window displays.

Step 2 Choose Tools > Service Activation as shown in Figure 2-1.

Figure 2-1 Cisco CallManager Serviceability Window Tools Menu

Step 3 From the Servers column, choose the server.

The server that you chose displays next to the Current Server title, and a box with configured services displays.

Activation Status column displays either Activated or Deactivated in the Cisco CallManager line as shown in Figure 2-2.

Figure 2-2 Service Activation Window

If the Activated status displays, the Cisco CallManager is active on the chosen server.

If the Deactivated status displays, continue with the following steps.

Step 4 Check the check box for Cisco CallManager.

Step 5 Click the Update button.

The Activation Status column displays Activated in the Cisco CallManager line.

Cisco CallManager is now active for the chosen server.


Perform the following procedure if the Cisco CallManager has been in service and you want to verify if it is currently active.

Procedure


Step 1 From Cisco CallManager Administration, choose Application > Cisco CallManager Serviceability.

The Cisco CallManager Serviceability window displays.

Step 2 Choose Tools > Control Center.

Step 3 From the Servers column, choose the server.

The server that you chose displays next to the Current Server title, and a box with configured services displays.

Activation Status column displays Activated in the CallManager line.

Cisco CallManager is active for the chosen server.



Tip Start and stop the Internet Information Server.


Use any of the following procedures to start or stop the Internet Information Server (IIS).

Procedure


Step 1 From the Start menu, choose Start > Programs > Administration Tools > Services.

A window displays listing the services.

To Stop Services

Step 2 Choose IIS Admin Service.

Step 3 Click the stop button (black square box at the top of the window).

Step 4 Click Yes.

To start Services

Step 5 Click the Start button.

Step 6 Choose World Wide Web Publishing.

Step 7 Click the start button (black square box with right arrow at the top of the window).

The IIS starts.


Procedure


Step 1 From the Start menu, choose Start > Programs > Administration Tools > Services.

A window displays listing the services.

To Stop Services

Step 2 Right-click IIS Admin Service.

Step 3 Choose Stop.

The IIS stops.

To start Services

Step 4 Click the Start button.

Step 5 Right-click World Wide Web Publishing.

Step 6 Choose Start.

The IIS starts.


Procedure


Step 1 From the Start menu, choose Start > Programs > Administration Tools > Services.

A window displays listing IIS Administration Service.

Step 2 Right-click IIS Admin Service and choose Stop.

The IIS stops.

Step 3 To start the IIS server, right-click IIS Admin Service and choose Start.

The IIS starts.


Where to Find More Information

Additional Cisco Documentation

Cisco CallManager Serviceability Administration Guide

Cisco CallManager Serviceability System Guide

Cisco CallManager Administration Guide

Installation Guide for Cisco CallManager

CiscoWorks2000 user documentation at the following URL:

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/index.htm