Cisco Unified CallManager Serviceability System Guide, Release 5.0(4)
Real-Time Monitoring Tool
Downloads: This chapterpdf (PDF - 237.0KB) The complete bookPDF (PDF - 2.6MB) | Feedback

Real-Time Monitoring Tool

Table Of Contents

Real-Time Monitoring Tool

Understanding the Real-Time Monitoring Tool

RTMT Components

RTMT Services, Servlets and Service Parameters

RTMT Collector, Alert Manager, and RTMT Reporter

Viewing a Summary

Monitoring Server Status

Understanding Server Logs

Monitoring Call-Processing Activity

Understanding Call-Processing Logs

Monitoring Services

Understanding Service Logs

Monitoring Devices

Understanding Device Logs

Monitoring CTI Applications, Devices, and Lines

Where to Find More Information


Real-Time Monitoring Tool


This chapter contains information on the following topics:

Understanding the Real-Time Monitoring Tool

RTMT Components

RTMT Services, Servlets and Service Parameters

RTMT Collector, Alert Manager, and RTMT Reporter

Viewing a Summary

Monitoring Server Status

Understanding Server Logs

Monitoring Call-Processing Activity

Understanding Call-Processing Logs

Monitoring Services

Understanding Service Logs

Monitoring Devices

Understanding Device Logs

Monitoring CTI Applications, Devices, and Lines

Where to Find More Information

Understanding the Real-Time Monitoring Tool

Real-Time Monitoring Tool, which runs as a client-side application, uses HTTPS and TCP to monitor device status, system performance, device discovery, and CTI applications in the Cisco Unified CallManager cluster. The tool also connects directly to devices via HTTPS to troubleshoot system problems.


Note Even when RTMT does not run, tasks such as alarm and performance monitoring updates occur in the background.


RTMT allows you to perform the following tasks:

Monitor a set of management objects that are preconfigured

Generate various alerts, in the form of e-mails, for objects when values go over/below user-configured thresholds

Collect and view traces in various default viewers that exist in RTMT

Translate Q931 messages

View syslog messages in SysLog Viewer

Work with performance-monitoring counters

You can install RTMT, which works for resolutions 800*600 and above, on a Windows 98, Windows XP, Windows 2000, or Red Hat Linux with KDE and/or Gnome client. To avoid CPU spiking while you collect and zip files, do not install RTMT on a server where you installed Cisco Unified CallManager. For information on how to install and launch RTMT, refer to the following sections in the Cisco Unified CallManager Serviceability Administration Guide:

Installing the Real-Time Monitoring Tool (RTMT)

Using RTMT

To connect to any node in the Cisco Unified CallManager cluster, you must enter the CCM Administrator application user authentication information in the User Name and Password fields when you launch the tool. Likewise, you must enter either the IP address or host name of the first node. If the authentication fails for any reason, the tool prompts you to reenter the server and authentication details. After the authentication succeeds, RTMT launches the monitoring module from local cache or from a remote node when the local cache does not contain a monitoring module that matches the backend version.

After you initially load RTMT, the load includes a default configuration that is called CM-Default. The first time that you use RTMT, it will use the CM-Default profile and display the summary page in the monitor pane. CM-Default dynamically monitors all registered phones for all Cisco Unified CallManager nodes. If your cluster contains five configured Cisco Unified CallManager nodes, CM-Default displays the registered phones for each node in a Cisco Unified CallManager cluster, as well as calls in progress and active gateway ports and channels.

You can configure RTMT to display the information that interests you, such as different performance counters for different features, in the monitor pane of RTMT and save the framework of your configuration in a profile. You can then restore the profile at a later time during the same session or the next time that you log in to RTMT. By creating multiple profiles, so each profile displays unique information, you can quickly display different information by switching profiles.

You save, restore, and delete Cisco Unified CallManager configuration information by using Profile in the System menu, as described in "Working with Configuration Profiles" section.

RTMT generates daily reports in PDF format for precanned objects, as described in the "RTMT Collector, Alert Manager, and RTMT Reporter" section.

RTMT arranges the preconfigured monitoring objects into the following major categories:

Summary

Server

Call Process

Service

Device

CTI

Performance

RTMT Components

The RTMT window comprises the following main components:

Menu Bar, which includes the following menu options:

System—Allows you to save, restore, and delete existing RTMT profiles, monitor Java Heap Memory Usage, go to the Serviceability Report Archive window in Cisco Unified CallManager Serviceability, log off or exit RTMT.

Monitor—Allows you to monitor precanned objects.

Search—Allows you to search for devices, such as phones and H.323 devices, and for CTI applications, devices, and lines to monitor.

Edit—Allows you to configure categories (for table format view), set the polling rate for devices and performance monitoring counters, hide the quick launch channel, and edit the trace setting for RTMT.

Devices—Allows you to search for devices and to view phone information, port/channel status, and so on.

Performance—Allows you to work with performance monitoring counters.

Tools—Allows you to work with alerts, collect traces, and view syslog messages.

Window—Allows you to close a single RTMT window or all RTMT windows.

Application—Allows you to browse to Cisco Unified CallManager Administration or Cisco Unified CallManager Serviceability

Help—Allows you to access RTMT documentation online help or to view the RTMT version.

Quick Launch Channel—Pane on the left side of RTMT window that displays icons that you can click to monitor various objects.

View/Tool tabs—Allows you to display preconfigured categories in the Quick Launch Channel; Tool tab displays alert, trace, and syslog viewer categories.

The View and Tools tabs differ in that anything related to the View tab gets saved as a profile that can be restored at any time that RTMT is invoked. The Tools tab relates only to the systemwide alert functionality. You cannot save the states.

Monitor pane—Pane where monitoring results display.

RTMT Services, Servlets and Service Parameters

RTMT uses the following services/servlets:

AMC service—This service starts up automatically after the installation and allows RTMT to retrieve real-time information that exists on nodes in the cluster.

The following list comprises some AMC service parameters that are associated with RTMT. For the latest list of parameters, choose Server > Service Parameters in Cisco Unified CallManager Administration. Then, choose the server and the AMC service.

Primary Collector

Failover Collector

Data Collection Enabled

Data Collection Polling Rate

Data Collection Reenumeration Interval

RIS Client Timeout

Server Synchronization Period

RMI Port Number

Alert Manager Enabled

Logger Enabled

Alarm Enabled

PerfMon Log Deletion Age

For information on these service parameters, click the i button that displays in the Service Parameter window of Cisco Unified CallManager Administration.

Cisco CallManager Serviceability RTMT service (in the Control Center—Network Services window)— Supporting the Cisco Unified CallManager Real-Time Monitoring Tool (RTMT), this service installs with Cisco Unified CallManager and starts up automatically after the installation.

Cisco RTMT Reporter servlet (in the Control Center—Network Services window)—This service, which installs with Cisco Unified CallManager and starts up automatically after the installation, allows you to publish reports for RTMT.

RTMT Collector, Alert Manager, and RTMT Reporter

RTMT Collector, a component that automatically gets installed with Cisco Unified CallManager, logs preconfigured monitoring objects information while Alert Manager, also automatically installed, logs alert histories into log files. Each preconfigured object belongs to one of several categories: devices, services, servers, call activities, and PPR. Each category has a separate log file, and alert details get logged in a separate file. Also, a separate log file exists to log important perfmon object values for Cisco CallManager-related services and processes.


Tip Although they require no configuration tasks to run, RTMT Collector and Alert Manager support redundancy. If the primary collector or manager fails for any reason, the secondary collector and manager perform the tasks until primary support becomes available. RTMT Collector, Alert Manager, and RTMT Reporter run on the first node to minimize call-processing interruptions.


The locally written log files appear in the primary collector server at /var/log/active/cm/log/amc. Because the primary collector changes because of failover and fallback scenarios, the log files can exist on more than one server in the Cisco Unified CallManager cluster.

Log files exist in csv format. You can read log files, except an alert log file, by using native NT perfmon viewer. New log files get created every day at 00:00 hours on the local system. New logs for devices, services, servers, and calls get created when the time zone changes, when a new node is added to the cluster, or during failover/fallback scenarios. The first column of all these logs comprises the time zone information and the number of minutes from the Greenwich Meridian Time (GMT). RTMT Reporter uses these log files as a data source to generate daily summary reports. The report, which is based on the default monitoring objects, generates every 24 hours for the following information:

Call Activity Status—Number of call attempted and number of calls completed for each Cisco Unified CallManager, each gateway, trunk, and overall cluster. Number of channels available, in-service for each gateway.

Device Status—Number of registered phones, gateways and trunks per each server and overall cluster.

Server Status—% CPU load,% memory used,% disk space used per server.

Service Status —For each CTI Manager, number of opened devices and lines. For each TFTP server, number attempted and failed requests.

Alert Status—Number of alerts per server. Number of alert per severity level for the cluster, including the top 10 alerts in the cluster.

Performance Protection Report—Trend analysis information on default monitoring objects that allows you to track overall system health. The report includes information for the for the last 7 days for each server.


Tip The RTMT reports display in English only.


The following service parameters apply to RTMT report generation: RTMT Reporter Designated Node, RTMT Report Generation Time, and RTMT Report Deletion Age. For information on these parameters, click the i button that displays in the Service Parameter window in Cisco Unified CallManager Administration.

For more information on the Serviceability reports, see the "Serviceability Reports Archive" section on page 9-1.

Viewing a Summary

The Summary option in RTMT allows you to monitor important common information in a single monitoring pane. In a summary, you can monitor the following information:

Virtual Memory usage

CPU usage

Registered phones

Calls in progress

Active gateway ports and channels

Monitoring Server Status

The Servers category monitors CPU and memory usage, processes, disk space usage, and critical services on each Cisco Unified CallManager server.

The CPU and Memory monitor provide information about the CPU usage and Virtual memory usage on each Cisco Unified CallManager server. For each CPU, the information includes the percentage of time each processor spends executing processes in different modes and operations (User, Nice, System, Idle, IRQ, SoftIRQ, and IOWait). The percentage of CPU is the total time spent executing in all the different modes and operations excluding the Idle time. For memory, the information includes the Total, Used, Free, Shared, Buffers, Cached, Total Swap, Used Swap, and Free Swap memory in Kbytes, and the percentage of Virtual Memory in Use.

The Processes monitor provide information about the processes that are running on the system. These processes include the process, process ID (PID), CPU percentage, Status, Shared Memory (KB), Nice (level), VmRSS (KB), VmSize (KB), VmData (KB), Thread Count, Page Fault Count, and Data Stack Size (KB).

The Critical Services monitoring category provides the name of the critical service, the status (whether the service is up, down, or activated), and the elapsed time during which the services have existed in a particular status for a particular Cisco Unified CallManager node.

Table 5-1 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults. For information on daily CPU, memory, and disk usage reports, see the "Server Statistics Report" section on page 9-5.

Table 5-1 Servers Category 

Monitored Objects (displayed)
Alert/Threshold/Default

CPU Usage (100% Idle) on each server.

Virtual Memory Usage (% memory in use out of total) on each server.

CPU and memory usage for all processes on each server

Call Processing node - CPU usage pegged X% over X seconds. Default specifies 90%, 30 seconds.

Non-CallProcessing node - CPU Usage pegged X% over X seconds. Default specifies 99%, 120 seconds.

Available memory equals less than X%. Default specifies 10%.

When CPU peg or excessive memory usage alerts occur, the name of the top process displays in the alert message. In case of dllhost service, the real meaningful name displays instead of generic name dllhost.

Disk space usage for all logical drives on each server.

Available disk space on the largest logical drive equals less than X%. Default specifies 10%.

State of activated critical services on each server.

Service state changed from Up->Down.


Understanding Server Logs

Every 5 minutes, the server data gets logged into the file as a single record. The system logs the data every 5 minutes for the following counters, based on the following calculation:

cpuUsage—Average of all the values that were collected in the last 5 minutes

MemoryInUse—Average of all the values that were collected in the last 5 minutes

DiskSpaceInUse—Average of all the values that were collected in the last 5 minutes for the active partition

The AMC service logs the server data in csv format. The header of the log comprises the time zone information and a set of columns with the previous counters for a Cisco Unified CallManager node. These sets of columns repeat for every node.

The following file name format of the server log applies: ServerLog_MM_DD_YYYY_hh_mm.csv. The first line of each log file comprises the header.

To download the server logs for viewing on your local computer, refer to Trace Collection and Log Central in RTMT

Monitoring Call-Processing Activity

The Call Process monitoring category monitors the following items:

Call Activity—You can monitor the number of calls that were attempted, calls that were completed, and call in progress for a particular Cisco Unified CallManager node or the entire cluster.

Gateway Activity—You can monitor gateway activity for each gateway type. Gateway activity monitoring includes the number of active ports, the number of ports in service, and the number of calls that were completed for each gateway type for a particular Cisco Unified CallManager node or the entire cluster.

Trunk Activity—The system monitors trunk activity by trunk type for a particular node or cluster. Trunk activity monitoring includes the number of calls in progress and the number of calls that were completed for a particular trunk type.

SDL Queue—SDL Queue monitoring monitors the number of signals in the SDL queue and the number of signals that were processed for a particular signal distribution layer (SDL) queue type. The SDL queue types comprise high, normal, low, and lowest queue. You can monitor the SDL queue for a particular node or the entire cluster.

SIP Activity—The system displays a summary of SIP requests, SIP responses, total number of failed incoming responses (4xx, 5xx, and 6xx), total number of failed outgoing responses (4xx, 5xx, and 6xx), number of retry requests, and number of retry responses.

Table 5-3 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults. For information on Cisco Unified CallManager call activity daily reports, see the "Call Activities Report" section on page 9-9.

Table 5-2 Call Activities Category 

Monitored Objects (displayed)
Alert/Threshold/Default

CallsAttempted, CallsCompleted, and CallsInProgress for each Cisco Unified CallManager node and cluster.

N/A

CallsAttempted, CallsCompleted, and CallsInProgress of each type of MGCP FXS/FXO/PRI/T1CAS/H.323 gateway, as well as SIP and H.323 Trunks for each Cisco Unified CallManager node and cluster.

N/A

Channel/Port Status of each MGCP FXS/FXO/PRI/T1CAS gateway.

N/A

SDL Queue activity on each Cisco Unified CallManager node.

N/A

MGCP FXS Gateway - Number of In-Service and Active ports for each Cisco Unified CallManager node and cluster.

Route-List exhausted

MGCP FXO Gateway - number of In-Service and Active ports for each Cisco Unified CallManager node and cluster.

Route-List exhausted

MGCP PRI Gateway - Number of In-Service and Active channels for each Cisco Unified CallManager node and cluster.

D-Channel out of service

Route List exhausted

MGCP T1CAS Gateway - number of In-Service and Active ports for each Cisco Unified CallManager node and cluster.

Route List exhausted


Understanding Call-Processing Logs

The system accumulates call-processing data in the memory whenever RTMT calls the LogCall API. Every 5 minutes, RTMT logs the data into the file as a single record and cleans the memory.

The system logs data every 5 minutes for the following counters on the basis of the following calculation:

cmCallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

cmCallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

cmCallsInProgress—Average of all the values that were collected in last 5 minutes

gwMGCP_FXS_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_FXO_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_PRI_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_T1_CAS_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwH323_CallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwH323_CallsInProgress—Average of all the values that were collected in last 5 minutes

gwH323_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkH323_CallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkH323_CallsInProgress—Average of all the values collected in last 5 minutes

trunkH323_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkSIP_CallsAttempted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

trunkSIP_CallsInProgress—Average of all the values that were collected in last 5 minutes

trunkSIP_CallsCompleted—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

gwMGCP_FXS_PortsInService—Average of all the values that were collected in last 5 minutes

gwMGCP_FXO_PortsInService—Average of all the values that were collected in lasts 5 minutes

gwMGCP_PRI_SpansInService—Average of all the values that were collected in last 5 minutes

gwMGCP_T1_CAS_SpansInService—Average of all the values that were collected in last 5 minutes

gwMGCP_FXS_ActivePorts—Average of all the values that were collected in last 5 minutes

gwMGCP_FXO_ActivePorts—Average of all the values that were collected in last 5 minutes

gwMGCP_PRI_ActiveChannels—Average of all the values that were collected in last 5 minutes

gwMGCP_T1_CAS_ActiveChannels—Average of all the values that were collected in last 5 minutes

The AMC service logs the call data in windows Performance tool-compatible csv format. The header of the log comprises the time zone information and a set of columns with the previously listed counters for a node. These sets of columns repeat for every node.

The following file name format of the Call Log applies: CallLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log file comprises the header.

Monitoring Services

The Service monitoring category monitors the activities of Cisco TFTP requests, heartbeat of different nodes, and database activities.

The Cisco TFTP service builds and serves files that are consistent with the trivial file transfer protocol, which is a simplified version of the File Transfer Protocol (FTP). Cisco TFTP builds configuration files and serves embedded component executables, ringer files, and device configuration files. You can view the total Cisco TFTP requests, requests not found, and requests that were aborted.

The tool (RTMT) monitors the heartbeat of Cisco CallManagers, Cisco TFTPs, and Cisco CallManager Attendant Console Server services for different nodes.The heartbeat acts as an indicator of the life of whatever it is monitoring. When the heartbeat is lost, a blinking icon appears in the lower, right corner of the RTMT window. To find when the heartbeat loss was detected, click the blinking icon. An e-mail can notify you of the heartbeat loss, if you configure the system to do so.

The database summary provides connection information for each node, such as the connection requests that are queued in the database, the connection requests that are queued in memory, the total number of active client connections, and the number of devices that are queued for a device reset.

Table 5-3 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults. For information on daily reports for CTI and Cisco TFTP usage statistics, see the "Service Statistics Report" section on page 9-7.

Table 5-3 Services Category 

Monitored Objects (displayed)
Alert/Threshold/Default

Number of open devices, lines, CTI connections, and active Cisco CallManager links for each CTI Manager.

N/A

TotalTftpRequests and TotalTftpRequestsAborted for each Cisco TFTP server.

N/A

Connection and replication status for each Directory server.

Connection failed.

Replication failed.

Heartbeat rate for each Cisco CallManager, Cisco TFTP, and Cisco CallManager Attendant Console Server services.

Cisco Unified CallManager heartbeat rate equals <0.x. Default equals 0.5.

Cisco TFTP heartbeat rate equals <0.x. Default specifies 0.5.

Cisco CallManager Attendant Console Server heartbeat rate equals <0.x. Default specifies 0.5.


Understanding Service Logs

The service data accumulates in the memory whenever RTMT calls the LogService API. Every 5 minutes, RTMT logs the data into the file as a single record and cleans the memory.

The system logs data every 5 minutes for the following counters, based on the following calculation:

ctiOpenDevices—Average of all the values that were collected in last 5 minutes

ctiLines—Average of all the values that were collected in last 5 minutes

ctiConnections—Average of all the values that were collected in last 5 minutes

ctiActiveCMLinks—Average of all the values that were collected in last 5 minutes

tftpRequests—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

tftpAbortedRequests—Cumulative (difference between last collected value and the first collected value in last 5 minutes)

The AMC service logs the service data in csv format. The header of the log comprises the time zone information and a set of columns with the counters that were previously listed for a Cisco Unified CallManager node. These sets of columns repeat for every node.

The following file name format of the Service Log applies: ServiceLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log comprises the header.

Monitoring Devices

The Device monitoring category provides a summary of devices, device search capability, and a summary of phones.

The device summary provides information on the number of registered phones, gateways, and media resource devices on each Cisco Unified CallManager. Table 5-4 provides information about the objects that RTMT monitors, the alert, thresholds, and defaults, and what kind of reports that RTMT generates for those devices. For information on daily reports on number of registered devices, see the "Device Statistics Report" section on page 9-2.

Table 5-4 Devices Category 

Monitored Objects (displayed)
Alert/Threshold/Default

Number of registered phones for each Cisco Unified CallManager and cluster.

Total number of registered phones drops by X% in consecutive polls. Default specifies 10%.

Number of registered gateways on each Cisco Unified CallManager and cluster.

(Warning) Clusterwide total number of registered gateways decreased in consecutive polls.

(Informational) Clusterwide total number of registered gateways increased in consecutive polls.

Number of registered media devices on each Cisco Unified CallManager and cluster.

(Warning) Clusterwide total number of registered media devices decreased in consecutive polls.

(Informational) Clusterwide total number of registered media devices increased in consecutive polls.

Media List exhausted.


The Device Search menu comprises the following items on which you can search: phones, gateway devices, H.323 devices, CTI devices, voice-messaging devices, media resources, hunt lists, and SIP trunks.

You can search on any device in the Cisco Unified CallManager cluster and choose the status of the devices, including registered, unregistered, rejected, any status, and devices that are only configured in the database. You can also search by any model, or a specific device model, and set up criteria that include several different attributes. For phones, you can also search on the basis of phone protocol.


Tip To find the matching item, RTMT requires that you activate the RIS service in the Service Activation window.


Results display in a table with a row for each matched device, a column for each of the specified attributes, and a time stamp of the device that has been opened/closed and the application that controls the device media.

The phone summary provides information on the number of Registered Phones, SIP phones, SCCP Phones, Partially Registered phones, and the number of failed registration attempts.

Understanding Device Logs

The device data accumulates in the memory whenever RTMT calls the LogDevice API. Every 5 minutes, RTMT logs the data into the file as a single record and cleans the memory.

The data gets logged every 5 minutes for the following counters based on the following calculation:

gatewayDevicesFXS—Average of all the values that were collected in last 5 minutes

gatewayDevicesFXO—Average of all the values that were collected in last 5 minutes

gatewayDevicesPRI—Average of all the values that were collected in last 5 minutes

gatewayDevicesT1—Average of all the values that were collected in last 5 minutes

gatewayDevicesH323—Average of all the values that were collected in last 5 minutes

The AMC service logs the device data in csv format. The header of the log comprises the time zone information and a set of columns with the previously listed counters for a node. These sets of columns repeat for every node.

The following file name format of the Device Log applies: DeviceLog_MM_DD_YYYY_hh_mm.csv.

The first line of each log file comprises the header.

Monitoring CTI Applications, Devices, and Lines

The CTI search menu allows you to search on the following CTI components:

CTI Applications

CTI Devices

CTI Lines

The CTI category monitors CTI Manager activities and provides CTI search capability. With CTI Manager, you can monitor the number of open devices, lines, and CTI connections. You can specify criteria for the CTI applications, devices, and lines that include CTI status, device name, application pattern, and attributes.


Tip To find the matching item, RTMT requires that you activate the RIS service in the Service Activation window in Cisco Unified CallManager Serviceability.


Results display in a table with a row for each matched device, a column for each of the specified attributes, and a time stamp of the device that has been opened/closed and the application that controls the device media.

Where to Find More Information

Related Topics

Alert Configuration in RTMT, Cisco Unified CallManager Serviceability Administration Guide

Configuring and Using Performance Monitoring, Cisco Unified CallManager Serviceability Administration Guide

Trace Collection and Log Central in RTMT, Cisco Unified CallManager Serviceability Administration Guide

Alerts, page 7-1

Performance Objects and Counters, page 6-1