Guest

Cisco Unified Communications Manager (CallManager)

Cisco CallManager: Troubleshooting RTMT Issues

Document ID: 109377

Updated: Aug 25, 2011

   Print

Introduction

This document addresses how to troubleshoot the Real-Time Monitoring Tool (RTMT) issues with Cisco Unified Communications Manager.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on Cisco Unified CallManager 4.x/5.x/6.x/7.x.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Problem: RTMT Java Exception Error

The RTMT does not work, and these error messages appear: java.util.MissingResourceException: Can't find bundle for base name org.apache.xerces.msg.XMLMessages,. The error message appears when a user attempts to access the RTMT, or after upgrading the operating system of the CallManager server.

Solution 1

Complete these steps in order to resolve this issue:

  1. Go to the Windows Start menu and choose Programs > Administrative Tools > Internet Services Manager.

  2. Expand the CallManager server name from the Internet Information Services window and click Default Web Site.

  3. Right click the AST and select the Properties option from the left side of the window.

  4. Select the Virtual Directory tab from the AST Properties window.

  5. Verify that the Execute Permissions field under the Application Settings section is set to Script and Executables.

  6. Restart the IIS through command prompt. Use this command: C:\utils>iisreset.

Solution 2

RTMT did not work after upgrading the Windows operating system and displays the same error.

Complete these steps in order to resolve this issue:

  1. Open the urlscan.ini file from the C:\Winnt\system32\inetsrv\URLscan\ location.

  2. Change the RemoveServerHeader parameter value from 0 to 1.

Problem: Unable to Pull QRT reports using RTMT

With Cisco Unified Communications Manager 6.x, pulling the Quality Report Tool (QRT) reports using RTMT fails even though QRT is enabled on Cisco IP Phones.

Solution

Complete these steps in order to resolve this issue:

  1. In the Cisco Unified Communications Manager server, open the Cisco Unified Serviceability page.

  2. Go to Tools > Control Center - Feature Services.

  3. Check if the Cisco Extended Functions service under Voice Quality Reporter Services is activated and running. If not, start the service. If it indicates that the service is running, restart it.

Error: [RTMT-ALERT-StandAloneCluster] RouteListExhausted

This error message is reported by RTMT:

[RTMT-ALERT-StandAloneCluster] RouteListExhausted
Explanation: An available route could not be found in the indicated route list

Solution

The RouteListExhausted messages indicate that CallManager was unable to send a call to those Route Groups in a Route List. This is usually as a result of no B channels or ports being available on a gateway for routing a call. If you have a Route List with an H.323 gateway as the first Route Group, and a second Route Group that should take any overflow, then sometimes the first H.323 gateway will send a User Busy back to the CallManager if all of its channels or ports are busy. If this occurs, then the call is not forwarded to the second Route Group in the Route List.

If this is how your own Route List is configured, then go to Cisco CallManager Administration > Service > Service Parameters > Select a server > Select Cisco CallManager and configure the Stop Routing on User Busy Flag service parameter to False.

This will mean that the CallManager will try other gateways to re-route the call on a User Busy before reaching the RouteListExhausted state. However, if there are no RouteListExhausted events in the Application Log, then the RouteListExhausted messages reported by RTMT can be a result of an Unallocated number or User Busy messages from the PSTN. In this case, this message can be ignored.

Error: [RTMT-ALERT-StandAloneCluster] CriticalServiceDown

This error message is reported by RTMT:

[RTMT-ALERT-StandAloneCluster] CriticalServiceDown
Service status is DOWN. Connection Database Proxy.

Solution

This error can occur when the value of the Database Proxy: Service Shutdown Timer is set as the default value, which is zero. In order to remove this error, set the value of Database Proxy Service Shutdown Timer to a value between 0 and 999. Perform these steps:

In the Cisco Unity Connection Administration page, go to System Settings > Advanced > Connection Administration. Enter a value between 0 and 999.

Note:  If you restart the server, the remote database proxy service will remain shut off. After a system restart, turn on the service manually.

To activate the Remote Database Proxy Service, perform these steps:

  1. Go to the Cisco Unity Connection Serviceability web admin page.

  2. On the Tools menu, choose the Service Management page.

  3. Click the Activate button next to the Connection Database Proxy item under Optional Services.

Error: RTMT-ALERT: DBChangeNotify queue delay over 2 minutes

This error message is reported by RTMT:

RTMT alert:
DBChangeNotify queue delay over 2 minutes. Current DB ChangeNotify queue
delay (150) is over 120-sec threshold.

Solution

DBChangeNotifyFailure—This alert occurs when the Cisco Database Notification Service experiences problems and might need to be halted. This condition indicates that Change Notification Requests queued in the database are "stuck" and system changes are not taking effect. If you use the default alert properties, the alert gets triggered when the DBChangeNotify queue delay is over two minutes. One alert is sent every 30 minutes.

Also, the DBL service is responsible for relaying these DBChangeNotify messages. In order to resolve the issue, you need to restart the Cisco Database Layer Monitor service on all servers in the cluster. The command for restarting the Cisco Database Layer Monitor service is shown here:

admin:utils service restart Cisco Database Layer Monitor
Service Manager is running
Cisco Database Layer Monitor[STOPPING]
Cisco Database Layer Monitor[STOPPING]
Cisco Database Layer Monitor[STOPPING]
Cisco Database Layer Monitor[STOPPING]
Commanded Out of Service
Cisco Database Layer Monitor[NOTRUNNIG]
Service Manager is running
Cisco Database Layer Monitor[STARTED]
admin:

Note: Restarting the Cisco DBL service will not reset any phones or devices. It just reloads the database cache and forces the CallManager service to get a fresh copy of the database.

Unable to Download RTMT Plug-in

User is unable to download the RTMT plug-in.

Solution

If you are using third-party download software, this issue can occur. Uninstalling the third-party download software will resolve the issue.

Unable to Monitor the Informix Database

Solution

If you already have an SNMP notification destination configured on CallManager with Write and Notify set, you can enable traps to be sent based on local syslog messages:

  1. You need to enable clogNotificationsEnabled (1.3.6.1.4.1.9.9.41.1.1.2). You can use this command to enable it:

    snmpset -c <community_string> -v 2c 14.48.46.2 .1.3.6.1.4.1.9.9.41.1.1.2.0 i 1
    
  2. Now, in CallManager, go to the serviceability page > Configuration, choose your server, and then Service Group > Database and Admin Services > Database Layer Monitor. Make sure local syslogs are Enabled and set to error.

  3. To check what alerts are at which levels (all of them are at error or alert), go to Alarm > Definitions. Choose the System Alarm Catalog, and then the DBAlarmCatalog. Provided there are all the database alerts we have and can trap on.

Unable to Download PDF Files

With CUCM 6.x, PDF files cannot be downloaded from the Serviceability Reports archive. This error message is received:

Page cannot be displayed

Solution

In order to resolve the issue, restart Cisco Serviceability Reporter from Tools > Control Center -Feature Services.

Error: Exceeds max limit for trace file download requests

This error message appears in RTMT:

Exceeds max limit for trace file download requests. 
Current limit is 2

Solution

Restart these system services in order to resolve the issue:

  1. Cisco Trace Collection Servlet

  2. Cisco Trace Collection Service

No Option Available when the QRT Softkey is Pressed

When the user presses the QRT softkey, nothing happens.

Solution

Restart the CTIManager service in order to resolve the issue.

Error: CDRFileDeliveryFailed BillingServerAddress

This alert is received on RTMT:

CDRFileDeliveryFailed BillingServerAddress : 10.1.11.1 AppID : Cisco CDR
Repository Manager ClusterID : NodeID : PUBLISHER . The alarm is
generated on Wed Jul 07 17:13:05 IST 2010.

Solution

Restart these services in order to resolve the issue:

  • CDR Repository Manager

    This service maintains and moves the generated CDRs that are obtained from the Cisco CDR Agent service. In a system that supports clusters (Cisco Unified Communications Manager only), the service exists on the first server.

  • Cisco SOAP - CDRonDemand

    This is a SOAP/HTTPS-based service that runs on the CDR Repository server. It receives SOAP requests for CDR file name lists that are based on a user-specified time interval (up to a maximum of 1 hour) and returns a list of file names that fit the time duration that is specified in the request. This service also receives requests for delivery of a specific CDR/CMR file with the file name and the transfer method (SFTP/FTP, server name, login info, directory) that is specified in the request.

Excessive mixer calls

With Cisco Unified Communications Manager 7.0, the IP Voice Media Streaming Application service driver shows this system log error: Excessive mixer calls.

Solution

Because this appears to be a cosmetic error, you can disable the RTMT alert related to the error message. Or if you want to get the issue permanently resolved, you can upgrade your CallManager server to any of the releases mentioned in Cisco bug ID CSCte19629 (registered customers only) . This bug ID contains the fix for this defect.

How to Get Usage Information on Hardware Conference

In order to monitor usage information on Hardware Conference from the Communication Manager, perform these steps:

  1. Log in to RTMT.

  2. Go to System > Performance > Open Performance Monitoring.

  3. Choose Cisco HW Conference Brige Device.

rtmt-error-tshoot-01.gif

Collect Raw CDR Data using RTMT

In order to collect raw Call Detail Records (CDR) data using RTMT, perform these steps:

  1. Log in to RTMT.

  2. Go to System > Tools > Trace > Trace & Log Central and select the following:

    Cisco CDR files on CM server
     Cisco CDR files on Publisher Processed
  3. Check All Servers for both the above services.

  4. Click Next, and again click Next. Give the appropriate time range for which you want to collect raw CDR data.

  5. Specify a Download File Directory and zip the files.

  6. Click Finish. Collect the files.

Pull the Report for the PRIs using the RTMT

Perform these steps in order to pull the Report for the Primary Rate Interface (PRI) using the RTMT:

  1. Go to RTMT > System > Performance > Cisco MGCP PRI, Calls Active or Channel XX status.

  2. Drag the object (Calls Active) to the right panel (if possible, select the last PRIs on the Route Groups).

  3. Right click and select Alert > Severity: Warning, Absolute Value 20.

  4. If you already have the email alert enabled, choose the profile for Action email. Click Save.

  5. If you do not have the email alert enabled, you need to configure this alarm on the RTMT. Perform these steps:

    1. Login to the RTMT and choose Alert Center.

    2. Right click on the XXXXXX alert, and choose Set Alert/Properties.... Configure the following and click Next:

      • The alert is enabled.

      • Select the severity of the alert.

      • Select the server that you want to make the report.

    3. From this window, choose Duration: Trigger alert immediately and click Next.

    4. From this window, configure the following and click Next:

      • Frequency: trigger alert on every poll

      • Schedule: trigger alert when it occurs

    5. Configure the following and click OK:

      • Check the Enable Email check box.

      • Go to Configuration > Add > Enter name and description.

      • Under recipients, click Add and insert the email to where you want to send the alert information.

Remove Recipient from Email Notification of RTMT

In order to remove a recipient(s) from email notification of RTMT, perform these steps:

  1. Go to RTMT > System > Tools > Alert > Alert Central.

    rtmt-error-tshoot-02.gif

  2. Choose the Alert Name to modify.

    rtmt-error-tshoot-03.gif

  3. Go to RTMT > System > Tools > Alert > Config Alert Action.

    rtmt-error-tshoot-04.gif

  4. Choose the action profile for email. Click Edit and select the Recipient(s) to be removed.

    rtmt-error-tshoot-05.gif

  5. Click Delete. Then, click OK and Close.

RTMT stuck at "loading please wait...."

The RTMT hangs because of an UnsupportedClass Exception as the JRE used in compiling the code and launching the RTMT is different. In order to remove this error, complete these steps:

  1. Install RTMT. By default, RTMT is installed at C:\Program Files\Cisco\CallManager Servicability.

  2. Copy the jre1.5.0_14 folder from C:\Program Files\Java to C:\Program Files\Cisco\CallManager Servicability\JRE.

  3. Go to C:\Program Files\Cisco\CallManager Servicability\jrtmt and open run.bat in a notepad. Then, replace the content with ..\JRE\jre1.5.0_14\bin\java -Dsun.java2d.d3d=false-Xbootclasspath/a:.;.\Lib\xerces.jar;.\Lib\log4j.jar;.\Lib\jcchart45 0J.jar;.\Lib\collections.jar -jar JRtmt.jar.

  4. Double click run.bat.

  5. If the double click does not work, go to Start > Run and type cmd.

  6. Type cd C:\Program Files\Cisco\CallManager Serviceability\jrtmt and then type run.bat. This opens the RTMT login screen.

    Note: Perform run.bat whenever you want to use RTMT. A shortcut created for run.bat on the desktop does not work.

    Refer to Cisco bug ID CSCsu86458 (registered customers only) for more information.

No line/device available

After pressing dial on the WebDialer pop up, the "No line / device available" error is received. This error message is reported by RTMT:

CTIManager: 1055: LEHIUCMPUB.ad.local: Apr 14 2011 22:28:26.545 
UTC :%UC_CTI-3-CtiDeviceOpenFailure:%[CTIconnectionId=3][DeviceName=SEPA40CC395368C]
UNKNOWN_PARAMTYPE:ReasonCode=-1932787499]
[AppID=Cisco CTIManager][ClusterID=StandAloneCluster][NodeID=LEHIUCMPUB][CTIAppID=Cisco
WebDialer][ProcessID=30024][ProcessName=CtiHandler]: Device Open failed

Solution

In order to resolve this issue, deactivate/reactivate the WebDialer service.

RTMT alerts sent via email fail to deliver

The RTMT email alerts fail to deliver if the DNS is not configured on the Cisco Unified Communications Manager server. If DNS is not configured on the Cisco Unified Communications Manager servers during or post install, the fully qualified domain name (FQDN) will not append to the email sent by RTMT. This causes the email to be undeliverable.

This issue is documented by Cisco bug ID CSCsk71330 (registered customers only) .

Solution

Configure DNS from the Command Line Interface (CLI) using these commands:

set network dns primary xxx.xxx.xxx.xxx

set network dns secondary xxx.xxx.xxx.xxx

set network domain yourdomain.com

After you enter the commands, reboot the node for the changes to take effect.

CUCM User is unable to Access the RTMT

After creating a new user in Cisco Unified Communications Manager, you cannot make this user access the RTMT.

Solution

In order to resolve this issue, complete these steps:

  1. Find the Standard Packet Sniffer Users group and click Add Users to Group.

  2. Add the specific end user who should access the RTMT to this group. Click Save.

Getting CTI RTMT alerts

Problem

This error is received continuously on the subscriber:

%CCM_CTI-CTI-3-kCtiEnvProcDevListRegTimeout: Directory change notification 
request time out. Cluster ID:StandAloneCluster Node ID:PMC-BMI-CM04

Note: When researching the error message, it states to check network connectivity with that node. All nodes are alive and working, but this error message is persistent.

Solution

In order to resolve this issue, restart the CTI Manager service on that node.

Related Information

Updated: Aug 25, 2011
Document ID: 109377