Guest

Cisco Unified Communications Manager (CallManager)

Unified Communications Manager 5.x: Troubleshooting Memory Issues

Cisco - Unified Communications Manager 5.x: Troubleshooting Memory Issues

Introduction

This document describes how to troubleshoot memory issues in Cisco Unified Communications Manager 5.x.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the Cisco Unified Communications Manager 5.x.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

Monitoring Virtual Memory

There are multiple ways to monitor memory usage in real time. The RTMT summary screen displays total virtual memory usage for all servers in the cluster in graphical form.

  • RTMT CPU and Memory Screen

    RTMT CPU and Memory screen displays total virtual usage in graphical form for all servers in the cluster. It also includes a table view that displays a break down of memory usage for each server in the cluster.

  • RTMT Process Screen

    The Process screen provides per-process memory usage information. In addition to memory information, this screen provides other information related to process.

  • RTMT Performance Viewer

    There are a number of system level counters under the Memory perfmon object and a number of process level counters under the Process perfmon object.

Memory “Free” and “Used” counters

Free counters show the amount of free memory, and Used counters show the amount of used memory. The issue with these counters is that Free counters show fairly small numbers, and Used counters show a number close to the total physical memory installed on the server. This can cause concern as many would conclude that the memory is running very low.

If you have remote access enabled, running the commands show tech runtime memory and show tech runtime cpu command will also give the same information of low memory. However, the system runs fine without any hint of memory starvation because of the way Linux manages memory. When the Linux system is not under memory pressure, all of the kernel’s caches will slowly grow over time. Linux prefers to keep larger caches just in case someone needs information in the cache. After all, there is no reason to return cache memory if no one really needs it. This cache memory is readily available when any process needs more memory. So the way we calculate the true usage of memory is by counting buffers and cached memory as well as free memory as available. % Mem Used and % VM Used are calculated based on this logic:

Actual used physical memory size = Total KBytes * % Mem Used
Actual used virtual memory size = (Total KBytes + Total Swap KBytes) * % VM Used

Common Memory Issues

Memory leak

If the process object VmSize continues to increase over time, memory leaks might be associated with the process.

Note: The VmSize counter displays the total virtual memory usage for a task in kilobytes (KB). It includes all code, data, shared libraries, and pages that have been swapped out

  • You might want to restart the service after you save the necesary troubleshooting information such as trace, RIS troubleshooting log, etc.

  • File Descriptor (FD) leaks can result in memory leaks. Use the show process list fd in order to check the FD count.

  • Check the thread counts. Thread leaks can also cause memory leaks since by default Linux allocates 10M stack memory when it spawns a thread. Thread count for each process is shown in the RTMT Process screen. There are perfmon counters for thread and FD counts under the System perfmon object.

Alert: LowAvailableVirtualMemory

This alert indicates available virtual memory is running low. Virtual memory consists of physical memory plus swap memory. This alert is based on comparing % VM Used against configured alert threshold. % VM Used is calculated as (Total KBytes - Free KBytes - Buffers KBytes - Cached KBytes + Shared KBytes + Used Swap KBytes) / (Total KBytes + Total Swap KBytes). When virtual memory runs out, the operating system kills process(es) in an unpredictable manner in order to free up virtual memory, so it is critical to take care of the situation first.

  1. If you suspect a memory leak, you need to restart that process. VmSize can be used to monitor the memory leak over a period of time. Also, VmRSS counter shows the amount of physical memory a process uses.

    Note:  VmRSS counter displays the virtual memory (Vm) resident set size (RSS) that is currently in physical memory in kilobytes (KB) This includes the code, data, and stack. For more information on the various process counters, refer to Performance Objects and Counters.

  2. The RTMT Summary and CPU and Memory screens show memory usage at the system level. The Process screen provides information on memory usage at the process level.

After upgrading to CUCM 7.x, you might receive this error: [RTMT-ALERT-StandAloneCluster] LowAvailableVirtualMemory.

The solution for this is to restart the Cisco Tomcat service from the CLI of the publisher by using this command:

  • utils service restart Cisco Tomcat

Alert: LowSwapPartitionAvailableDiskSpace

This alert indicates that the available swap partition is running low. The swap partition is part of virtual memory; therefore, the low available swap partition disk space indicates low virtual memory as well.

  1. Find out how much swap space and virtual memory are still available by looking at Disk Usage and CPU and Memory screens.

  2. Use the RTMT Process screen in order to find out which process uses the most memory (sort by VmSize).

  3. This alert can also occur due to a memory leak so you want to check if any process uses an unusual amount of memory, and if so, you need to restart that process.

Also, refer to RTMT Alert:LowSwapPartitionAvailableDiskSpace for more information.

Alert: LowActivePartitionAvailableDiskSpace

RTMT displays an LowActivePartitionAvailableDiskSpace alert, which indicates there is a low amount of disk space available in the active partition.

The Cisco Unified Communication Manager is designed to run with very little available space in the active partition. Therefore, in most cases this alarm does not represent a problem. However, additional disk space is made available in the 5.1(3) and 6.1 releases. To get rid of this issue, you can upgrade to CUCM 5.1(3) and 6.1 releases. If upgrading to these releases is not an option, then, change the alarm threshold in RTMT using the following procedure:

  1. Log into RTMT, and choose Tools > Alert > Alert Central.

  2. Right-click LowActivePartitionAvailableDiskSpace, and choose Set Alert Properties.

  3. Click Next.

  4. Change the value to a lower value.

  5. Click Next, and then click Next.

  6. Click Activate.

Known Issues

Here are some of the known defects in CUCM 5.x that cause disk full or are created by disk full issues:

  • CSCsl16440 (registered customers only) : CAR Systemoverview report failure causing active partition full.

  • CSCsj95433 (registered customers only) : CAR reports should write to common partition, active partition full.

  • CSCsf11187 (registered customers only) : System does not fully start - database is not started by servm.

  • CSCsd93783 (registered customers only) : Loader throws OutOfMemoryError when run for long time

  • CSCsi53145 (registered customers only) : RTMT LowActivePartitionAvailableDiskSpace raised after upgrade.

Related Information

Updated: Mar 24, 2009
Document ID: 109478