Guest

Cisco Unity

Unity Virtual Memory Fragmentation - Identification and Solutions

Document ID: 51477

Updated: Jan 31, 2006

   Print

Introduction

A Windows 2000 server describes the use of virtual bytes as, "the current size in bytes of the virtual address space the process is using. Use of virtual address space does not necessarily imply that one corresponds the use of either disk or main memory pages. Virtual space is finite, and if you use too much, the process can limit its ability to load libraries."

The primary Cisco Unity service, AvCsMgr, allocates virtual bytes for a variety of processes. These processes include message delivery and audio recording, and playback. This document discusses:

  • AvCsMgr virtual memory usage patterns.

  • Ways to identify potential virtual memory fragmentation issues.

  • Solutions you can use to mitigate and correct excessive virtual memory usage.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on Cisco Unity versions 3.0 and later.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to the Cisco Technical Tips Conventions for more information on document conventions.

Virtual Memory Usage Pattern

Figure 1 illustrates a typical virtual memory usage pattern for the AvCsMgr process on a Cisco Unity system.

The time period shown begins with the initial startup of Cisco Unity. As system activity peaks and normal levels of fragmentation occur, the amount of virtual memory that AvCsMgr uses stabilizes to the horizontal pattern that Figure 1 shows. The maximum amount of virtual memory that AvCsMgr uses and the period of time to reach this state varies, based on the voice call traffic and the number of subscribers on the system.

Figure 1: Typical AvCsMgr Virtual Memory Usage

unity-virt-memory-frag-1.gif

Figure 2 illustrates a potentially problematic pattern of AvCsMgr virtual memory usage. Consistent increases after the point at which the AvCsMgr virtual memory plateaus can indicate excessive AvCsMgr virtual memory allocation or fragmentation. The pattern that Figure 2 illustrates does not necessarily indicate a problem. For example, the sharp increases are not unusual when large numbers of subscribers are added to the system, or as system activity reaches a new peak since the last restart of the server. If this pattern is observed, Cisco recommends that you closely monitor the activity in order to determine whether usage is excessive.

Figure 2: Potentially Problematic AvCsMgr Virtual Memory Usage

unity-virt-memory-frag-2.gif

AvCsMgr virtual memory usage should never exceed 1.65 GB (or 2.4 GB on systems that run Windows Advanced Server and that uses the /3GB switch).

Use Performance Monitor to Monitor AvCsMgr Virtual Memory Usage

You can log the Cisco Unity virtual memory usage pattern with the use of Microsoft Windows Performance Monitor. Refer to the Performance Monitoring chapter of the Cisco Unity Maintenance Guide for information on the use of Performance Monitor.

Complete these steps in order to view this information real-time in the Performance Monitor:

  1. Select Start > Programs > Administrative Tools > Performance in order to start Performance Monitor.

  2. Click System Monitor in the tree view on the right.

  3. Right-click the control in the control view in the right panel.

  4. Click Add Counters in the popup menu.

  5. Select the Cisco Unity computer you wish to monitor.

  6. Select the Process Performance object.

  7. Scroll down the counter list on the left and select the Virtual Bytes counter.

  8. Scroll down the instance list on the right and select the AvCsMgr instance.

  9. Click Add in order to add the Virtual Bytes counter to the control view. You might need to change the scale to correctly view this counter in the Graph view.

  10. In the control view in the right panel, right-click the control and in the popup menu, click Properties.

  11. In the Data tab of the System Monitor Properties dialog box, click Scale and then select 0.0000001 from the drop-down list.

  12. Change the vertical scale value to 200 in the Graph data of the System Monitor Properties dialog box.

  13. Click OK.

    For example, the maximum value reported in Figure 1 corresponds to 1201238093 bytes or 1.20 GB. If you observe this counter in real time, this value should not be higher than 1.65 GB (or 2.4 GB on systems that run Windows Advanced Server and that use the /3GB switch).

    Cisco recommends that you use Performance Monitor alerts in order to notify the Cisco Unity administrator when the Virtual Bytes value exceeds the 1.65 GB limit (or 2.4 GB on systems that are running Windows Advanced Server and that use the /3GB switch).

Configure a Performance Monitor Alert for the AvCsMgr Virtual Bytes Counter

Complete these steps in order to configure a Performance Monitor alert for the AvCsMgr Virtual Bytes counter:

  1. Select Start > Programs > Administrative Tools > Performance in order to start the Performance Monitor.

  2. Double-click Performance Logs and Alerts and then click Alerts.

    Any existing alerts are listed in the details pane. A green icon indicates that the alerts are running; a red icon indicates that alerts have been stopped.

  3. Right-click in a blank area of the details pane and select New Alert Settings.

  4. In the Name field, enter the name of the alert and then click OK.

  5. Under the General tab, in the comment section, enter a comment to describe the alert.

  6. Click Add.

  7. Click Use Local Computer Counters to monitor counters from the computer on which the Performance Logs and Alerts service run.

    Alternatively, in order to monitor counters from a specific computer regardless of where the service runs, click Select Counters from Computer and specify the name of the computer you want to monitor.

  8. Select the Process Performance object.

  9. Scroll down the counter list on the left and select the Virtual Bytes counter.

  10. Scroll down the instance list on the right and select the AvCsMgr instance.

  11. Click Add and then close the dialog box.

  12. In the Alert field (when the value is), specify Over and in the Limit field, specify the value that triggers the alert.

  13. Specify the amount and the unit of measure for the update interval (for example, 1 minute) in the Sample Data Every field.

  14. Select the action to be executed when the alert is triggered under the Action tab.

    Cisco recommends that you configure the alert service to write to the Event log in addition to any other action you want to set up. Event log alerts of this type can also be configured to notify an administrator via the EMS utility found at the Cisco Unity Tools Event Monitoring Service page.

  15. Under the Schedule tab, select Start Scan Manually and Stop Scan Manually.

  16. Click OK.

  17. Select the alert you just configured, and then right-click the alert and select Start.

    Note: Alert needs to be started every time the Cisco Unity system restarts.

Problem

Virtual address space for user mode applications that run on a Windows 2000 Server or Windows 2000 Advanced Server is limited to 4 GB. By default, the 4 GB of address space is divided into 2 GB each for user mode and for kernel mode processes. The Cisco Unity system (a user mode process) only exceeds the 2 GB limit when there is excessive virtual memory fragmentation. When the 2 GB limit is reached, it causes the AvCsMgr service to shut down unexpectedly.

Symptoms on Cisco Unity 3.x Systems

As underlying components are no longer able to allocate memory, subscribers can hear the failsafe conversation (when Cisco Unity plays the prompt "Sorry, system temporarily unavailable. Please try again later"), and one or more of these errors appears in the application event log:

Source: VBRuntime
EventID: 1
Description:
The VB Application identified by the event source logged this Application 
AvSubSetupUtilsENU: Thread ID: 5984, Logged: modPhraseUtilGeneral: 
ErrorGettingProperty: Error getting property: MailUserExtension 

Source: AvPhraseServer_MC
EventID: 10002
Description:
AvPhrase::Make - error returned from Make for PhraseServer [AvSubMsgCountENU], 
PhraseID [AnnounceMsgCount], Message [Invalid procedure call or argument], 
Return code [800A0005h].

Source: AvArbiter_MC
EventID: 1020
Description:
[Port 26] Failed routing a continued call to application [PHGreeting] with data 
object type [0]. Using Failsafe instead.

Symptoms on Cisco Unity 4.0 Systems

As underlying components are no longer able to allocate memory, subscribers can hear the failsafe conversation (when Cisco Unity plays the prompt "Sorry, system temporarily unavailable. Please try again later"), and one or more of these errors appear in the application event log:

Event Source: CiscoUnity_ConvMsg
Event ID: 10015
Description:
IAvDohAttachments::get_All returned [0x8007000e] on line 191 of file 
e:\views\Unity4.0.1.55\un_Core2\ConversationEng\AvStateSvr\AvSMsgSend.cpp.

Event Source: CiscoUnity_ConvSubscriber
Event ID: 119
Description:
An unexpected error has occurred while a subscriber is on the phone 
with Unity. This can potentially result in the subscriber hearing the 
failsafe conversation. Technical details are - IAvDohMessages::get_All returned 
[0x8007000e] on line 899 of file 
e:\views\Unity4.0.1.55\un_Conv1\AvConvSubscriber\AvConvSubMsgCountSvr\AvSCOSAndMsgCount.cpp.

Event Source: CiscoUnity_Wav
Event ID: 806
Description:
Cisco Unity’s multi-media component has encountered a serious error.
Explanation:
A serious failure has occurred on port 5 while writing data to an audio stream. 
Depending upon the severity of the failure, parties on this call could experience 
unexpected behavior or be disconnected. In some cases, further calls on this 
port will not be handled correctly.
Technical Details:
Thread 0x00001564 had a Failure on Port 5 in AvWav
Description: 
File: e:\views\Unity4.0.1.55\un_Miu\UnityAvWav\STRMIO.CPP(267)
Method: StreamIoWrite
Failure: call to IStream:Write with error(8007000e).

Event Source: CiscoUnity_ConvAddrSearch
Event ID: 106
Description:
Unexpected error occurred in the Subscriber searcher conversation. 
Technical details - IAvDohComposites::AddMany returned [0x8007000e] 
on line 2159 of file 
e:\views\Unity4.0.1.55\un_Conv3\AvConvUtilities\AvConvAddrSearchSvr\AvSListNamesSetup.cpp 
For further investigation, please turn on all the diagnostics for Address Searcher.

Event Source: CiscoUnity_ConvAddrSearch
Event ID: 106
Description:
Unexpected error occurred in the Subscriber searcher conversation. 
Technical details - SortObjectsInIterator returned [0x8007000e] on line 
1373 of file 
e:\views\Unity4.0.1.55\un_Conv3\AvConvUtilities\AvConvAddrSearchSvr\AvSListNamesSetup.cpp 
For further investigation, please turn on all the diagnostics for Address Searcher.

Event Source: CiscoUnity_ConvAddrSearch
Event ID: 106
Description:
Unexpected error occurred in the Subscriber searcher conversation. 
Technical details - AddNamesToSelectionList returned [0x8007000e] on line 263 of file
e:\views\Unity4.0.1.55\un_Conv3\AvConvUtilities\AvConvAddrSearchSvr\AvSListNamesSetup.cpp 
For further investigation, please turn on all the diagnostics for Address Searcher.

Event Source: CiscoUnity_ConvAddrSearch
Event ID: 106
Description:
Unexpected error occurred in the Subscriber searcher conversation. 
Technical details - GetObjectList returned [0x8007000e] on line 482 of file 
e:\views\Unity4.0.1.55\un_Conv3\AvConvUtilities\AvConvAddrSearchSvr\AvSListNamesSetup.cpp 
For further investigation, please turn on all the diagnostics for Address Searcher.

Condition

These symptoms can occur if the virtual memory usage on the Cisco Unity system is very close to the 2 GB limit. However, the 2 GB threshold is a theoretical limit, and the Cisco Unity system starts to exhibit these problems when the virtual memory usage exceeds 1.65 GB (or 2.4 GB on systems that run Windows Advanced Server and that use the /3GB switch). Problems are more likely to occur on high-end systems (more than 48 ports and 2,500 users) with a high volume of voice call traffic. The virtual memory fragmentation does not affect normal functionality or performance until the Cisco Unity system exceeds the 1.65 GB limit (or 2.4 GB on systems that run Windows Advanced Server and that use the /3GB switch).

Solutions

Cisco Unity 3.0(1) through 4.0(1) Systems on Windows 2000 Server

Choose one of these solutions:

Cisco Unity 3.0(1) through 3.1(4) Systems on Windows 2000 Advanced Server

Choose one of these solutions:

Cisco Unity 3.1(5) through 4.0(1) Systems on Windows 2000 Advanced Server

Choose one of these solutions.

  • Upgrade to Cisco Unity 4.0(3) with SR1 and use the HeapDeCommitFreeBlockThreshold setting (see Cisco Unity 4.0(2) through 4.0(3) and Later Systems for details).

  • Follow the instructions in this section to use the /3GB switch, and make adjustments to the Windows 2000 Advanced Server memory management. These changes increase the amount of virtual memory available on the Cisco Unity system and extend the period before the system exhibits problems from virtual memory fragmentation.

    warning Warning: Even after you enable the /3GB switch, it is critical that you monitor AvCsMgr virtual memory. If underlying problems cause increases in AvCsMgr virtual memory fragmentation, the Cisco Unity system can still exhibit problems. Restart the Cisco Unity system if the Virtual Bytes counter reaches 2.4 GB.

    Note: Do not do this procedure if you run the Windows 2000 Server. This procedure is designed for use only with the Windows 2000 Advanced Server.

Part I – Enable the /3GB Switch

Complete these steps.

  1. On the Cisco Unity system, use Notepad to open the Boot.ini file from <root directory>\Boot.ini, where <root directory> is the directory in which the operating system binaries are installed.

  2. Modify Boot.ini by adding the /3GB parameter to the startup line. For example,

    [boot loader]
    timeout=30
    default=multi(0)disk(0)rdisk(0)partition(2)\WINNT
    [operating systems]
    multi(0)disk(0)rdisk(0)partition(2)\WINNT="Microsoft Windows 2000
    Advanced Server" /fastdetect /3GB
  3. Save the Boot.ini file.

  4. 4. If you use Cisco Unity version 3.1(5), install CiscoUnity3.1(5)_ES20, which is available from Cisco Technical Support. If you use Cisco Unity version 3.1(6) through 4.0(1), continue with Part II – Set Page Table Entry Registry Settings.

Part II – Set Page Table Entry Registry Settings

These registry keys are designed to increase the amount of Page Table Entries to increase system stability on a Cisco Unity System using the /3GB switch.

  1. Start the Registry Editor (Regedit.exe).

    caution Caution: If you change the wrong registry key or enter an incorrect value, this can cause the server to malfunction. Before you edit the registry, confirm that you know how to restore it if a problem occurs. (Refer to the "Restoring" topics in Registry Editor Help.) Note that for Cisco Unity failover, registry changes on one Cisco Unity server must be made manually on the other Cisco Unity server, because registry changes are not replicated. If you have any questions about how to change registry key settings, contact Cisco Technical Support.

  2. If you do not have a current backup of the registry, select Registry > Export Registry File, and save the registry settings to a file.

  3. Expand the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management key.

  4. Double-click the LargeSystemCache value.

  5. Change the value to 0x00000000 (0).

  6. Double-click the PagedPoolSize value.

  7. Change the value to 0x00000000 (0).

  8. Double-click the SystemPages value.

    Note: Keep a record of the current value for SystemPages. You need to restore this value if you remove the /3GB switch.

  9. Change the value to 0xffffffff (4294967295).

  10. Right-click and choose New. Then choose the DWORD value.

  11. Name the value PoolUsageMaximum.

  12. Double-click the PoolUsageMaximum value.

  13. Change the value to 0x0000003c (60).

  14. Close the Registry Editor.

  15. Restart the Cisco Unity server in order for the changes to take effect.

    Note: On Cisco Unity systems that use the /3GB switch and third party voiceboard integrations, problematic behavior has been observed while attempting to install or upgrade the Cisco Unity software. Cisco bug ID CSCdw36215 (registered customers only) describes the symptoms and workaround for this.

Cisco Unity 4.0(2) through 4.0(3) and Later Systems

This procedure makes adjustments to the Windows 2000 memory management in order to provide better control of virtual memory use and to reduce virtual memory fragmentation, without the use of the /3GB switch. It also applies memory allocation fixes available in Cisco Unity 4.0(2) and Cisco Unity 4.0(3).

Note: In Cisco Unity 4.0(4), setup detects when a system has 2 GB or more of RAM, and automatically enables the registry setting described in Part I – Enable the "HeapDeCommitFreeBlockThreshold" Registry Key.

Two known defects contribute to virtual memory fragmentation in Unity 4.0 systems: Cisco bug ID CSCec32012 (registered customers only) and CSCed41828 (registered customers only) . Based on the version of Cisco Unity installed on the server, download the applicable updates from Cisco.com:

  • For Cisco Unity 4.0(3), install the latest Service Release. (Note that SR1 for Cisco Unity 4.0(3) also requires 4.0(3) ES64).

  • For Cisco Unity 4.0(2) systems, where an upgrade to Cisco Unity 4.0(3) and an installation of 4.0(3) SR1 is not an option, install CiscoUnity4.0(2)_ES42 and CiscoUnity4.0(2)_ES43.

Part I – Enable the “HeapDeCommitFreeBlockThreshold” Registry Key

The HeapDeCommitFreeBlockThreshold modification specified in this section is intended for use only with a Cisco Unity system that has 2 GB or more of RAM, that runs on a Windows 2000 Server or Windows 2000 Advanced Server.

  1. Start the Registry Editor (Regedit.exe).

    caution Caution: If you change the wrong registry key or enter an incorrect value, this can cause the server to malfunction. Before you edit the registry, confirm that you know how to restore it if a problem occurs. (Refer to the "Restoring" topics in Registry Editor Help.) Note that for Cisco Unity failover, registry changes on one Cisco Unity server must be made manually on the other Cisco Unity server. This is because registry changes are not replicated. If you have any questions about how to change registry key settings, contact Cisco Technical Support.

  2. If you do not have a current backup of the registry, select Registry > Export Registry File, and save the registry settings to a file.

  3. Expand the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager key.

  4. Double-click the HeapDeCommitFreeBlockThreshold value.

  5. Change the value to 0x00040000 (262,144).

  6. Click OK.

  7. Close the Registry Editor.

  8. Restart the Cisco Unity server for the changes to take effect.

Part II – Remove the /3GB Switch

After you complete Part I, the Cisco Unity server no longer requires the /3GB switch. If the system uses the /3GB switch, be sure to remove the switch from the Boot.ini file and continue with Part III.

  1. On the Cisco Unity system, use Notepad to open the Boot.ini file from <root directory>\Boot.ini, where <root directory> is the directory in which the operating system binaries are installed.

  2. Modify Boot.ini by removing, if present, the /3GB parameter from the startup line.

    For example, change this from:

    [boot loader]
    timeout=30
    default=multi(0)disk(0)rdisk(0)partition(2)\WINNT
    [operating systems]
    multi(0)disk(0)rdisk(0)partition(2)\WINNT="Microsoft Windows 2000
    Advanced Server" /fastdetect /3GB

    to:

    [boot loader]
    timeout=30
    default=multi(0)disk(0)rdisk(0)partition(2)\WINNT
    [operating systems]
    multi(0)disk(0)rdisk(0)partition(2)\WINNT="Microsoft Windows 2000
    Advanced Server" /fastdetect
  3. Save the Boot.ini file.

Part III – Restore the System Page Table Entry Registry Settings

If the registry settings were modified when you enabled the /3GB switch, restore them to the original values now that the /3 GB switch is removed.

  1. Start the Registry Editor (Regedit.exe).

    caution Caution: If you change the wrong registry key or enter an incorrect value, it can cause the server to malfunction. Before you edit the registry, confirm that you know how to restore it if a problem occurs. (Refer to the "Restoring" topics in Registry Editor Help.) Note that for Cisco Unity failover, registry changes on one Cisco Unity server must be made manually on the other Cisco Unity server, because registry changes are not replicated. If you have any questions about changing registry key settings, contact Cisco Technical Support.

  2. If you do not have a current backup of the registry, select Registry > Export Registry File, and save the registry settings to a file.

  3. Expand the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management key.

  4. Double-click the LargeSystemCache value.

  5. Change the value to 0x00000000 (0).

  6. Double-click the PagedPoolSize value.

  7. Change the value to 0x00000000 (0).

  8. Double-click the SystemPages value.

  9. Restore the original value that was in place when the /3GB switch was enabled, or use the default value of 0x00000000 (0).

  10. Click the PoolUsageMaximum value to highlight it.

  11. Choose Edit, and then Delete.

  12. Click Yes.

  13. Close the Registry Editor.

  14. Restart the Cisco Unity server for the changes to take effect.

Error : The virtual memory necessary to run your Exchange server is fragmented in such a way that performance may be affected

In Cisco Unity 4.x with Exchange 2000 SP3, Unity crashes and the user experiences high CPU, slow response times and delay. The Cisco Unity application event log reports this error message:

Event Type: Error
  Event Source: MSExchangeIS
  Event Category: Performance
  Event ID: 9582 The virtual memory necessary to run the Exchange server is fragmented 
  in such a way that normal operation may begin to fail. It is highly recommended that 
  you restart all Exchange services to correct this issue.

This issue can occur when virtual memory fragmentation occurs on your Exchange 2000 computer. Also, this issue can occur when the server has IMAP4 users, and these users log on or log off.

Solution

In order to resolve this issue, refer to the Microsoft KB article 883527 leavingcisco.com.

Related Information

Updated: Jan 31, 2006
Document ID: 51477