Guest

Cisco Unity

Troubleshooting Cisco Unity Audio Quality

Document ID: 41042

Updated: Feb 28, 2008

   Print

Introduction

This document describes techniques to troubleshoot problems that you might experience with Cisco Unity audio quality. Audio quality for the Cisco architecture for voice, video and integrated data (AVVID) solution is a perceptual measure of how good audio sounds as it reaches the intended recipient. With any subjective evaluation, it is most important to realize the measurable areas of audio quality and identify how these areas affect the reported audio quality problem.

For example, if a reported problem is “loud Unity system prompts,” volume can be described as the problem area that impacts audio. If you isolate the investigation to volume levels, you can take the steps in this document to further investigate what part of the environment might change volume levels.

After you categorize the type of audio quality issue, you can determine the source of the audio distortion thorough a process of elimination. This can be a complex process, depending on the number of devices in any AVVID deployment that originate, control, or deliver audio streams to Cisco Unity. Keep in mind the complexity of the environment and the various paths the audio stream take, to more quickly identify the source of audio distortions. Each of the known areas of audio quality issues in this document follow this logic as troubleshooting progresses.

For example, when the problem area that affects audio quality is known to be volume, there are several ways to change volume in the AVVID environment. The Volume Levels and Gain Control section first addresses any potential volume changes Cisco Unity can make to an audio stream; it then covers other known sources of volume modification that can impact audio received by Cisco Unity. If none of the topics in that section correct the audio distortion, check the other listed symptoms for a similar sounding audio issue.

Prerequisites

Requirements

Readers of this document should have knowledge of these topics:

  • How to set Cisco Unity traces

  • How to use Netmon (from the Microsoft Windows 2000 Resource Kit)

  • How to work with the Windows registry

In addition, you should know this information about the Cisco Unity installation, before you complete the troubleshooting in this document:

  • Cisco Unity version number

  • Integration type including the Telephony Application Program Interface (TAPI) service provider (TSP) version number

  • Cisco Unity message recording format (codec type)

  • Codices used within the integration and any other regions of the deployment

  • Installation history (new install, upgrade)

  • Local topology information can be also useful to understand the pathways that audio travels to Unity

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco Unity version 2.4.6.161, 3.x versions through 3.1(5), and 4.x versions through 4.0(2).

  • Netmon Network Analysis Tool from the Windows 2000 Resource Kit

  • Sniffer Pro version 4.5 from Network Associates, Inc. (NAI).

  • Cisco Unity utility AudioStat, available in Cisco Unity 4.0(1) and later versions; and Capripper utility, available in Cisco Unity 4.0(2) and later

Note: Not all are required for every type of audio quality issue described.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to the Cisco Technical Tips Conventions for more information on document conventions.

Audio Distortion Types and Troubleshooting

Note: The links in this document contain the most current information known on the type of audio distortion possible in the environment. Review the symptoms of each before you attempt troubleshooting steps.

Volume Levels and Gain Control

This section describes volume level and gain control troubleshooting techniques.

Symptoms

The symptom of this problem area relates to unexpected changes in the volume level of a voice mail message or volume levels from Cisco Unity measured as “too loud” or “too quiet.”

Volume level and gain control issues are most often reported when audio is too quiet in a message for the user to understand the content. There are also possible fluctuations of volume that could be reported as creating a “garbled” message, where the volume goes up or down regardless of the volume of a speaker’s voice (for example: Audio Volume Fluctuation on Phone Load).

Cisco Unity versions 3.1(2) and later also contain an automatic gain control (AGC) feature, which might contribute to fluctuations in volume if incorrect settings are introduced in the configuration. This is most often noted by large increases in volume at the end of a message (for example: Audio Volume Fluctuation from Incorrect AGC Settings).

Finally, adjustments to recording or playback gain where AGC is used can modify the incoming audio stream before it reaches the AGC operation. This can cause extreme volume levels in the message (for example: Audio Volume Extremes from Incorrect TSP Settings).

Troubleshooting

Audio volume distortion can occur at any point in a voice stream, from an originating cellular phone on an external call through to the IP phone that receives a message from that caller. Volume gain might be distorted with an increase or decrease in volume through these devices:

  • IP Phone (no configurable gain)

  • Analog gateways (gain increase or decrease)

  • Digital Signal Processors (DSPs) (gain increase or decrease)

  • Unity TSP (either on-board AGC through a Dialogic TSP or playback and record gain values in Cisco Unity TSP registry values)

  • Unity AGC (increase and decrease through AGC)

  • PSTN devices (unknown, and usually not programmatically altering volume, although “noise-reducing” microphones can adjust input gain)

Cisco Unity versions later than 3.1(2) with AGC enabled have a target nominal volume level of –26 decibels by default. When other devices in the deployment might also control gain levels, it can be easiest to isolate a volume issue by first disabling these settings on gateways that deliver audio to Cisco Unity.

Note: When the gain levels vary on different gateways, the settings should be noted so that you can return the levels to the original values after you troubleshoot (if needed).

Note: Ensure that you do not increase the volume of speakers or headphones while you are troubleshooting this type of issue.

Use these steps to troubleshoot volume level and gain control issues:

  1. Identify which audio streams are affected by the volume problems.

    Are all audio streams from Cisco Unity at a low volume? If only a particular part of an audio stream is affected (for example, only Cisco Unity system prompts or only messages), proceed to Step 2.

    1. When all audio from Cisco Unity is at a low volume and phone-to-phone volumes are low, first check gain settings on gateways in the environment.

      Use Cisco Unity to validate volume levels:

      1. During a maintenance window, enable the Cisco Unity Diagnostic Tool (UDT) for MiuIO Miscellaneous (23).

        ts_unityaudioq_01.gif

        This diagnostic reports the input power levels of all incoming audio samples, as shown in this example:

        MiuIO  23  [Thread 0x00000EB0] [Port 2] [AvWav: Miscellany]
        Power = -31.012355dB  Gain adjustment = 5.000000dB.
      2. From a single source (a local IP phone, for example), leave messages for Cisco Unity with similar volume levels. Applying a steady speaking voice to say the same 10 to 15 second phrases.

        An internal call should serve as the “control” point for volume on the IP network. Calls that are made out through the PSTN and back into Cisco Unity show the volume levels that come through any number of analog gateways.

      3. After you complete all of the test calls, stop the UDT trace and gather the diagnostic for the time period that the calls were made.

      4. Find the individual calls by their timestamp in the diagnostic, and view the Power and Gain adjustment settings for each call.

      5. If calls consistently have a Gain adjustment of �5db or greater, either the gain on the gateways need adjustment or the Cisco Unity TSP registry settings should be checked (proceed to Step 2).

    2. When only audio from Cisco Unity is at a low volume, check the TSP playback and record volume levels from the registry key on the Unity server:

      1. Run Regedit and verify the keys in HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\Cisco TSP.

      2. The WaveDBGainPlayback and WaveDBGainRecord keys should both have the value of 0.

        ts_unityaudioq_02.gif

      A clear example of when this setting can affect audio is when a gateway has a positive gain applied and the WaveDBGainRecord has a negative value. The volume level is incorrectly being adjusted at the gateway and Cisco Unity. A volume increase applied at the gateway followed by a volume decrease at the TSP could result in the same lower volume level going into Cisco Unity as that which entered the gateway.

      In the same way, when a gateway has negative gain applied and WaveDBGainRecord has a negative value, any value on the WaveDBGainPlayback could be useless to adjust low volume messages played from Unity.

      Finally, a negative WaveDBGainRecord value with a negative WaveDBGainPlayback value will always decrease the volume of any audio played out of Unity. Older TSP versions (pre 3.0) might retain a WaveDBGainRecord value of 5, which causes distorted volume levels on recorded messages.

      Note: If the Cisco Unity version is 3.1(2) or later and AGC is active, the WaveDBGainPlayback and WaveDBGainRecord values should always be 0. As with the multiple points of volume adjustment described in the previous paragraph, if the audio volume level is already manipulated by AGC, then it is not recommended that you adjust either recording or playback volume levels in the TSP.

  2. Check if the registry setting values for AGC are set incorrectly. The bad values are usually:

    • AGCsamplesize is 4e20 hex (20000 decimal) and should be 1f40 hex (8000 decimal)

    • AGCgainthreshold is 28 hex (40 decimal) and should be 5 hex (5 decimal)

    If so, change the registry settings to the correct values, as described in the AGC Registry Keys table that can be found in the Audio Quality document.

    Note: If the quality of the prompts are unacceptable when you use the G.729 codec, you can change the codec to G.711 for better audio quality.

  3. Identify which part of the audio stream is affected by the volume change: Cisco Unity system prompt, messages left for subscribers from outside the local network deployment (analog or digital), or messages left for subscribers from other subscribers.

    When only messages from callers outside of the VoIP deployment have volume levels above or below messages left by subscribers or Cisco Unity system prompts, verify gain levels on gateways that deliver this audio:

    1. If Cisco Unity Legacy PBX integrations are present, refer to Cisco Unity-Legacy PBX Integration: Adjusting the Volume in Registry for more information.

    2. If all messages are affected, but Cisco Unity system prompts are at an acceptable level, check the version of Unity currently running.

      AGC default settings should be checked, if the Cisco Unity version is 3.1(2) or later. Refer to these documents for more information:

      You should verify prompts and TSP gain levels, if the Unity version is earlier than 3.1(2). Refer to Audio Decision Tree for Cisco Unity 2.46 and 3.0x for more information.

    3. If only some messages are affected, verify the phone load deployed for DDTs issue Cisco bug ID CSCdy27331 (registered customers only) .

    4. See the Connectivity, Jitter, Packet Delay, and the ‘Garbled Message’ section of this document, if the volume level is fluctuating quickly and consistently within any message. Even though the volume is actually zero in many parts of these types of distorted messages, it can be perceived as a volume fluctuation issue.

    5. If only some subscribers are affected by different volume levels on phone and computer playback, refer to Cisco Unity ViewMail for Outlook (VMO) Audio Volume Settings for more information.

Note: If you experience a bad quality for Music on Hold (MOH), make sure that the music audio source files comprise .wav files in one of these formats:

  • 16-bit PCM (stereo or mono) (16k Hz or 32k Hz or 48k Hz or 8k Hz or 44.1k Hz sample rate)

  • 8-bit CCITT g.711 a-law or mu-law (stereo or mono) (8k Hz sample rate)

Related Information and Known Problems

Refer to these documents for more information about volume level and gain control issues:

Connectivity, Jitter, Packet Delay, and the ‘Garbled Message’

This section describes audio connectivity distortion troubleshooting techniques.

Symptoms

“Garbled message” is most often reported when a message left for a Cisco Unity subscriber is either missing enough audio content to be considered unintelligible or when the packets are interleaved such that the content sounds out of order. Rather than longer specific periods of silence in the message, which might be related to volume levels (as previously noted), the garbled message might have other distortions present.

A bad network interface card on a Cisco Unity server is the source of audio distortions heard in this example: Audio Distortion Due to a Faulty Network Interface Controller (NIC).

This example is notable because of the distortion that is inserted for brief intervals at various points in the message. Not only are missing packets the cause of silence insertions, but additional signals from other audio streams—which the NIC is delivering incorrectly—are heard as “pops” throughout the message.

Notice the difference in this example, which contains distortions due to packet delay in the network: Audio Distortion Due to Packet Delay.

This example is notable for the silence insertions—during large packet delays—that are increasing throughout the message.

Troubleshooting

Audio connectivity distortions can also occur at several points in a network from a gateway receiving an external call through to the IP phone that receives a message from that caller. Use these steps to troubleshoot audio connectivity distortion issues:

  1. Identify which audio streams are affected by the garbled audio.

    Are all audio streams from Cisco Unity garbled?

    1. When only Cisco Unity system prompts are garbled, verify the fix for garbled prompts playing from Unity through a Catalyst 6000, refer to DDTs issue Cisco bug ID CSCdx36894 (registered customers only) . The distortion associated with this defect is heard in this example: Unity Garbled Prompts Distortion.

    2. Where all audio from Cisco Unity is garbled and phone-to-phone audio is also garbled, verify the duplex settings on the Unity NIC configuration and on the switch port to which Unity is connected. These should not be set to auto-negotiate, but rather to hard-coded values (usually 10/100 full duplex).

      This eliminates the possibility that auto-negotiation of packet delivery is a cause or contributor to distortion of the audio streams.

    3. If the distortion continues after taking the step above, check the network topology and available bandwidth to Cisco Unity. Check relevant gateways for alignment errors.

    4. If alignment errors are found on any switch ports, and the Cisco Unity server is an HPQ DL380G2 (MCS7837, MCS7847), then the distortion could be associated with an incorrect NIC driver. These errors are typically reported as alignment errors, runt packets, or Frame Check Sequence (FCS) errors by switches. The NC-Series NIC statistics might report the errors as either Transmit Underruns or Receive Overruns. Check the N100NT.SYS driver on the Cisco Unity server. If it is version 5.30.74.001, the solution is to install an older version of the driver from: http://welcome.hp.com/country/us/en/support_task.html leavingcisco.com.

    5. If the distortion continues or the NIC driver downgrade does not apply to the hardware, the audio stream that is entering the Cisco Unity server must be validated for any distortion per the instructions in the next step (Step 2).

  2. Identify whether the source of distortion is internal or external to Cisco Unity.

    This is a simplified description of the audio stream path from the Cisco Unity NIC to a wave file:

    1. Incoming audio stream from the network —> NIC

    2. NIC —> Cisco Unity’s avaudio.sys

    3. Cisco Unity’s avaudio.sys —> (through Microsoft Windows wave driver) to UnityAvWav

    4. UnityAvWav converts the stream to PCM, applies AGC, and converts the audio to the destination codec format —> temporary file created

    5. Temporary file completed at end of recording —> wave file created and passed to the mailstore, after leading or trailing silence is trimmed

    When you identify the inconsistencies in the audio stream based on the different paths it takes before it becomes a wave file delivered through email, you can isolate the source of distortion to one of these reasons:

    • An external source of audio distortion

    • A defective NIC card

    • A bandwidth-related distortion that impacts the wave driver logic while collecting the incoming audio stream

    An objective comparison of the audio streams as they pass through the components is necessary to confirm any of these sources.

    caution Caution: You should not use the methods in Steps 3 and 4 as general troubleshooting methods, if other steps have not been attempted. To capture audio traffic without discretion is not recommended, and it might affect the AVVID network.

    caution Caution: You should use a maintenance window when you do Steps 3 and 4, as the actions could impact service.

    Use a test phone and test subscriber to obtain the best objective measure of audio quality, unless a specified endpoint has been identified as the source of distortion.

  3. Collect the incoming audio stream from the network to the NIC, for a comparison of external audio to audio written to the mailstore.

    Span the Unity port at the network device to which Cisco Unity is connected, and collect the packets through a Sniffer. You can also use Netmon (from the Microsoft Windows 2000 Resource Kit) from any Windows 2000 server that is external to the Unity server, to collect the data going to the Unity server.

    Note: It is recommended that you only run Netmon on the Cisco Unity server itself after the audio stream has been collected external to the Unity server NIC.

    For packet captures, use a capture filter to take UDP packets from any source with a destination of the Cisco Unity server IP address. Save the captures from the testing time period, to compare them to the data that you will collect in the next procedure.

    1. In order to collect the incoming audio to the Cisco Unity wave driver—Use Netmon on the Cisco Unity server to collect the data incoming UDP audio streams. After Netmon is installed, complete these steps to configure Netmon:

      1. Copy the RtpParser.dll file to …\WINNT\system32\NETMONFull\PARSERS. This file is available in the Audio directory of the Cisco Unity \Commserver\Utilities directory.

      2. Choose Start > Control Panels > Administrative Tools > Network Analysis Tools, to open Netmon.

      3. Select the appropriate interface and capture only UDP traffic to Unity.

    2. In order to collect the audio stream the wave file created (written by the Cisco Unity wave driver)—Most often, problem audio is reported from a user directly as they receive the voice mail(s) that contain distortion. If so, request these messages to help isolate the issue. For further troubleshooting, it is best to create a test subscriber account that you can access from a mail client, so that you can save the voice mail wave files to disk. If this is not possible, have the messages forwarded from this test mailbox to an outside address, to assist in the collection of the wave files. Finally, if neither of these are possibilities (where a voice-mail only system is in place, for example), this procedure allows for direct access to the wave files before they are delivered:

      1. Stop the Microsoft Exchange Information Store service from the Exchange mailstore with which Cisco Unity is homed.

        This places Cisco Unity into the Unity Message Repository (UMR) mode, where all messages sent to subscribers go first to the …\Commserver\UnityMTA directory. For more information, refer to How to Start Cisco Unity in UMR Mode.

      2. Collect the resulting wave files from this directory after test messages are sent.

        caution Caution: No messages are delivered on the Exchange mailstore during this time. This procedure impacts any incoming call—use only during a maintenance window.

      3. Restart the Microsoft Exchange Information Store service after the tests are completed.

  4. In order to compare the collected data, use the Capripper utility to convert the packet capture files into wave files (also found in the Audio directory of the \Commserver\Utilities directory in Cisco Unity version 3.1(5) and later):

    1. Copy the capripper.exe file and the cap file to be decoded into a new directory.

    2. Drag the .cap file containing the test audio streams onto the capripper.exe to create a series of .wav files named for the endpoints in each call, as shown in these screens:

      ts_unityaudioq_03.gif

      ts_unityaudioq_04.gif

    3. Listen to the test calls from the packet capture and from the mailstore. If there is any difference between the stream collected from the network to the NIC and the stream collected from the NIC to the Unity wave driver, there is likely a hardware issue with the NIC.

      For instance, this example—Packet Capture Encoded to a Wave File—collected from traffic to the NIC, compared with this example—Unity Message Wave File—collected from the Cisco Unity system, shows an issue with a NIC hardware failure. In this case, if the server has a dual-NIC configuration, disable the currently used NIC and enable the second NIC. If no more reports of distortion occur, the NIC is the source of the audio distortion. If the server has only one NIC available, install a PCI-NIC where possible and switch all traffic to run through this interface. If no further distortion is heard, contact Cisco Systems Technical Support to report the defective equipment.

      When the previous steps show that audio that enters the Unity server and passes through the NIC is of acceptable quality, but distortions still exist in the wave file written to the drive, you must collect the audio stream with Netmon. Determine the arrival time of the packets for the audio stream. If there is more than 20ms of delay between packets, the wave driver might insert silence into the wave file as a normal operation. If packet captures show delay as the cause of multiple silence insertions, DDTs issue Cisco bug ID CSCdx41866 (registered customers only) , addressed in Cisco Unity version 3.1(5), might broaden the window for packets to be received by the wave driver. Although you can set a greater packet delay with this registry key, this indicates unacceptable delay in the network that you should investigate.

      In order to investigate the delay, run the AudioStat utility (found in Cisco Unity versions 4.x and later) from the Audio directory of the \Commserver\Utilities directory. This tool displays average delays and silence insertions for each recording along with the source IP address of the device that is sending audio to Cisco Unity.

      These screens show the AudioStat utility:

      ts_unityaudioq_05.gif

      ts_unityaudioq_06.gif

      If any single device shows increased latency, investigate that link in the network. The AudioStat utility helps to isolate where in the topology latency is introduced.

      Another frequent cause of jitter and choppy voice messages is when Voice Activity Detection (VAD) is enabled. You can overcome this when you turn off VAD if bandwidth is not an issue.

Related Information and Known Problems

Refer to these documents for more information about audio connectivity distortion issues:

DTMF Recognition and Voicemail Responsiveness including Audio Transcoding

This section describes dual tone multifrequency (DTMF) recognition and voice mail responsiveness troubleshooting techniques.

Symptoms

DTMF recognition and voice-mail response issues that affect audio quality are most often reported as one-way audio during a call to Cisco Unity or a lack of response from the voice mail system after digit entry by the caller. The voice mail responsiveness issues can sometimes be correlated to the integration factors listed later in this section, but they most often might depend on Exchange deployment configuration as described in Cisco Unity: Delays in the Subscriber Conversation.

Troubleshooting

Silence suppression, voice activity detection (VAD), and comfort noise might also contribute to message clipping or cut off (where the system is recognizing silence as a call termination event). This section contains a list of documents to assist with configuration and troubleshooting.

Refer to these troubleshooting documents, if the Cisco Unity integration involves a legacy PBX:

Refer to these troubleshooting documents, if the Cisco Unity integration involves Cisco CallManager through the Cisco-Unity TSP:

Related Information and Known Problems

For more information about DTMF recognition and voice mail responsiveness issues, refer to these documents:

Echo

This section describes echo troubleshooting techniques.

Symptoms

Echo is included here as a general audio quality issue that affects AVVID deployments, which should be familiar to anyone that troubleshoots audio quality issues. It has yet to be the type of issue that could ever impact a message left on a Cisco Unity system. Echo can occur during an IP call when the distortion is generated through digital or analog gateways.

Troubleshooting

For more information on the sources of echo and troubleshooting, refer to these documents:

Related Information

Updated: Feb 28, 2008
Document ID: 41042