Troubleshooting DSPs on the PA-VXA/PA-VXB/PA-VXC T1/E1 Voice Port Adaptors and the PA-MCX MIX-Enabled Multichannel T1/E1 Port Adaptors for Cisco 7200/7400/7500 Series Routers

Introduction

This document discusses techniques used to verify the basic functionality of the Digital Signal Processors (DSPs) on the PA-VXA/PA-VXB/PA-VXC Voice Port Adaptors for the Cisco 7200/7300/7400/7500 router platforms. The DSPs are necessary for packet telephony technologies such as Voice over IP (VoIP), Voice over Frame-Relay (VoFR), and Voice over ATM (VoATM). The DSPs are responsible for the conversion of voice from analog to digital forms and the other way around, to set the gain and attenuation parameters, for the operation of Voice Activity Detection (VAD), and more. Proper hardware and software operation of the DSPs is necessary to ensure that calls can be established and maintained correctly. This document also includes discussion of how to identify and troubleshoot the DSPs which are used by the PA-MCX MIX-enabled Multichannel T1/E1 Port Adaptors for the Cisco 7200 router platform, when these voice-capable Port Adaptors are used for voice termination. When used for voice termination, the PA-MCX T1/E1 Port Adaptors dynamically obtain DSP resources from a PA-VXA/PA-VXB/PA-VXC Voice Port Adaptor installed on the same Cisco 7200 router.

For more information on the PA-VXA/PA-VXB/PA-VXC Voice Port Adaptors refer to Understanding the PA-VXA/VXB/VXC Voice Port Adaptors for the Cisco 7200/7300/7400/7500 Voice Gateways.

For more information on the PA-MCX MIX-enabled Multichannel Port Adaptors refer to Mix-Enabled T1/E1 Port Adapter for Cisco 7200VXR Series Routers.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the software and hardware versions below.

• A PA-VXA/VXB/VXC Voice Port Adaptor installed in a suitable Cisco 7200/7300/7400/7500 Voice Gateway running an appropriate Cisco IOS® Software Release to support the port adaptor.

For more information, refer to Voice Gateway Hardware Compatibility Matrix (Cisco 7200, 7300, 7400, 7500).

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Problem

These symptoms can be attributed to DSP hardware or software issues:

• No audio is heard by either party, or there is only one-way audio on the voice path after the call is connected.

• Call setup failure, such as the inability to detect or transmit proper Channel Associated Signaling (CAS) state transitions.

• Voice ports are stuck in the PARK state and cannot be used.

• Error messages, on the console or in the router log, that complain of DSP timeouts.

Solution

If you have experienced some of the previously described problems, you might see DSP timeout messages in the router log, such as these:

*Jun 23 23:50:09.313: %VTSP-3-DSP_TIMEOUT: DSP timeout on event 6:
                      DSP ID=0x1: DSP error stats, chnl info(1, 16, 0)

*Jun 23 23:50:09.313: %VTSP-3-DSP_TIMEOUT: DSP timeout on event 6:
                      DSP ID=0x1: DSP error stats, chnl info(1, 16, 0)

These messages indicate that the response from the DSP resource (1, 16, 0) might not be as it should and it might not be able to handle packet voice calls. The three DSP resource parenthetical digits are represented as (N, D, C) and are interpreted this way:

• N—The reference slot number where the PA-VXA/PA-VXB/PA-VXC is installed on the router.

• D—A DSP number on the port adaptor.

• C—The channel number on that DSP.

Follow the steps in the remainder of this document to resolve the problem.

Step 1: Issue the test dsprm Command

Issue the hidden test dsprm N command in the enable mode to query the DSPs. This command determines if the DSPs are responsive.

Note: A hidden command is one that can not be parsed with a ? command and for which the Tab key cannot be used to auto-complete the command. Hidden commands are not documented and some of the output is used strictly for engineering purposes. Hidden commands are not supported by Cisco.

The value of N for the command depends upon the router platform where the PA-VXA/PA-VXB/PA-VXC resides. Use this table to obtain the value of N:

Platform	Cisco 7200/7300/7400	Cisco 75001
N	The bay number in which the port adaptor resides	Cisco IOS Software Releases earlier than 12.2(13.4), 12.2(13.4)T	2 × Versatile Interface Processor (VIP) slot number + the bay number in which the port adaptor resides
		Cisco IOS Software Releases 12.2(13.4), 12.2(13.4)T and later, per Cisco bug ID CSCdx95752 (registered customers only)	VIP slot number/the bay number in which the port adaptor resides

Note:  ¹ The PA-VXB-2TE1+ and PA-VXC-2TE1+ Voice Port Adaptors might have special VIP revision requirements for full support. Refer to Field Notice: PA-2FE-TX, PA-2FE-FX, PA-VXC-2TE1+, and PA-VXB-2TE1+ Incompatible with Some Older VIP2-50 Versions for details.

For example, for a Cisco 7200 router with a PA-VXC-2TE1+ in port adaptor bay number 3, you issue the test dsprm 3 command. For a Cisco 7500 router with a PA-VXC-2TE1+ in port adaptor bay number 1 on a VIP in slot number 4, N = 2 × 4 + 1 = 9. Therefore, you issue the test dsprm 9 command or the test dsprm 4/1 command, based on the Cisco IOS software release in use.

The next output example is from the hidden test dsprm N command for a Cisco 7200 router with a PA-VXC-2TE1+ in bay number 1, with Cisco IOS Software Release 12.2(12).

Note: If you use a console to access the gateway, logging console must be enabled to see the command output. If you use Telnet to access the router, terminal monitor must be enabled to see the command output.

7200_Router# test dsprm 1

Section:
1 - Query dsp resource and status
2 - Display voice port's dsp channel status
3 - Print dsp data structure info
4 - Change dsprm test Flags
5 - Modify dsp-tdm connection
6 - Disable DSP Background Status Query and Recovery
7 - Enable  DSP Background Status Query and Recovery
8 - Enable DSP control message history
9 - Disable DSP control message history
q - Quit

Select option 1 from the menu. This triggers the Cisco IOS Software to ping the DSP and then wait for a response from it. If a response is received, the DSP is ALIVE message is displayed, which declares that the DSP functions properly. If the Cisco IOS Software did not receive a response, the DSP is not responding message is displayed.

Caution: You should only use option 1 from the test dsprm N command. If you select other options, you might cause your router to reload or cause other problems to occur.

This is an example of the output that is generated after you select option 1 from the menu:

Select option : 1

Dsp firmware version: 3.4.52
Maximum dsp count: 30
On board dsp count: 30
Jukebox available
Total dsp channels available 120
Total dsp channels allocated 48
Total dsp free channels 72
Quering dsp status......
MS-7206-12A#
*Jun 23 23:56:13.463: dsp 0 is ALIVE
*Jun 23 23:56:13.463: dsp 1 is ALIVE
*Jun 23 23:56:13.463: dsp 2 is ALIVE
*Jun 23 23:56:13.463: dsp 3 is ALIVE
*Jun 23 23:56:13.463: dsp 5 is ALIVE
*Jun 23 23:56:13.463: dsp 6 is ALIVE
*Jun 23 23:56:13.463: dsp 7 is ALIVE
*Jun 23 23:56:13.463: dsp 8 is ALIVE
*Jun 23 23:56:13.463: dsp 10 is ALIVE
*Jun 23 23:56:13.463: dsp 11 is ALIVE
*Jun 23 23:56:13.463: dsp 12 is ALIVE
*Jun 23 23:56:13.463: dsp 13 is ALIVE
*Jun 23 23:56:13.463: dsp 14 is ALIVE
*Jun 23 23:56:13.463: dsp 16 is not responding
*Jun 23 23:56:13.463: dsp 17 is ALIVE
*Jun 23 23:56:13.463: dsp 18 is ALIVE
*Jun 23 23:56:13.463: dsp 20 is ALIVE
*Jun 23 23:56:13.467: dsp 21 is ALIVE
*Jun 23 23:56:13.467: dsp 22 is ALIVE
*Jun 23 23:56:13.467: dsp 23 is ALIVE
*Jun 23 23:56:13.467: dsp 24 is ALIVE
*Jun 23 23:56:13.467: dsp 25 is ALIVE
*Jun 23 23:56:13.467: dsp 26 is ALIVE
*Jun 23 23:56:13.467: dsp 27 is ALIVE
*Jun 23 23:56:13.467: dsp 28 is ALIVE
*Jun 23 23:56:13.467: dsp 29 is ALIVE
*Jun 23 23:56:13.467: dsp 4 is ALIVE
*Jun 23 23:56:13.467: dsp 15 is ALIVE
*Jun 23 23:56:13.467: dsp 19 is ALIVE
*Jun 23 23:56:13.467: dsp 9 is ALIVE
7200_Router#

In the output from option 1, note the DSP firmware version number and the number of onboard DSPs. Count the number of DSPs that report as ALIVE and ensure that this number matches the number of onboard DSPs. DSPs should either report as ALIVE or not responding. Occasionally, a DSP will not respond at all. If a DSP does not respond, determine the number of the DSP (D) that is absent from the output. In the previous example, all DSPs are ALIVE except DSP number 16, which reports as not responding. This indicates that the DSP is faulty, which can be due to either a hardware or a software issue.

Step 2: Issue the show voice dsp Or show voice dsploc Command

This step is optional, but it is useful to correlate problem T1/E1 timeslots with unresponsive DSPs. From Step 1, you know that DSP 16 is not responding and that you are logging DSP timeout messages for DSP 16. You can issue the show voice dsp command to view how timeslots and DSP resources are allocated by the Cisco 7200/7400/7500. That command also monitors this information:

• Timeslot (TS) to DSP (DSP NUM) and DSP channel (CH) mappings

• Transmit (TX) and receive (RX) packet counters

• Number of DSP resets (RST) per DSP

• DSP firmware version

• Current voice codec in use

• Current state of the DSP channel

In the next output example of the show voice dsp command, timeslot 06 is mapped to DSP 016 on the T1 CAS link. You can monitor timeslot usage on an T1/E1 voice link via the PBX and on the router, to determine which timeslots have voice problems. If a call is placed over timeslot 6 on this particular T1 CAS link, it is likely that the calling or called party local to this voice gateway will experience dead-air audio or a CAS signal problem.

7200_Router# show voice dsp

DSP  DSP             DSPWARE CURR  BOOT                         PAK     TX/RX
TYPE NUM CH CODEC    VERSION STATE STATE   RST AI VOICEPORT TS ABORT  PACK COUNT
==== === == ======== ======= ===== ======= === == ========= == ===== ============
C549 000 00 g729r8    3.4.52 busy  idle      0  0 1/0:12    13     0  19468/19803
C549 001 00 g729r8    3.4.52 busy  idle      0  0 1/0:14    15     0  19467/19790
C549 002 00 g729r8    3.4.52 busy  idle      0  0 1/0:10    11     0  19463/19802
C549 003 00 g729r8    3.4.52 busy  idle      0  0 1/0:2     03     0  19462/19813
C549 004 00 g729r8    3.4.52 busy  idle      0  0 1/0:17    18     0  19459/19807
C549 005 00 g729r8    3.4.52 busy  idle      0  0 1/0:21    22     0  19459/19786
C549 006 00 g729r8    3.4.52 busy  idle      0  0 1/0:18    19     0  19445/19788
C549 007 00 g729r8    3.4.52 busy  idle      0  0 1/0:4     05     0  19441/19780
C549 008 00 g729r8    3.4.52 busy  idle      0  0 1/0:15    16     0  19440/19759
C549 009 00 g729r8    3.4.52 busy  idle      0  0 1/0:20    21     0  19438/19774
C549 010 00 g729r8    3.4.52 busy  idle      0  0 1/0:9     10     0  19489/19824
C549 011 00 g729r8    3.4.52 busy  idle      0  0 1/0:3     04     0  19486/19845
C549 012 00 clear-ch  3.4.52 busy  idle      0  0 1/0:23    24     0  19481/19812
C549 013 00 g729r8    3.4.52 busy  idle      0  0 1/0:7     08     0  19479/19806
C549 014 00 g729r8    3.4.52 busy  idle      0  0 1/0:0     01     0  19467/19814
C549 015 00 g729r8    3.4.52 busy  idle      0  0 1/0:1     02     0  19464/19796
C549 016 00 g729r8    3.4.52 busy  idle      0  0 1/0:5     06     0  19464/19795
C549 017 00 g729r8    3.4.52 busy  idle      0  0 1/0:13    14     0  19454/19785
C549 018 00 g729r8    3.4.52 busy  idle      0  0 1/0:8     09     0  19446/19797
C549 019 00 g729r8    3.4.52 busy  idle      0  0 1/0:22    23     0  19443/19778
C549 020 00 g729r8    3.4.52 busy  idle      0  0 1/0:6     07     0  19437/19764
C549 021 00 g729r8    3.4.52 busy  idle      0  0 1/0:19    20     0  19421/19765
C549 022 00 g729r8    3.4.52 busy  idle      0  0 1/0:11    12     0  19472/19791
C549 023 00 g729r8    3.4.52 busy  idle      0  0 1/0:16    17     0  19449/19792

7200_Router#

Note: Unlike DSP resources on an NM-HDV module, DSP resources on the Cisco 7200/7300/7400/7500 are not allocated to T1/E1 timeslots at router boot-time. On the Cisco 7200/7300/7400/7500 routers, a DSP resource is dynamically allocated to a timeslot when a packet voice call is made. The show voice dsp command only displays DSP channel-to-timeslot maps for active voice calls.

When you use PA-MCX MIX-enabled Multichannel T1/E1 Port Adaptors to terminate voice traffic, you must determine timeslot-to-DSP resource maps in a different manner. The PA-MCX Port Adaptors do not have any DSP resources of their own, so they draw on free DSPs from a PA-VXA/PA-VXB/PA-VXC Voice Port Adaptor that is installed on the same Cisco 7200 voice router, to obtain their DSP resources. The PA-VXA/PA-VXB/PA-VXC Voice Port Adaptors, conversely, always use one of their own DSPs for their own voice ports and can not farm them from other similar Voice Port Adaptors. Refer to Voice Gateway Application for the Cisco 7200 Series for more information.

In some cases, there will be multiple PA-VXA/PA-VXB/PA-VXC Voice Port Adaptors installed. Even though you know the timeslot on a PA-MCX T1/E1 voice port that has problems, it is difficult to determine which DSP is actually mapped to that particular timeslot. The algorithm by which DSPs are farmed out to PA-MCX T1/E1 voice ports is actually quite simple to understand. For each new voice call placed on a PA-MCX T1/E1 voice port, Cisco IOS software searches, in sequential order, for a free DSP from a PA-VXA/PA-VXB/PA-VXC Voice Port Adaptor installed in chassis Slot1, then Slot2, then Slot3, and so forth, until all chassis slots are exhausted.

When PA-MCX Port Adaptors are used for voice termination, and you know which timeslot on a particular T1/E1 voice port is suspected as related to a problem DSP, you can issue the hidden show voice dsploc command to display a table similar to show voice dsp. The show voice dsploc command is available in Cisco IOS Software Releases 12.2(15)T later, and can subsequently also be found in Cisco IOS Software Release 12.3 mainline and 12.3T trains.

This is an output example of the show voice dsploc command, which is edited to show only voice ports from a PA-MCX Port Adaptor:

7206VXR-A# show voice dsploc

DSP  DSP  DSP            DSPWARE CURR  BOOT                       PAK     TX/RX
TYPE FARM NUM CH CODEC   VERSION STATE STATE RST AI VOICEPORT TS ABORT  PACKCOUNT
==== ==== === == ======= ======= ===== ===== === == ========= == ===== ============
C549   2  013 01 g729r8   4.3.15 busy  idle    0  0 4/0:4     04     0  78291/79579
C549   2  014 01 g729r8   4.3.15 busy  idle    0  0 4/0:0     24     0  78285/79585
C549   2  015 01 g729r8   4.3.15 busy  idle    0  0 4/0:2     02     0  78247/79516
C549   2  016 01 g729r8   4.3.15 busy  idle    0  0 4/0:3     03     0  78128/79408
C549   2  017 01 g729r8   4.3.15 busy  idle    0  0 4/0:1     01     0  78043/79336
C549   2  018 01 g729r8   4.3.15 busy  idle    0  0 4/0:5     05     0  78027/79280

This particular output is from a Cisco 7206VXR voice router where a voice port has been configured on a PA-MCX-8TE1 (installed in chassis Slot4) and a PA-VXC-2TE1+ has been installed in chassis Slot2. As you can see from the DSPFARM column, timeslots 1 through 5 and 24, on voice ports 4/0:0 through 4/0:5, are mapped to DSPs 13 through 18 on the PA-VXC-2TE1+ in chassis Slot2.

Step 3: Issue the dspint DSPfarm Command

You can reset an individual DSP on the PA-VXA/PA-VXB/PA-VXC Voice Port Adaptors to restart the DSP. Issue the dspint DSPfarm command in configuration mode, to reset an individual DSP. This is an output example of a manual reset of DSP 16:

7200_Router# configure terminal

Enter configuration commands, one per line.  End with CNTL/Z.

7200_Router(config)# dspint DSPfarm 1/0

7200_Router(config-dspfarm)# ?

DSP farm configuration commands:
  codec          Configure DSP codec
  default        Set a command to its defaults
  description    Interface specific description
  exit           Exit from dspfarm configuration mode
  load-interval  Specify interval for load calculation for an interface
  no             Negate a command or set its defaults
  reserve        Number of DSP(s) reserved for it's own PA
  reset          Reset DSP(s)
  shut           Shutdown DSP(s)

7200_Router(config-dspfarm)# reset ?

<0-30> List of DSPs to reset
<cr>

7200_Router(config-dspfarm)# reset 16

7200_Router(config-dspfarm)#
* Jun 23 23:59:18.227: %DSPRM-5-UPDOWN: DSP 16 in slot 1, changed state to up
7200_Router(config-dspfarm)#

7200_Router(config-dspfarm)# ^Z

7200_Router#
*Jun 24 19:07:06.527: %SYS-5-CONFIG_I: Configured from console by console
7200_Router#

After you complete this step, the DSP should function as expected and should be able to process voice calls again. If DSP error messages continue, note which DSP is affected and repeat the DSP reset process. Proceed to Step 4, if the DSP reset process does not solve the problem.

Step 4: Verify the Port Adaptor Software and Hardware

You need to determine whether there is a software or hardware problem with the PA-VXA/PA-VXB/PA-VXC Voice Port Adaptor, if you still receive DSP error messages.

Caution: You should schedule a maintenance window to perform the Online Insertion and Removal (OIR) procedures that are described in this section, because unexpected results might occur during the process.

If you are running a Cisco IOS software release with a DSP firmware version earlier than 3.4.49 or 3.6.15, then the problem might be due to a known DSP firmware issue (Cisco bug ID CSCdu53333 (registered customers only) ). If this is the case, you should upgrade your Cisco IOS software to a release in which the bug is resolved, so that this defect can be eliminated as a possible cause. As part of the Cisco bug ID CSCdu53333 (registered customers only) solution, a recovery algorithm is included. When a Voice Telephony Services Provider (VTSP) timeout message is generated by the Cisco IOS Software, the DSP automatically resets itself in an attempt to recover from the cause of the timeout because, in most instances, the timeout occurs on the PA-VXA/PA-VXB/PA-VXC when the DSP is not responding.

Subsequent to the resolution of Cisco bug ID CSCdu53333 (registered customers only) , Cisco bug ID CSCin79311 (registered customers only) resolved a defect with the DSP recovery mechanism. Prior to the fix, even when automatic DSP recovery was enabled, the DSPs on a PA-VXA/PA-VXB/PA-VXC Voice Port Adaptor might not actually recover from a DSP crash event. The resolution for Cisco bug ID CSCin79311 (registered customers only) is available in Cisco IOS Software Releases 12.3(10a), 12.3(12), 12.3(11)T, and later releases from these trains.

If you are running a Cisco IOS Software release with a DSP firmware version in which the solutions from Cisco bug IDs CSCdu53333 (registered customers only) and CSCin79311 (registered customers only) are integrated, remove and re-install the PA-VXA/PA-VXB/PA-VXC from the Cisco 7200/7400/7500; these platforms support OIR. Remember that, on the 7200/7300/7400 platforms, the OIR of an individual port adaptor is supported. However, on the 7500 platform, OIR is supported only for the entire VIP carrier board which houses the port adaptor.

The OIR procedure is a less intrusive step to troubleshoot the problem than a power cycle of the Cisco 7200/7300/7400/7500. If the OIR fails to rectify the DSP problems, then reload the entire router.

If you run a Cisco IOS Software release with a DSP firmware version in which the Cisco bug IDs CSCdu53333 (registered customers only) and CSCin79311 (registered customers only) solutions are integrated, and the OIR step to troubleshoot did not resolve the DSP problems, and a reload of the 7200/7300/7400/7500 also did not resolve the DSP problems, then note whether the same DSPs respond or not.

If you still receive DSP error messages for the same DSPs, then there is most likely a hardware issue and you need to replace the entire PA-VXA/PA-VXB/PA-VXC Voice Port Adaptor. If the unresponsive or missing DSPs differ between manual DSP resets, OIR attempts, or router reloads, then the problem is more likely software-related. For software-related problems, open a case (registered customers only) with Cisco Technical Support, to request an engineer’s assistance to troubleshoot the problem and to offer more direction.

Related Information

• Voice Hardware: C542 and C549 Digital Signal Processors (DSPs)
• Voice Gateway Hardware Compatibility Matrix (Cisco 7200, 7300, 7400, 7500)
• Understanding the PA-VXA/VXB/VXC Voice Port Adaptors for the Cisco 7200/7300/7400/7500 Voice Gateways
• Voice Technology Support
• Voice and Unified Communications Product Support
• Troubleshooting Cisco IP Telephony
• Technical Support - Cisco Systems