Introduction
This document describes solutions to some common digital signal processor (DSP) related isses and ways to determine if the issue is hardware related or not.
DSP is not in UP State
- Collect the output of the show voice dsp group all command.
- Reset the DSPs if necessary and enter the test dsp device <slot> <dsp-id> reset command.
- If the DSPs do not return to an UP state, reboot the router.
- If one of the DSPs fails and cannot be recovered when you use any of these methods, enter the test dsp device <slot#> <DSP_ID> remove | reset | restore command in order to minimize the impact of DSPs that go into a DOWN state.
- Return material authorize (RMA) any DSP that does not return to an UP state.
Troubleshoot Packet Voice Digital Signal (PVDM -2)
- On the router, copy the startup configuration onto a TFTP server with the command copy startup-config-tftp.
- Open the startup configuration file with a text editor.
- Add these debugs at the start of the file:
do debug dsp-resource-manager flex download
do debug dsp-resource-manager flex error
do debug dsp-resource-manager flex detail
do debug vpm dsp
- Save the startup configuration file.
- Copy the startup configuration file to the router's startup-config. For example, run copy tftp://<ip>/startup-config startup-confi in the route console. Debug is enabled in the startup configuration.
- Reload the router and collect the logs.
DSP Timeouts
Each time a query is sent, a timer is started and you must wait for its response. When a response is not received, the DSP_TIMEOUT message appears. There are several reasons why DSP timeouts occur:
- Messages are sent to DSP in incorrect mode
- Messages are sent in incorrect sequence
- Messages are sent with incorrect parameters
- Platform issues or internal DSP problems
There are several decoding events:
- A: PLAYOUT_DELAY : Playout Delay Stats
- B: PLAYOUT_ERROR : Playout Error Stats
- C: RX_STAT : Receive Packet Stats
- D: TX_STAT : Transmit Packet Stats
- E: LEVEL_STATS : At Call Cleanup
- F: ERROR_STATS : At Call Cleanup
- G: FAX_RELAY : In Fax Relay Mode
- H: MODEM_RELAY : In Modem Relay Mode
- I: CLOCK_COMP : In PassThru Mode
See this example:
%DSMP-3-DSP_TIMEOUT: DSP timeout on DSP 0/1:1: event 0xDC, DSMP timed out,
while waiting for statistics from the DSP. DSMP State = S_DSMP_COLLECTING_STATS
Timeouts that occur on a single DSP channel might indicate a hardware failure. Follow these steps:
- Enable Mini-Logger.
- Collect the output of show log as soon as the timeout event occurs.
DSP Alarms
See this example of a DSP alarm:
Mar 25 05:44:09.872: %DSPRM-2-DSPALARM: Received alarm indication from dsp (0/3).
Resetting the DSP.
Mar 25 05:44:09.872: %DSPRM-3-DSPALARMINFO: 001A 0000 0080 0000 0001 0050 6170 6963
6861 6E28 3132 3630 2900 0000 0000 0000 0000
Enter the command show voice dsp detailed and check if there was a crash in the restory (RST) number of resets per DSP and alarm indication (Al) fields. See this example:
*DSP VOICE CHANNELS*
CURR STATE : (busy)inuse (b-out)busy out (bpend)busyout pending
LEGEND : (bad)bad (shut)shutdown (dpend)download pending
DSP DSP DSPWARE CURR BOOT PAK TX/RX
TYPE NUM CH CODEC VERSION STATE STATE RST AI VOICEPORT TS ABRT PACK COUNT
====== === == ========= ========== ===== ======= === == ========= == ==== ============
C5510 001 01 g711ulaw 28.3.8 busy idle 0 0 0/3/3 12 0
130554418/67 79
C5510 001 02 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 03 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 04 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 05 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 06 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 07 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 08 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 09 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 10 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 11 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 12 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 13 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 14 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 15 None 28.3.8 idle idle 0 0 0 0/0
C5510 001 16 None 28.3.8 idle idle 0 0 0 0/0
C5510 002 01 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 02 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 03 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 04 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 05 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 06 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 07 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 08 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 09 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 10 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 11 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 12 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 13 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 14 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 15 None 28.3.8 idle idle 1 1 0 0/0
C5510 002 16 None 28.3.8 idle idle 1 1 0 0/0
Enter the command show voice dsp error and show specific errors. See this example:
DSP: 0/1:1
Timeout: 10317799.760 10316844.140 10316735.720 10316688.136
Number of errors overwritten: 213
Alarm: None recorded
Message drop: None recorded
DSP: 0/1:2
Timeout: 10316625.468 10316034.948 10315828.816 10315650.844
Number of errors overwritten: 63
Alarm: None recorded
Message drop: None recorded
DSP: 0/1:3
Timeout: 10316656.860 10301703.084 10301341.276 10300263.408Number of errors overwritten: 5
Configure the mini-logger and the crash-dump. See the Configure Crash Dump and Enable Mini-Logger section for instructions on how to configure the mini-logger and the crash-dump and collect the complete output of the show log command.
No More Ring Descriptors Error
This error is seen in two scenarios:
- DSP crashes
- The DSP buffer is full
See this example:
Jun 19 16:25:04 UTC: %C5510-4-NO_RING_DESCRIPTORS: No more ring descriptors
available on slot 0 dsp 2.
Jun 19 16:25:09 UTC: %C5510-4-NO_RING_DESCRIPTORS: No more ring descriptors
available on slot 0 dsp 2.
If the DSP crashes, use the approach in the DSP Alarms section.
If the DSP is full, enable the mini-logger and collect the output of the show log command as soon as the event occurs.
TDM Connect Issue
See this example for a TDM connect issue:
FLEXDSPRM-3-TDM_CONNECT failed to connect voice-port (0/0/0) to dsp_channel(0/0/0)
Cisco bug ID CSCuj64211 is a very common issue in this scenario. Reboot the router as a temporary solution to the problem until you can upgrade to one of the fixed releases of the product.
See these useful show commands:
show voice call status
show voice call summary
show call active voice brief
show voice dsp group all
show voice dsp
show tdm connection slot <>
Show tdm connection slot <>
Show tdm map slot <>
Show tdm connection slot <> wic <>
Show tdm map slot <> wic <>
See these debugs that you must collect for a test call:
debug voip ccapi inout
debug tdm
debug vpm all
debug voip vtsp all
debug voip dsm all
debug dsp-resource-manager flex all
For this issue, you can use Cisco IOS? Embedded Event Manager (EEM) script. See this example:
event manager applet voice_debug
event tag log1 syslog pattern "FLEXDSPRM-3-TDM_CONNECT failed to connect
voice-port (0/0/0) to dsp_channel(0/0/0)" maxrun 200
event tag log2 syslog pattern "hwic_t1e1_wic_bp_disconnect: disconnect failed"
maxrun 200
trigger
correlate event log1 and event log2
action 1 cli command "enable"
action 2 cli command "show tdm connection slot 0 | append disk0:debug.txt"
action 3 cli command "show call active voice brief | append disk0:debug.txt"
action 4 cli command "show call history voice brief | append disk0:debug.txt"
action 5 cli command "show voice call summ | append disk0:debug.txt"
action 6 cli command "show voice vtsp call fsm | append disk0:debug.txt"
action 7 cli command "show voice dsmp stream | append disk0:debug.txt"
action 8 wait 180
action 9 cli command "undebug all"
action 10 syslog msg "Stopping debug"
action 11 cli command "show logging | append disk0:debug.txt"
/debug.txt"
See this link for more details on the configuration process for an EEM script: EEM Configuration for Cisco Integrated Services Router Platforms.
Configure CrashDump and Mini-Logger
The DSP Mini-Logger and CrashDump utilities are useful tools that gather evidence that help troubleshoot DSP crashes. The Mini-Logger captures forensic evidence up to the point of the DSP crash, while CrashDump captures evidence during the crash and right after the crash occurs. Enter these commands in order to enable these features:
DSP CrashDump
voice dsp crash-dump file-limit
voice dsp crash-dump destination < destinationFilesystem:filename >
For example: voice dsp crash-dump destination flash:dspcrashdump.
DSP Mini-Logger
voice dsp <slot#> command history buffer control
voice dsp <slot#> command history buffer periodic
voice dsp <slot#> command history max-logger-print
voice dsp <slot#> command history enable
Best Practice Settings
Use these best practice settings:
- Set the CrashDump file limit no higher than a value of 5.
- Crash Dump destination should be a TFTP filesystem. Flash is also acceptable if TFTP is not available.
- MiniLogger buffers for control and intervals can be set up to a value of 10,000.
- MiniLogger max-logger-print can reach a value of up to 1,000.
- Disable console logging or you will cause the CPU to spike to dangerous values.
- Send everything, which includes the syslog, to an external server.
Feedback