Deploy Diagnostic Signatures on ISR, ASR, and Catalyst Network Devices

Introduction

This document describes how to deploy a Diagnostic Signature (DS) in order to automatically collect the diagnostics data that is required to troubleshoot issues with the Cisco Integrated Services Routers (ISRs), Cisco Aggregation Services Routers (ASRs), and Cisco Catalyst 6500 Series switches and 7600 Series routers.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The show commands that are used throughout this document were captured from a Cisco ISR 3945 Series router that runs Cisco IOS^® Version 15.4(2)T3.

The DS support is available in the Cisco IOS versions that are listed in this table:

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

The DSs are XML files that contain information about problem trigger events and actions to be taken in order to remediate or troubleshoot. They are created by the Cisco Technical Assistance Center (TAC) and can be deployed via Smart Call Home, or you can manually load the file onto the device. You can use the Diagnostic Signature Lookup Tool (DSLT) in order to find the right DS for the given type of issue.

In this document, you will learn how to find and deploy a DS in order to troubleshoot PVDM-3 DSP crashes. This DS performs these actions:

1. Configures the router in order to generate Digital Signal Processor (DSP) crashdumps.
   
   
2. Detects DSP crashes via the writing out DSP dump to file <file name> syslog message.
   
   
3. Collects the relevant show commands.
   
   
4. Sends the collected data via email to attach@cisco.com.

At the time of installation, you are prompted to enter the maximum number of DSP crashdump files to be created, the case number to be used in the email subject, and the email address to be notified of the DSP crash occurrence.

Deploy

This section describes how to deploy the DS.

Configure

Note: The DSs are a part of the call-home feature in the Cisco IOS.

The first step that you must complete in order to deploy the DS is to configure the call-home feature. Once the call-home feature is enabled, the DS support is automatically enabled and configured for the use of the CiscoTAC-1 profile by default.

Here is an example configuration for Smart Call Home in the Cisco IOS:

service call-home
call-home
 contact-email-addr router@cisco.com
 mail-server 192.0.2.33 priority 1
 http-proxy "192.0.2.60" port 3128
 no syslog-throttling
 profile "CiscoTAC-1"
  active
  destination transport-method http
  no destination transport-method email

Router#show call-home diagnostic-signature 
Current diagnostic-signature settings:
 Diagnostic-signature: enabled
 Profile: CiscoTAC-1 (status: ACTIVE)
 Environment variable:
           Not yet set up

Downloaded DSes:
                                       Last Update
DS ID    DS Name   Revision Status     (GMT+00:00)
-------- --------- -------- ---------- ------------

Router#

Download

The second step is to find the signature ID or download the DS XML file. In order to do this, provide the platform, product, problem scope, problem type, and software version in Diagnostic Signature Lookup Tool as shown here:





Use this information in order to download the DS via Smart Call Home with the DS ID:

call-home diagnostic-signature download 10492

Use this information in order to manually load the DS file onto the device:

Router#copy ftp://192.0.2.10/DS_10492.xml flash:
Destination filename [DS_10492.xml]? 
Accessing ftp://192.0.2.10/DS_10492.xml...!
[OK - 3804/4096 bytes]

3804 bytes copied in 0.476 secs (7992 bytes/sec)

Router#
Router#call-home diagnostic-signature load flash:DS_10492.xml
Load file flash:DS_10492.xml success
Router#

Here is a high-level view of individual blocks within the DS file:

Router#show call-home diagnostic-signature 10492
ID              : 10492
Name            : DS_PVDM3_DSP_Crash_Event_1

Functionality:
This DS configures DSP crash dump generation, identifies crash event and
collects show commands required to troubleshoot and identify root cause.

This DS will have no impact on the performance of the router.

Prompts:
    Variable: ds_number_of_files               Prompt: Number of crashdump files
     to be stored in the flash    (1-5)
        Type: integer    Range: 1..5                                            
    Variable: ds_case_number                   Prompt: Enter TAC Case Number
     (Case number to which diagnostics data need to be uploaded)
        Type: regexp    Pattern: 6[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]       
    Variable: ds_user_email                    Prompt: Enter Notification Email-Address
     (Email address to which problem occurrence needs to be notified)
        Type: regexp    Pattern: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]+     

Prerequisite:
    Type            : CMD
    Element List    : 
        CMD : config t
        CMD : voice dsp crash-dump destination flash:dsp_crashdump
        CMD : voice dsp crash-dump file-limit $ds_number_of_files

Event:
    Action Tag      : a1 
    Event Tag       : e1
    Type            : syslog
    Syslog Pattern  : .*writing out DSP dump to file ([^[:space:]]+).*

Includes action steps that may impact device state: No
Action:
    Action Tag      : a1
    Type            : EMAILTO
    Email To        : attach@cisco.com,$ds_user_email
    Subject         : DSP Crashdump
    Attach SR       : $ds_case_number
    Element List    : 
        DATA: show version
        DATA: show voice dsp group all
        DATA: show call active voice brief
        DATA: show call active video brief
        DATA: show call active fax brief
        DATA: show sccp connection
        DATA: show dspfarm all
        DATA: dir $ds_dsp_crashdump_file
        DATA: show run

Postrequisite:
    Type            : CMD
    Element List    : 
        CMD : config t
        CMD : no voice dsp crash-dump destination flash:dsp_crashdump
        CMD : no voice dsp crash-dump file-limit $ds_number_of_files
        CMD : end

Router#

Install

Note: This step is required only for signatures that are in the pending status after download.

After you have configured and downloaded the DS, you must then install it:

Router#show call-home diagnostic-signature 
Current diagnostic-signature settings:
 Diagnostic-signature: enabled
 Profile: CiscoTAC-1 (status: ACTIVE)
 Environment variable:
           Not yet set up

Downloaded DSes:
                                                             Last Update
DS ID    DS Name                         Revision Status     (GMT-04:00)
-------- ------------------------------- -------- ---------- -------------------
10492    DS_PVDM3_DSP_Crash_Event_1      1.0      pending    2015-06-04 20:01:24

Router#

During the installation process, you are prompted with questions that are defined in the Prompts section:

Router#call-home diagnostic-signature install 10492
Number of crashdump files to be stored in the flash (1-5) 5
Enter TAC Case Number (Case number to which diagnostics data need to be uploaded)
 600000001
Enter Notification Email-Address (Email address to which problem occurrence needs
 to be notified) attach@cisco.com
All prompt variables are configured successfully.

Router#

Router#show call-home diagnostic-signature 
Current diagnostic-signature settings:
 Diagnostic-signature: enabled
 Profile: CiscoTAC-1 (status: ACTIVE)
 Environment variable:
           Not yet set up

Downloaded DSes:
                                                             Last Update
DS ID    DS Name                         Revision Status     (GMT-04:00)
-------- ------------------------------- -------- ---------- -------------------
10492    DS_PVDM3_DSP_Crash_Event_1      1.0      registered 2015-06-04 20:01:24

Router#

Once the DS is registered, the actions that are specified in the Prerequisite section are executed. In this example, the commands that are related to DSP crashdump generation are configured:

Router#show run | section voice dsp
voice dsp crash-dump file-limit 3
voice dsp crash-dump destination flash:dsp_crashdump
Router#

Verify

This section describes how to verify that the DS is installed and operates correctly.

Trigger Event

Cisco recommends that you simulate the problem trigger in order to ensure that the DS works correctly. For example, you can simulate a DSP crash via the test voice driver command, as shown here:

Router#test voice driver
Enter Voice NM  slot number : 0

C29xx/C39xx Debugging Section;

1 - FPGA Registers Read/Write
2 - 5510 DSP test
3 - DSPRM test
5 - IOCTRL TDM Registers Read/Write
6 - IOCTRL HDLC Registers Read/Write
7 - IOCTRL TDM Memory Read/Write
8 - get conn store address
9 - TDM PLL Read/Wrire
10 - SP2600 DSP test
11 - Quit

Select option : 10

SP2600 DSP Testing Section:

1 - Display Device Information
2 -  Reset 1 DSP
3 -  Reset All DSPs
4 -  Download DSP Firmware
5 -  JTAG Read DSP Memory
6 -  JTAG Write DSP Memory
7 -  Keepalive Enable/Disable
8 -  Display DSP Keepalive Status
9 -  Simulate DSP Crash
10 - ACK Testing
11 - Set Mbrd_dsp_debug Value
12 - PLD watch dog timers Enable/Disable
13 - Send Status_Request DSP Message
14 - Display Host and DSP MAC Address
15 - Display PLD and BOOTLOADER Version
16 - GigE enable/disable port
17 - Reset TDM port
18 - Show ports receiving oversubscription tone
19 - Display firmware build string
20 - Simulate All ARM Crash
21 - Simulate All ARM Crash after All DSS Crash
22 - Read PVDM PLD register
23 - Write PVDM PLD register
24 - Import DSP command file
25 - Switch DSP application between HR image and Streamware
26 - Show video capabilities of a DSP
27 - QUIT

Select option : 9
(1=DSP, 2=ARM) :1
Enter DSP id : 1
Enter Mode: 
Mode 1: Simulates Assert Condition
Mode 2: Simulates Endless loop
Mode 3: Stop High Level Responses to Commands

Enter Mode: 1

SP2600 DSP Testing Section:

1 - Display Device Information
2 -  Reset 1 DSP
3 -  Reset All DSPs
4 -  Download DSP Firmware
5 -  JTAG Read DSP Memory
6 -  JTAG Write DSP Memory
7 -  Keepalive Enable/Disable
8 -  Display DSP Keepalive Status
9 -  Simulate DSP Crash
10 - ACK Testing
11 - Set Mbrd_dsp_debug Value
12 - PLD watch dog timers Enable/Disable
13 - Send Status_Request DSP Message
14 - Display Host and DSP MAC Address
15 - Display PLD and BOOTLOADER Version
16 - GigE enable/disable port
17 - Reset TDM port
18 - Show ports receiving oversubscription tone
19 - Display firmware build string
20 - Simulate All ARM Crash
21 - Simulate All ARM Crash after All DSS Crash
22 - Read PVDM PLD register
23 - Write PVDM PLD register
24 - Import DSP command file
25 - Switch DSP application between HR image and Streamware
26 - Show video capabilities of a DSP
27 - QUIT

Select option : 27

C29xx/C39xx Debugging Section;

1 - FPGA Registers Read/Write
2 - 5510 DSP test
3 - DSPRM test
5 - IOCTRL TDM Registers Read/Write
6 - IOCTRL HDLC Registers Read/Write
7 - IOCTRL TDM Memory Read/Write
8 - get conn store address
9 - TDM PLL Read/Wrire
10 - SP2600 DSP test
11 - Quit

Select option : 11
Router#

Here is the output from the show log command:

032517: Jun  5 00:02:46.300: writing out DSP dump to file
 flash:dsp_crashdump-1433462566-1
032517: Jun  5 00:02:46.300: writing out DSP dump to file
 flash:dsp_crashdump-1433462566-1, sequence 
032517: Jun  5 00:02:46.300: writing out DSP dump to file
 flash:dsp_crashdump-1433462566-1, timestamp 
032532: Jun  5 00:02:46.344: DS-ACT-TRACE: call_home_ds_regexp_paren_str_get[2571],
 run regular expression once with pattern .*writing out DSP dump to file
 ([^[:space:]]+).*
032534: Jun  5 00:02:46.344: DS-ACT-TRACE: : writing out DSP dump to file flash:
dsp_crashdump-1433462566-1
032551: Jun  5 00:02:46.348: CALL-HOME-TRACE: Event 41 description <032517:
 Jun 5 00:02:46.300: writing out DSP dump to file flash:dsp_crashdump-1433462566-1>

Enter the show call-home diagnostic-signature statistics command in order to verify whether the problem event trigger has been detected by the DS:

Router#show call-home diagnostic-signature statistics 
                                           Triggered/     Average Run  Max Run
DS ID     DS Name                          Max/Deinstall  Time(sec)    Time(sec)
--------  -------------------------------  -------------  -----------  ---------
10492     DS_PVDM3_DSP_Crash_Event_1       0/0/N          0.000        0.000    

3900-12#
3900-12#
3900-12#
3900-12#show call-home diagnostic-signature statistics 
                                           Triggered/     Average Run  Max Run
DS ID     DS Name                          Max/Deinstall  Time(sec)    Time(sec)
--------  -------------------------------  -------------  -----------  ---------
10492     DS_PVDM3_DSP_Crash_Event_1       1/0/N          15.152       15.152   

Router#

Validate Action

The final step that you must complete in order to verify the DS deployment is to validate whether actions such as these are performed correctly:

• Command execution
  
  
• Script execution
  
  
• Collected data transmission via Email or Smart Call Home with collected data

In this example, an email is sent to attach@cisco.com with the output of the DSP-related show commands.