Guest

Cisco PGW 2200 Softswitch

PGW 2200 Softswitch: Collect Troubleshooting Information for TAC Cases

Document ID: 27580

Updated: Jan 20, 2006

   Print

Introduction

When you open a case with the Cisco Technical Assistance Center (TAC), some preliminary information is required in order to better identify and qualify the issue. Some of this information is always required and other information depends upon the nature of the issue. If you are asked to collect this information by the engineer after you open your case, it results in resolution delay. Also, a good description of the problem is requested, which needs to include the call flow of the way that you ran into the problem at that time. The collection of Message Definition Language (MDL), snooper, sniffer and debug information can been attached to the case notes when you open the case.

The primary goal of this document is to identify the required preliminary information, based on the type of issue. This is so you can provide information to the engineer immediately. The second goal is to provide general guidelines to follow when you collect information for the TAC in order to avoid repetitive testing and recollection of identical data.

This document is intended for Cisco customers that support Data and Voice Signaling Solutions based on Cisco PGW 2200 Softswitch (formerly SC 2200 and VSC 3000, or Cisco Telephony Controller) Media Getaway Controller (MGC) software.

Prerequisites

Requirements

Support personnel need to be familiar with MGC-based solutions and their components. For further information, refer to the links supplied in the Related Information

Related Cisco Support Community Discussions section of this document.

Components Used

The information in this document is based on these software versions:

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Basic Information

Complete these steps:

  1. Before you collect any information, make sure you are logged on as a member of mgcgrp. In order to do this issue the id command.

    collect_troubleshoot_info1.gif

  2. Issue the uname command in order to find out you UNIX version.

    collect_troubleshoot_info2.gif

  3. Issue the prtconf command in order to find out the RAM size.

    collect_troubleshoot_info3.gif

    Another way to determine the RAM size is to issue the uname command.

    collect_troubleshoot_info4.gif

  4. Issue the df command in order to determine the amount of free disk space available.

    collect_troubleshoot_info5.gif

  5. Save the results of the commands described in the earlier steps and redirect the output to the file with either the > ./filename.txt command or the >> ./filename.txt command for multiple attachments.

    In order to collect information about Layer 2 and 3 of the NIC operation, issue the ifconfig and netstat commands. Some of this information is good for case notes updates.

    collect_troubleshoot_info6.gif

  6. The netinfo.txt file that is created needs to be sent to the TAC engineer. The patch level is checked with the use of the pkginfo command with the output redirected to a file.

    collect_troubleshoot_info7.gif

  7. The patches-installed.txt file that is created needs to be sent to the TAC engineer. Issue the mml command to see the PGW 2200 software version that currently runs.

    collect_troubleshoot_info8.gif

  8. Issue the rtrv-ne-health command in order to see the overall health of the network element. This applies to Release 9.

    collect_troubleshoot_info9.gif

Export the Cisco PGW 2200 Configuration

In most cases, the current Cisco PGW 2200 Softswitch configuration that the TAC requests is used to analyze it for configuration errors. This configuration is not the output of the prov-rtrv:all command, as it does not show the provisioning commands that are used. Instead, TAC needs either the configuration script for the problem recreation or the current *.dat files from the system.

  1. In order to export and save the current configuration, issue the prov-exp:all command from MML.

    The config-exported.tar file that is created needs to be sent to the TAC engineer.

    collect_troubleshoot_info10.gif

  2. Use the tar archiving utility in UNIX in order to save current *.dat files.

    The config.tar file that is created needs to be sent to the TAC engineer who works on the case.

    collect_troubleshoot_info11.gif

Further Information Requested by the TAC

If the problem is not solved at this stage, more information is required. Collect the information described in this section on log messages and alarms.

  1. Check for OS Solaris messages. File messages and messages.x needs to be sent to the TAC engineer.

    collect_troubleshoot_info12.gif

  2. Issue the tail command in order to check for Cisco PGW 2200 Softswitch application messages.

    All platform.log files related to the problem need to be sent to the TAC engineer.

    collect_troubleshoot_info13.gif

  3. Issue the rtrv-alms command in order to check for Cisco PGW 2200 Softswitch alarms.

    collect_troubleshoot_info14.gif

    If you want to collect alarms from log files from the CLI with readable dates and times, create a reprocessing script in Perl (cut and paste this script):

    alias rdalm 'perl -F, -anwe '\''print 
    unpack("x4 A15", localtime($F[1])),".$F[2]: @F[0,3..7]"'\'''

    CD into the Alarms directory:

    cd /opt/CiscoMGC/var/spool
    

    Execute this command:

    rdalm alm_yyymmdd* 

    where yyyymmdd is the relevant date.

  4. Issue the rtrv-dest:all command in order to check the basic SS7 network status.

    collect_troubleshoot_info15.gif

Coredumps

In case a coredump is found under the directory /opt/CiscoMGC/var, run the two UNIX commands pstack and pmap under Solaris 2.8 and attach the output to the case notes. Also upload the core file to the case notes.

Note: At Cisco the core file is analyzed via other tools:

Example : 
Under /opt/CiscoMGC/var 
<logging as root> 
#pstack <core_file> 
#pmap <core_file>

Call Traces

In order to avoid asking ''call traces'' which give a more granular tracing option on the PGW 2200 for the 'call engine process state' of that call, the MDL tracer gives a C++ object instantiated by the engine that contains a collection of TraceFile objects for that call. This MDL trace can give some details where the problem is related on the PGW 2200 and is welcome during the handling of the case. The best scenario is to upload as much information as possible during the time you encounter the problem. Based on the scenario or solution you run into, you can collect the information detailed in these sections:

Collect the PGW 2200 MDL Trace

Use this procedure in order to collect an MDL trace via the MML command STA-SC-TRC (Start Trace).

Based on which Cisco PGW 2200 Softswitch version you run, detailed information can be found for:

  1. Identify the Originating SS7 SigPath Number or the Originating TrunkGroup Number on which calls are placed.

  2. Rotate the log: run script under /opt/CiscoMGC/bin/log_rotate.sh.

  3. Start the MDL trace:

    mml>sta-sc-trc:<ss7sigPath name | orig trunkgroup number>:CONFIRM
    
  4. Perform a test (make a call).

  5. Stop the MDL trace:

     mml>stp-sc-trc:all
    
  6. Identify the Call Id (C:) of the bad call.

    If this test call is made in a test environment, only one CALL_ID displays.

    Note: These files can contain tracings from many calls that are all mixed up together if the capture is taken on a production PGW. Each tracing record in the file has a specific record type and records information of a type that relates to that record. Each record has a Call ID that relates it to a specific call.

  7. Convert the MDL trace into a readable format:

    1. Go to the /opt/CiscoMGC/var/trace directory.

    2. Run this command:

      get_trc.sh <trace file name>
      

      For example:

      /opt/CiscoMGC/var/trace
      mgcusr@mgc-bru-20%get_trc.sh _ss7path_20040116103221.btr
          get_trc.sh ca/sim/sp Trace File Utility Mistral Version 1.2
          The ANALYSIS mdo file is:  GENERIC_ANALYSIS.mdo
          Retrieving _ss7path_20040116103221.btr trace file Call ID's, please wait...
          Enter one of the following commands:
          S = Simprint in less
          F = Simprint with printing of sent and received Fields in less
          D = Display trc trace in less
          G = Display trc trace in less (Generated)
          C = Convert to trc trace file
          A = Display CA file in less
          N = Move to Next call ID
          P = Move to Previous call ID
          L = List call ID's in current file
          X = Set SP flags
          H = Print Help
          Q = Quit get_trc.sh
          Or just enter the ID of the call you want if you know it
          Use (N)ext and (P)revious to move between the call ID's
          _ss7path_20040116103221.btr contains 1 call(s)
         ==> Working on call 1 ID 23 H = Help [S/F/D/G/C/A/N/P/L/H/Q/id]? 
  8. Type Call Id at the prompt in order to jump to the MDL trace of the bad call.

  9. Choose option C in order to convert the trace file.

    Note: The .btr files are binary trace files that are produced by the PGW tracer function. The main part of the file name is given in the VSC MML command sta-sc-trc. The PGW always adds a .btr extension to these files. By using the C option, the file is converted into a text format and the extension has .trc files that are text trace files. They contain detailed line by line trace information from the MDO code that is run in the simulation replay that produces the file. Therefore, they contain MDL traces.

  10. The trace file is in /opt/CiscoMGC/var/trace.

  11. Collect the platform.log information under /opt/CiscoMGC/var/log. In some cases the TAC engineer can ask for other platform.log information related to the problem that is reported while the TAC case is handled.

Collect HSI MDL Traces

Refer to HSI Data Collection for Technical Support Service Requests for information and procedures on how to collect HSI MDL traces.

Collect Snooper/Sniffer Traces

Use this procedure to collect snooper/sniffer traces if you have installed Packet Telephony Center - Monitoring and Troubleshooting (PTCMT), or you run an old version of snooper (which is helpful in order to have a good understanding of the call flow).

  1. Run snoop on all Solaris platforms.

    1. In order to collect UNIX snoop information, log in as a superuser and run the command snoop -x42 -o snoop.log <IP address>.

    2. Press Ctrl +C to exit snoop.

    3. Upload the snoop.log file to the case-notes.

      Note: Explain in the case notes that this file has been captured via the UNIX snoop command.

  2. Run the Cisco snooper application.

    1. In order to collect Cisco snooper information, log in as a superuser and run the command ./snooper int INTERFACE PARMS LIST or you can run ./snooper which provides a full description.

    2. For the Nailed solution, run ./snooper int hme'x' isdn ss7 rlm > snooper_int1, where x is the interface number. You can also find this when you issue the ifconfig -a command. Also, upload the snooper_int1 file to the case notes.

    3. For the Switched solution, enter ./snooper int hme'x' mgcp ss7 eisup >snooper_int1, where x is the interface number. You can also find this when you issue the ifconfig -a command. Also, upload the snooper_int1 file to the case notes.

    4. For cases where output from two interfaces needs to be captured, use this approach:

      % ( snooper int hme0 rudp & ; sleep 1 ; snooper int hme1 rudp & ) >> test
      % ps -ef | grep snooper | grep -v grep
      root 10748 10737 1 20:52:54 pts/15 0:00 snooper int hme1 rudp
      root 10736 1 1 20:52:53 pts/15 0:00 snooper int hme0 rudp
      % tail -f test
  3. Run PTCMT. For more information, refer to Cisco Packet Telephony Center - Monitoring and Troubleshooting.

    In order to collect PTCMT information, log in as a superuser and run the command ./ptcmt int INTERFACE PARMS LIST or you can run ./snooper which provides a full description.

    • For the Nailed solution, enter ./ptcmt int hme'x' isdn ss7 rlm > snooper_int1, where x is the interface number. You can also use the ifconfig -a command. Also, upload the snooper_int1 file to the case notes.

    • For the Switched solution, enter ./ptcmt int hme'x' mgcp ss7 eisup >snooper_int1, where x is the interface number. You can also use the ifconfig -a command. Also, upload the snooper_int1 file to the case notes.

Collect debug Information on the Gateway

Based on if you use a Nailed solution [Ni2+] or a Switched solution [MGCP], debug information between the PGW 2200 and the gateway can provide detailed information of the reported problem.

  • For the Nailed solution, enter the debug isdn q931 command and upload the details to the TAC case.

  • For the Switched solution, enter the debug mgcp packet command and upload the details to the TAC case.

Note: Be aware that you do not run this debug command while the CPU load is above 60 percent. You can check this with the show proc cpu command. Also, based on the problem that is reported, other debug commands can be requested by the TAC engineer.

caution Caution: Collection of call traces can impact system performance and calls can be dropped. Call traces on a live system need to be done only at the request of the TAC engineer in case you are not sure about the collection of the log information.

Collect System Data

Cisco PGW 2200 Softswitch software includes a data collection script. When you run this script, a data snapshot of your system is saved to a log file. You should run this script shortly after you discover a problem and prior to taking any corrective action. Refer to Collecting System Data for Cisco TAC for more information.

Here is the collect data script:

mssol-pgw-6% collectdata
the location of the log file is /opt/CiscoMGC/var/log/200806111552.mssol-pgw-6.log
mssol-pgw-6% ls -al /opt/CiscoMGC/var/log/200806111552.mssol-pgw-6.log
-rw-rw-r--   1 mgcusr   mgcgrp    266375 Jun 11 15:52 /opt/CiscoMGC/var/log/200806111552.mssol-pgw-6.log
mssol-pgw-6% more /opt/CiscoMGC/var/log/200806111552.mssol-pgw-6.log

Mini Parse Analyzing Tool

The mini_parse.pl is a tracing tool that can provide a detailed analysis of call flows. This tool is located in the /opt/CiscoMGC/bin folder. Mini-parse provides a simple flow diagram of events that are contained in an MDL trace file (.trc). The usage is shown here:

mini_parse.pl [-d] [-b] [-i] [-m] [-s] <tracefile>

where

  • -d: adds additional message decode

  • b: adds additional B number analysis info

  • m: prints messages only (no internal signals)

  • i: adds additional IN info

  • s: adds state transitions

The output can be redirected to a file.

Per Call Trace

You can perform advanced call traces with Cisco PGW 2200 Softswitch Release 9.7(3) Patch 8 and later. The advanced call trace is based on the existing call trace function and adds the calling party number, the called party number, the machine congestion level (MCL) setting, the cause value, and the call duration as call trace criteria. This enhancement makes the call trace more accurate and reduces system performance impacts on the Cisco PGW 2200 Softswitch when the Cisco PGW 2200 Softswitch preforms call traces.

For information on how to start the call trace, refer to the Starting A Call Trace (on Release 9.7(3) Patch 8) section of the Cisco PGW 2200 Softswitch Release 9 Operations, Maintenance, and Troubleshooting Guide.

Related Information

Updated: Jan 20, 2006
Document ID: 27580