Cisco PGW 2200 Softswitch Release 9.8 Operations, Maintenance, and Troubleshooting Guide
Troubleshooting Cisco ITP-L Signaling
Downloads: This chapterpdf (PDF - 352.0KB) The complete bookPDF (PDF - 6.01MB) | Feedback

Troubleshooting Cisco ITP-L Signaling

Table Of Contents

Troubleshooting Cisco ITP-L Signaling

Cisco ITP-L Signaling Overview

IP Signaling Backhaul

Types of Encapsulation

PDU Verb Types

Backhaul Message IDs

Connection Management

Backhaul Statistics

Backhaul Congestion

Link Status

Troubleshooting SS7 Link Problems

Checking Link Configuration Files

Checking UDP Traffic Flows

Checking Connection between Cisco PGW 2200 Softswitch and Cisco ITP-L

Checking the T1/E1 Link State

Verifying the Link Alignment Status

Verifying Exchanged Point Codes

Cross-Checking Configuration Files

Troubleshooting Cisco ITP-L-to-STP Signaling Links

MTP1 Communication Problems

Identifying MTP1 Communication Problems

Resolving MTP1 Communication Problems

MTP2 Communication Problems

Identifying MTP2 Communication Problems

Resolving MTP2 Communication Problems

Troubleshooting Cisco ITP-L to Cisco PGW 2200 Softswitch Communications

Identifying MTP3 and Higher Layer Problems

Resolving MTP3 and Higher Layer SS7 Communication Problems

Identifying Ethernet Connectivity Problems

Identifying IP Communication Problems

Identifying RUDP Communications Problems

Cisco ITP-L Error Messages


Troubleshooting Cisco ITP-L Signaling


Revised: March 7, 2011, OL-0800-14

Cisco ITP-Ls function as signaling link interfaces to Signaling System 7 (SS7) Signal Transfer Point (STP) mated pairs on the SS7 network side. On the network side, Cisco ITP-Ls function as IP interfaces to the Cisco PGW 2200 Softswitches. Several different SS7 messages pass between the Cisco ITP-Ls and STPs. Messages also pass through Cisco switches between the Cisco ITP-Ls and Cisco PGW 2200 Softswitches.

Each STP in a mated pair is active continuously under normal operating conditions. SS7 message traffic normally flows between both STPs of the mated pair and the Cisco ITP-Ls, as shown in Figure B-1.

Figure B-1 Cisco ITP-L Hardware and I/O Signaling

This chapter includes the following sections:

Cisco ITP-L Signaling Overview

Troubleshooting SS7 Link Problems

Troubleshooting Cisco ITP-L-to-STP Signaling Links

Troubleshooting Cisco ITP-L to Cisco PGW 2200 Softswitch Communications

Cisco ITP-L Error Messages.

Cisco ITP-L Signaling Overview

This section contains the following subsections:

IP Signaling Backhaul

Connection Management

IP Signaling Backhaul

The Cisco PGW 2200 Softswitch and Cisco ITP-L communicate using the Reliable User Datagram Protocol (RUDP) that is proprietary to Cisco to perform IP signaling backhaul.

You can issue the following Cisco ITP-L command line interface (CLI) command to trace backhaul messages:

debug ss7 mtp2 backhaul channel

The following sections describe IP signaling backhaul:

Types of Encapsulation

PDU Verb Types

Backhaul Message IDs

Types of Encapsulation

There are two types of data encapsulation that are associated with IP signaling backhaul messages:

Non-PDU messages—Defined as session manager control messages. These messages serve to control active and standby sessions with the respective Cisco PGW 2200 Softswitches.

PDU messages—Messages the Session Manager delivers to the Message Transfer Part (MTP) Level 2 (MTP2). These messages serve to control MTP2 and to send and receive message signaling unit (MSU) messages.

PDU Verb Types

There are three PDU verb types that are associated with IP signaling backhaul commands and messages:

Requests—Messages that are sent only from the Cisco PGW 2200 Softswitch to the Cisco ITP-L. These messages request the Cisco ITP-L to perform some action, such as to connect the link (align link), disconnect the link, or to return its statistics.

Confirmations—Messages that are sent from the Cisco ITP-L to the Cisco PGW 2200 Softswitch to indicate that a requested action completed successfully or failed.

Indications—Asynchronous messages that are sent from the Cisco ITP-L to the Cisco PGW 2200 Softswitch to indicate that the Cisco ITP-L detected a state change, such as link alignment lost.

Backhaul Message IDs

There are five types of IP signaling backhaul message IDs:

Backhaul reset commands

Connection management commands

Backhaul statistics messages

Flow control

Link status

Backhaul Reset

There are two types of IP signaling backhaul reset commands:

SoftReset (Link reset)—Command is sent from the Cisco PGW 2200 Softswitch to the Cisco ITP-L to put the backhaul signaling link in the Out-of-Service (OOS) state. If the command succeeds, the Cisco ITP-L does not respond because the backhaul link is out of service. Sample message trace:

00:10:18: SoftResetRequest

HardReset (CPU reset)—Command is sent from the Cisco PGW 2200 Softswitch to the Cisco ITP-L to cause a CPU reset on the Cisco ITP-L. Sample message trace:

00:10:18: HardResetRequest

Connection Management

There are two IP signaling backhaul connection commands, a connection request and a disconnect request. Each command has a corresponding confirmation message.

Connection request—Command that is sent from the Cisco PGW 2200 Softswitch to the Cisco ITP-L to request link alignment. The request indicates if the alignment is in a normal or emergency state.

Connection confirmation—Reply sent by the Cisco ITP-L to the Cisco PGW 2200 Softswitch in response to a connection request. The reply indicates whether the connection request succeeded or failed. Sample message trace:

4d10h: MTP2: rcvd Conn Req - Emergency  ch=0 Statistics—

Disconnect request—Cisco PGW 2200 Softswitch sends this command to the Cisco ITP-L to change the status of an In-Service (IS) link to the status out of service (OOS). The request is always processed.

The Cisco ITP-L transmits a status indicator, an Out-of-Service (OOS) message on the SS7 link.

Disconnect confirmation—Reply sent by the Cisco ITP-L to the Cisco PGW 2200 Softswitch in response to a disconnect request. The reply indicates whether the disconnect request succeeded or failed.

Disconnect indication—Cisco ITP-L sends this asynchronous message to the Cisco PGW 2200 Softswitch. The message indicates that the Cisco ITP-L detected a state change that calls for a disconnect request command, such as link alignment lost. Sample message trace:

4d10h: MTP2: send Disc Ind  ch=0  reason=0x7-LSSU condition

The following sections describe connection management:

Backhaul Statistics

Backhaul Congestion

Link Status

Backhaul Statistics

There are two IP signaling backhaul statistics messages:

Stats request—Cisco PGW 2200 Softswitch sends this command that to the Cisco ITP-L to request the Cisco ITP-L to return its MTP Level 1 (MTP1) and MTP2 statistics. The request is always processed.

The command includes an action value to specify one of three options: (1) return the statistics and reset the statistics collection, (2) return the statistics, or (3) reset the statistics collection.

Stats confirmation—Reply sent by the Cisco ITP-L to the Cisco PGW 2200 Softswitch in response to the stats request command. Sample message trace:

4d10h: MTP2: rcvd Statistics Req-Send&Reset   ch=0 

Backhaul Congestion

The Cisco ITP-L sends an asynchronous message that includes a congestion indication to the Cisco PGW 2200 Softswitch. The message indicates that the backhaul signaling link is entering (Onset) or exiting (Abate) congestion.

Sample message trace:

Mar  1 005616.707 MTP2 send Flow Ind  ch=0  status=0x0 start congestion

The Cisco ITP-L has two possible types of congestion. Both are determined in the same manner, but control flow in different directions.

MTP2 signaling congestion—SS7 congestion that is detected on each SS7 link.

Backhaul congestion—Pertains to the active Session Manager session.

The percentage of free receive buffers determines the congestion onset and abatement.

Congestion onset—Indication that the signaling node is congested.

When the number of free receive buffers drops below 20 percent, the Cisco ITP-L sends a backhaul congestion onset message to the Cisco PGW 2200 Softswitch. At this point, the Cisco PGW 2200 Softswitch holds all backhaul traffic that is destined for the Cisco ITP-L.

Congestion abate—Indication that congestion has cleared.

When the number of free receive buffers rises above 40 percent, a backhaul congestion abate message is sent from the Cisco ITP-L to the Cisco PGW 2200 Softswitch. At this point, the Cisco PGW 2200 Softswitch resumes sending backhaul traffic that is destined for the Cisco ITP-L.

Link Status

Congestion status is maintained for backhaul.

Example of a congestion status message:

Nomad-C#sho ss7 mtp2 state 
SS7 MTP2 states for channel 0
Protocol version for channel 0 is Bellcore GR-246-Core Issue 2, Dec 1997 
MTP2LSC_INSERVICE       MTP2IAC_IDLE           
MTP2TXC_INSERVICE       MTP2RC_INSERVICE       
MTP2SUERM_MONITORING    MTP2AERM_IDLE          
MTP2CONGESTION_IDLE    
Congestion Backhaul  = Abate 
Remote Processor Outage  = FALSE 

Troubleshooting SS7 Link Problems

The following sections describe methods of troubleshooting SS7 link problems:

Checking Link Configuration Files

Checking UDP Traffic Flows

Checking Connection between Cisco PGW 2200 Softswitch and Cisco ITP-L

Checking the T1/E1 Link State

Verifying the Link Alignment Status

Verifying Exchanged Point Codes

Cross-Checking Configuration Files

Checking Link Configuration Files

Check the configuration files on the Cisco PGW 2200 Softswitch and the Cisco ITP-L. The IP addresses and UPD ports must match.

MTP2 Configuration:

Is the channel configured to the proper MTP2 variant?

Do the MTP2 variant protocol parameters match the remote configuration?

Session Manager Configuration:

Are the proper number of sessions defined, session-0 and session-1?

Do the session configurations match the Cisco PGW 2200 Softswitch session configurations?

Do the RUDP parameters match the Cisco PGW 2200 Softswitch RUDP configuration?

Checking UDP Traffic Flows

Check User Datagram Protocol (UDP) traffic flows between the Cisco PGW 2200 Softswitch and the Cisco ITP-L by entering the following commands:

log on 2600, enable 
debug ip udp

Depending on your configuration, the response should look like the following:

2600-1#debug ip udp
UDP packet debugging is on
2600-1#
15:06:53: UDP: rcvd src=10.15.13.6(7000), dst=10.15.13.2(7000), length=32
15:06:53: UDP: rcvd src=10.15.13.6(7000), dst=10.15.13.2(7000), length=32
15:06:53: UDP: sent src=10.15.13.2(7000), dst=10.15.13.6(7000), length=164
15:06:53: UDP: sent src=10.15.13.2(7000), dst=10.15.13.6(7000), length=164
15:06:53: UDP: rcvd src=10.15.13.6(7000), dst=10.15.13.2(7000), length=12
15:06:55: UDP: sent src=10.15.13.2(7000), dst=10.15.13.6(7000), length=32
15:06:55: UDP: rcvd src=10.15.13.6(7000), dst=10.15.13.2(7000), length=12

Check for traffic in both directions. If there is traffic, see the "Checking Connection between Cisco PGW 2200 Softswitch and Cisco ITP-L" section.

Otherwise, verify IP addresses, try to ping in both directions, reload the Cisco ITP-L software, check subnets, and check the VLANs on the LAN switches.

Checking Connection between Cisco PGW 2200 Softswitch and Cisco ITP-L

Check that the Cisco PGW 2200 Softswitch connects to the Cisco ITP-L:

debug ss7 mtp2 backhaul ip upd N

Where N identifies the specific MTP link. Valid values are from 0 through 3.

The Cisco ITP-L does not attempt to align the link until it receives an MTP3 Connect Indication from the Cisco PGW 2200 Softswitch. Issue the preceding debug command to reveal the MTP3 primitives between the Cisco ITP-L and the Cisco PGW 2200 Softswitch.

Checking the T1/E1 Link State

Check the T1 or E1 link state by observing the LEDs on the Cisco ITP-L. Ensure that the framing options match on both sides of the physical link.

Verifying the Link Alignment Status

Check alignment status of a link by entering the following debug command:

debug ss7 mtp2 iac N

Where N identifies the specific MTP link. Valid values are from 0 through 3.

Table B-1 describes various debug outputs from the previous command, the probable cause, and the recommended recovery. This traffic is exchanged only when the link is initially opened. If the link is already In-Service, nothing is displayed.

Table B-1 Debug Outputs, Probable Causes, and Recovery Actions

Debug Output
Probable Cause
Recovery Action

No output.

Link is already aligned.

MTP2 is not started.

1. Check that term monitor is on.

2. Reload Cisco ITP-L.

3. Cross-check configuration files.

15:12:33: itu2IAC_Start chnl=0 MTP2IAC_IDLE
15:12:34: itu2IAC_Stop chnl=0 MTP2IAC_NOT_ALIGNED
15:12:39: itu2IAC_Start chnl=0 MTP2IAC_IDLE
15:12:51: itu2IAC_T2_TMO chnl=0 MTP2IAC_NOT_ALIGNED
15:12:56: itu2IAC_Start chnl=0 MTP2IAC_IDLE
15:13:07: itu2IAC_T2_TMO chnl=0 MTP2IAC_NOT_ALIGNED
15:13:12: itu2IAC_Start chnl=0 MTP2IAC_IDLE

MTP2 does not flow across the link.

Check DS0 assignment (should use the same time slot on both sides of the physical link) and the DS0 speed (defaults to 56 kb/s on T1 and 64 kb/s on E1). The following items can change the DS0 speed:

conf t
contr E1 0/0
channel-group 0
timeslot 1 speed 56

15:14:32: itu2IAC_Start chnl=0 MTP2IAC_IDLE
15:14:33: itu2IAC_Rcvd_SIE chnl=0 MTP2IAC_NOT_ALIGNED
15:14:33: itu2IAC_Rcvd_SIE chnl=0 MTP2IAC_ALIGNED
15:14:37: itu2IAC_T4_TMO chnl=0 MTP2IAC_PROVING
15:14:38: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial0/0:0, changed state to up
15:14:45: itu2IAC_Rcvd_SIOS chnl=0 MTP2IAC_IDLE
15:14:46: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial0/0:0, changed state to down
15:14:50: itu2IAC_Start chnl=0 MTP2IAC_IDLE
15:14:50: itu2IAC_Rcvd_SIE chnl=0 MTP2IAC_NOT_ALIGNED
15:14:50: itu2IAC_Rcvd_SIE chnl=0 MTP2IAC_ALIGNED
15:14:54: itu2IAC_T4_TMO chnl=0 MTP2IAC_PROVING
15:14:55: %LINEPROTO-5-UPDOWN: Line protocol on
Interface Serial0/0:0, changed state to up

The link is able to align, but fails in the PROVING sequence.

It is generally a mismatch in point codes.

Check the provisioning settings for the SLC, OPC, and DPC in the Cisco PGW 2200 Softswitch, as described in the "Bouncing SS7 Links" section on page 6-101.

Use an SS7 sniffer tool to look at the exchanged point codes. The procedure in "Verifying Exchanged Point Codes" section describes how to use the Cisco ITP-L debug tools to see the point codes.


Verifying Exchanged Point Codes

Check exchanged point codes by entering the following command:

debug ss7 mtp2 pac N

Where N identifies the specific MTP link. Valid values are 0 through 3.

Table B-2 describes various debug outputs from this command, the probable cause, and the recommended recovery when the Cisco PGW 2200 Softswitch is using ANSI SS7. Table B-3 describes the debug outputs when the Cisco PGW 2200 Softswitch is using ITU SS7.

Table B-2 Debug Outputs, Probable Causes, and Recovery Actions (ANSI SS7)

Debug Output
Probable Cause
Recovery Action

No output

MTP2 is not started.

1. Check that term monitor is on.

2. Reload the Cisco ITP-L.

3. Cross check configuration files.

15:08:31: MTP2 incoming trace enabled on channel 0.
15:08:31: MTP2 outgoing trace enabled on channel 0.
15:08:34: ---- Outgoing Rudp msg (41 bytes) ----
SM_msg_type 0x00008000
protocol_type 0x0001
msg_ID 0x0000
msg_type 0x0011
channel_ID 0x0000
bearer_ID 0x0000
length 0x0019
data 0xB2236ED6 0x006FD600 0x11F01122 0x33445566
0x778899AA 0xBBCCDDEE

15:08:34: ---- Incoming Rudp msg (41 bytes) ----
SM_msg_type 0x00008000
protocol_type 0x0001
msg_ID 0x0000
msg_type 0x0010
channel_ID 0x0000
bearer_ID 0x0000
length 0x0019
data 0xB2006FD6 0x236ED600 0x21F01122 0x33445566
0x778899AA 0xBBCCDDEE

Cisco PGW 2200 Softswitch is exchanging messages with remote STP.

Exchanged point codes must match before communication can be established successfully.

Checkpoint codes: in data 0xB2236ED6 0x006FD6:

1.  236ED6 should be read
D6-6E-23 and converted in decimal: 214-110-035, which is the point code of the Cisco PGW 2200 Softswitch.

2.  006FD6 should be read D6-6F-00 and converted in decimal: 214-111-000, which is the point code of the STP in this example.

The values in the incoming and outgoing messages must match.


Table B-3 Debug Outputs, Probable Causes, and Recovery Actions (ITU SS7)

Debug Output
Probable Causes
Recovery Action

No output

MTP2 is not started.

1. Check that term monitor is on.

2. Reload the Cisco ITP-L.

3. Cross check configuration files.

Nov 14 15:15:13.276: ---- Outgoing Rudp 
msg (31 bytes) ----
SM_msg_type    0x00008000
protocol_type  0x0001
msg_ID         0x0000
msg_type       0x0011
channel_ID     0x0000
bearer_ID      0x0000
length         0x000F
data           0xC10E0519 0x111180A5 
0xA5A5A5A5 0xA5A5A51C 

Nov 14 15:15:13.288: ---- Incoming Rudp 
msg (31 bytes) ----
SM_msg_type    0x00008000
protocol_type  0x0001
msg_ID         0x0000
msg_type       0x0010
channel_ID     0x0000
bearer_ID      0x0000
length         0x000F
data           0xF1648443 0x112180A5 
0xA5A5A5A5 0xA5A5A520 

Cisco PGW 2200 Softswitch is exchanging messages with remote STP.

Exchanged point codes must match before communication can be established successfully.

Checkpoint codes in the outgoing RUDP message: in data 0xC10E0519 0x111180A5:

The part in bold-face type should be read as
11 19 05 0E C1 and converted in bit-swapped HEX: 0001000100011001000001010000111011010001.

1. The HEX part in bold-face type is the point code of the Cisco PGW 2200 Softswitch, 0.161.6.

2. The HEX part in italic type is the point code of the STP in this example, 0.140.4.

Checkpoint codes in the incoming RUDP message: in data 0xF1648443 0x112180A5:

The part in bold-face type should be read
11 43 84 64 F1 and converted in bit-swapped HEX: 0001000101000011100001000110010011110001.

1. The HEX part in bold-face type is the point code of the STP, 0.140.4.

2. The HEX part in italic type is the point code of the Cisco PGW 2200 Softswitch, 0.161.6.

The values in the incoming and outgoing messages must match.


Cross-Checking Configuration Files

Cross-check the configuration files by entering the following command:

debug ss7 mtp2 iac 0

You should see a response like the following Cisco PGW 2200 Softswitch sample configuration file:

File: XECfgParm.dat (extract)

*.ipAddrLocalA = 10.15.13.6 # Should be same as *.IP_Addr1
*.ipAddrLocalB = 10.15.13.22
*.ipAddrPeerA = 0.0.0.0 # Failover peer's address
*.ipAddrPeerB = 0.0.0.0

*.IP_Addr1 = 10.15.13.6 # Address of interface on motherboard
*.IP_Addr2 = 10.15.13.22
*.IP_Addr3 = 0.0.0.0
*.IP_Addr4 = 0.0.0.0
*.stPort = 0

This file defines the Cisco PGW 2200 Softswitch IP addresses that are used to communicate with the Cisco ITP-L.

To check if they match with the Solaris configuration, issue the Solaris ifconfig -a command.

File: sigChanDev.dat
001d0001 0 1 00080002 00030014 00060001 0
001d0002 0 1 00080001 00030014 00060001 1
001d0003 1 1 00080001 00030014 00060002 1
001d0004 1 1 00080002 00030014 00060002 0
001d0005 2 1 00080002 00030014 00060001 0
001d0006 2 1 00080001 00030014 00060001 1
001d0007 3 1 00080001 00030014 00060002 1
001d0008 3 1 00080002 00030014 00060002 0

Note The last digit in each line (0 or 1 in this example) identifies the link ID on the Cisco ITP-L. The last digit value can be 0, 1, 2, or 3 and is identified, misleadingly, as a time slot in Cisco PGW 2200 Softswitch provisioning. A Cisco ITP-L can only have two STP links.


File: sigChanDevIp.dat

001d0001 IP_Addr1 7000 10.15.13.2 7000
001d0002 IP_Addr1 7000 10.15.13.2 7000
001d0003 IP_Addr2 7000 10.15.13.19 7000
001d0004 IP_Addr2 7000 10.15.13.19 7000
001d0005 IP_Addr1 7000 10.15.13.4 7000
001d0006 IP_Addr1 7000 10.15.13.4 7000
001d0007 IP_Addr2 7000 10.15.13.21 7000
001d0008 IP_Addr2 7000 10.15.13.21 7000

This file associates the Cisco PGW 2200 Softswitch IP address and UDP port to the Cisco ITP-L IP address and UDP port. IP_Addr1 and IP_Addr2 are defined in XECfgParm.dat.

You should not edit these files using vi. If you use a provisioning tool, any change you make is lost. The only exception is XECfgParm.dat file (and changes can be lost anyway).

Cisco ITP-L sample configuration:

controller T1 0/0
framing esf
linecode b8zs
channel-group 0 timeslots 1
!
controller T1 0/1
framing esf
linecode b8zs
channel-group 0 timeslots 1
!
interface Ethernet0/0
ip address 10.15.13.2 255.255.255.240
no ip directed-broadcast
!
interface Serial0/0:0
no ip address
no ip directed-broadcast
!
interface Ethernet0/1
ip address 10.15.13.18 255.255.255.240
no ip directed-broadcast
!
interface Serial0/1:0
no ip address
no ip directed-broadcast
!


ip classless
ip route 172.18.0.0 255.255.0.0 10.15.13.1
no ip http server
!
ss7 set failover-timer 3
ss7 session-0 address 10.15.13.6 7000 10.15.13.2 7000
ss7 session-0 retrans_t 600
ss7 session-0 cumack_t 300
ss7 session-0 kp_t 2000
ss7 session-0 m_retrans 2
ss7 session-0 m_cumack 3
ss7 session-0 m_outseq 3
ss7 session-0 m_rcvnum 32
!
line con 0
transport input none
line aux 0
line vty 0 4
login
!
end

Troubleshooting Cisco ITP-L-to-STP Signaling Links

Cisco ITP-Ls interface with STPs through linksets. STP linksets can support a maximum of 16 individual SS7 signaling links. You can configure each Cisco ITP-L to interface with as many as two individual SS7 signaling links. Cisco ITP-Ls support SS7 Message Transfer Part Levels 1 and 2 (MTP1 and MTP2) in the Cisco ITP-L-to-STP signaling link interfaces.

When you replace a Cisco ITP-L with a new unit, complete the following steps to determine whether the original Cisco ITP-L was the cause of the SS7 communication problem:


Step 1 Connect an SS7 protocol analyzer to a patch panel monitor port to monitor the SS7 message traffic entering or leaving the Cisco ITP-L-to-STP link.

Step 2 Monitor the SS7 message traffic (if any) between the STP and the Cisco ITP-L.

Step 3 If the Cisco ITP-L is receiving SS7 traffic from the STP, go to Step 4.

If the Cisco ITP-L is not receiving SS7 message traffic, go to the "MTP1 Communication Problems" section.

Step 4 Ensure that the Cisco ITP-LSS7 is tranceiving Message Signaling Units (MSUs), Fill-in Signaling Units (FISUs), or Link Status Signaling Units (LSSUs).

Step 5 If the Cisco ITP-L is transceiving SS7 LSSU messages, go to the "MTP2 Communication Problems" section.

If the Cisco ITP-L is not transceiving SS7 LSSU messages, go to Step 6.

Step 6 If the Cisco ITP-L is transceiving SS7 MSU and FISU messages, go to the "Troubleshooting Cisco ITP-L to Cisco PGW 2200 Softswitch Communications" section.

Step 7 If the Cisco ITP-L is not transceiving SS7 MSU and FISU messages, replace the faulty Cisco ITP-L with a backup unit, if one is available.

If the problem is no longer present with the replacement unit, you can test the faulty unit offline to determine the cause of the problem.


MTP1 Communication Problems

The next two sections describe the procedures for identifying and solving MTP1 communication problems. The initial indication of signaling problems may change in T1 (or E1) status. Check for alarms on the T1 (or E1) interface before performing any of the following procedures.

Identifying MTP1 Communication Problems

MTP1 standardizes SS7 signaling link physical connectivity. When an MTP1 problem occurs, there is a physical connection break or a virtual break (something that causes the symptoms of a physical connection break, such as no power to a card slot) in the signaling link path. A break is identified when no Message Signaling Unit (MSU), Fill-In Signaling Unit (FISU), or Link Status Signaling Unit (LSSU) traffic can be sent or received over the SS7 link. MTP1 communication problems are normally the result of either a hardware failure, a cabling problem, or a physical interface problem.

Figure B-2 Physical Layer, MTP1 Communication Problems

Resolving MTP1 Communication Problems

If monitoring the SS7 link with a protocol analyzer reveals no MSU, FISU, or LSSU message traffic, complete the following steps:


Step 1 Ensure that power is on to the Cisco ITP-L.

Step 2 Check to ensure that the STP signal link cabling is connected correctly to the Cisco ITP-L.

Step 3 Disconnect the Cisco ITP-L from both the STP and the LAN switch for offline testing. Connect two recommended SS7 protocol analyzers, one to the STP interface the other to the IP interface.

One SS7 protocol analyzer must be equipped to send SS7 test messages to (or receive SS7 test messages from) the Cisco ITP-L over the V.35 interface. The other SS7 protocol analyzer sends SS7 test messages to (and receives SS7 test message from) the Cisco ITP-L over the IP interface.


Caution Do not leave the Cisco ITP-L connected to the LAN switch, and thus the Cisco PGW 2200 Softswitch, while injecting SS7 test messages into the Cisco ITP-L. The Cisco PGW 2200 Softswitch might not properly recognize the SS7 test messages that the protocol analyzer sends, which could cause error conditions between the Cisco PGW 2200 Softswitch and the Cisco Media Gateway.

Step 4 Test Cisco ITP-L ports and hardware by conducting a loop test of the signal link, excluding connectivity to the distant-end SS7 node and the LAN switch.

If the test does not discover an MTP1 problem, the MTP1 problem likely resides within the STP node or the connection to the STP node.

Step 5 If the problem is within the Cisco ITP-L, replace the unit.


MTP2 Communication Problems

The next two sections describe the procedures for identifying and solving MTP2 communication problems.

Identifying MTP2 Communication Problems

MTP2 standardizes SS7 signaling link alignment. MTP2 communication problems occur when Cisco ITP-Ls cannot establish data link alignment with STPs. When this misalignment occurs, the FISUs and MSUs cease to be transmitted.

Whenever an SS7 link has good physical connectivity (MTP1), but cannot align to send and receive either FISU or MSU traffic, LSSUs replace the FISUs and MSUs.

Figure B-3 Data Link Layer, MTP2 Communication Problem

Resolving MTP2 Communication Problems

If monitoring of the SS7 link with a protocol analyzer reveals no MSU or FISU message traffic (only LSSU traffic), complete the following steps:


Step 1 Check to ensure that the signal link cabling is correctly connected to the Cisco ITP-L.

Step 2 Disconnect the Cisco ITP-L from both the STP and the LAN switch for offline testing. Connect two recommended SS7 protocol analyzers.

One SS7 protocol analyzer must be equipped to send SS7 test messages to (and receive SS7 test messages from) the Cisco ITP-L over the V.35 interface. The other SS7 protocol analyzer sends SS7 test messages to (and receives SS7 messages from) the Cisco ITP-L over the IP interface.

Step 3 Test router ports and hardware by conducting a loop test of the signal link, excluding connectivity to the distant-end SS7 node.

Step 4 If the test does not discover an MTP2 (link alignment) problem, the problem likely resides within the distant end STP node.

If you discover a problem with the Cisco ITP-L, replace the unit.


Troubleshooting Cisco ITP-L to Cisco PGW 2200 Softswitch Communications

Cisco ITP-Ls communicate with a Cisco PGW 2200 Softswitch through a Cisco Catalyst MSR, which serves as a LAN switch. Under normal conditions, all Cisco ITP-Ls actively process SS7 message traffic from the STPs. However, only one of the two Cisco PGW 2200 Softswitches actively processes traffic at a time. One Cisco PGW 2200 Softswitch always stays in a hot-standby mode while the other actively processes message traffic.

Routing, call control, network management, and all other SS7 application data is framed within SS7 protocol layers MTP3 and higher. The Cisco ITP-Ls, which terminate the MTP1 and MTP2 layers, pass MTP3 and higher-layer SS7 protocol data between the Cisco PGW 2200 Softswitches and STP mated pairs.

Cisco ITP-L to Cisco PGW 2200 Softswitch communication comprises multiple Cisco ITP-Ls, which pass SS7 message traffic on to the Cisco PGW 2200 Softswitches through the LAN switches. Each STP linkset coming into a Cisco ITP-L normally has links that are connected to at least two Cisco ITP-Ls to ensure sustained network operation.

The Reliable User Datagram Protocol (RUDP), that is proprietary to Cisco, is used for Cisco ITP-L to Cisco PGW 2200 Softswitch communication. In a fault-tolerant configuration, for example, Ethernet 10BASE-T links connect each Cisco ITP-L to two LAN switches. Ethernet 100BASE-T links connect each LAN switch to both the active Cisco PGW 2200 Softswitch and the standby Cisco PGW 2200 Softswitch.

The following sections describe troubleshooting Cisco PGW 2200 Softswitch and Cisco ITP-L communications:

Identifying MTP3 and Higher Layer Problems

Identifying Ethernet Connectivity Problems

Identifying IP Communication Problems

Identifying RUDP Communications Problems

Identifying MTP3 and Higher Layer Problems

Although the Cisco ITP-Ls normally pass MTP3 and higher-layer data directly to the Cisco PGW 2200 Softswitches, Cisco ITP-L hardware can also be the cause of MTP3 and higher layer SS7 communication problems. Cisco ITP-L-originated MTP3 or higher layer SS7 problems can affect message traffic over a certain link, or just the links that transceive through a certain Cisco ITP-L.

Cisco ITP-Ls package received SS7 message data into RUDP datagrams that are transmitted through the LAN switches onto the Cisco PGW 2200 Softswitch. This process is reversed (Cisco ITP-L strips RUDP datagrams) as the Cisco ITP-Ls add standard SS7 message framing when the Cisco PGW 2200 Softswitches send SS7 messages to the Cisco ITP-Ls.

If a tested SS7 link has connectivity (MTP1) and alignment (MTP2), but network management tools report SS7 error messages, there is probably an MTP3 or higher layer SS7 communication problem. This problem requires testing to verify Cisco ITP-L operation.

Figure B-4 MTP3 and Higher-Layer SS7 Protocol Processing

Resolving MTP3 and Higher Layer SS7 Communication Problems

Coordinate a signaling link test of the SS7 transceive path within the system, excluding connectivity to distant end SS7 nodes. Use a recommended SS7 protocol analyzer to send SS7 messages to the suspected Cisco ITP-L while monitoring the output of the Cisco ITP-L with a recommended protocol analyzer. If the test does not discover an MTP1 (connectivity), MTP2 (link alignment), or MTP3 or higher layer problem, the problem probably is not the Cisco ITP-L.


Caution Do not leave the Cisco ITP-L connected to the LAN switch, and thus the Cisco PGW 2200 Softswitch, while injecting SS7 test messages into the Cisco ITP-L. The Cisco PGW 2200 Softswitch might not properly recognize the SS7 test messages that your protocol analyzer generates, which could cause error conditions between the Cisco PGW 2200 Softswitch and the media gateway.

Identifying Ethernet Connectivity Problems

SS7 message components are Ethernet framed and transceived through one of the two Cisco ITP-L Ethernet 10BASE-T ports. A physical break or a virtual break prevents a percentage of messages from traveling along the Cisco PGW 2200 Softswitch path (Cisco ITP-L-to-switch-to-Cisco PGW 2200 Softswitch). Use the ping utility to perform echo response tests. The ping utility should identify Ethernet connectivity problems within these components.

Identifying IP Communication Problems

ITP-L traffic is routed, rerouted, and (if necessary) retransmitted to the Cisco PGW 2200 Softswitches through the LAN switches. Monitor the following Internet Control Message Protocol (ICMP) error-reporting datagrams to help identify IP communication problems:

Destination Not Reachable/No Echo Reply

Source Quench/Receiving Buffer Congestion

Redirection Required

Time to Live Exceeded

Parameter Problems

Timestamp Request/Reply

Echo Request/Reply

If IP communication is good, the RUDP application layer software could be the cause of the problem. You should be able to use echo and timestamp request messages and monitoring response messages to identify RUDP, IP, or Ethernet communication problems within the system.

Identifying RUDP Communications Problems

You can use the following command to discover a reason for RUDP communication problems:

debug ss7 sm session ses_num


Where ses_num is the number of the affected session.

The system returns a response like the following:

SM: Failed to open session[0], return code = `x'

Where x is the RUDP return code. The valid value is an integer from 1 through 14. These return codes are defined as follows:

1—RUDP_OPTION_NOT_SUPPORTED

2—RUDP_NOT_READY

3—RUDP_CONNREC_RESOURCE_UNAV

4—RUDP_BUFFER_RESOURCE_UNAVAIL

5—RUDP_EVENT_RESOURCE_UNAVAIL

6—RUDP_EVENT_ENQUEUE_FAILED

7—RUDP_INVALID_CONN_HANDLE

8—RUDP_BUFFER_TOO_LARGE

9—RUDP_EMPTY_SEND_BUFFER

10—RUDP_CONNECTION_NOT_OPEN

11—RUDP_SEND_WINDOW_FULL

12—RUDP_REMOTE_PORT_REQUIRED

13—RUDP_REMOTE_ADDRESS_REQUIRED

14—RUDP_LOCAL_PORT_IN_USE

Cisco ITP-L Error Messages

Table B-4 lists the Cisco ITP-L error messages that the Cisco PGW 2200 Softswitch broadcasts.

Table B-4 Cisco ITP-L Error Messages 

Message Name
Definition
Recommended Action
OWNERR, PQUICC, LOG_ERR, 
MSG_TRACEBACK| MSG_PROCESS

An internal software error has occurred.

Call your technical support representative for a software upgrade for the Cisco ITP-L.

INITFAIL, PQUICC, LOG_ALERT, 
0, "PQUICC(%d%%d), 
SCC%d init failed"

The software failed to initialize and restart a 1T serial card.

Clear the serial interface. If the message recurs, call your technical support representative for assistance.

CTSLOST, PQUICC, LOG_ALERT, 
0, "PQUICC(%d%%d),  
Clear to Send Lost"

The CTS1 input signal on a DTE2 serial interface became inactive while transmitting a frame. This message results from a communication line failure or cable disconnection.

Check the serial interface cable and communication equipment, such as the CSU/DSU3 .

UNDERFLO, PQUICC, LOG_ALERT, 
0, "PQUICC(%d%%d), 
Transmit Underflow"

While transmitting a frame, the local buffer of the serial controller chip received insufficient data. Data could not be transferred to the chip fast enough to keep pace with its output rate. Normally, such a problem is temporary, depending on transient peak loads within the system.

The system should recover; no action is required.

LINEFLAP, PQUICC, LOG_ALERT, 
0, "PQUICC(%d%%d), Excessive 
modem control changes"

The system received too many modem control signal interrupts. Modem control signals are hardware handshake signals between data terminal equipment (DTE) and data communications equipment (DCE). The signals include either a data carrier detect (DCD) or a data set ready (DSR), or both.

Check the serial interface cable. The error can occur if the cable is disconnected or has become loose and is picking up noise. If the cable appears to be connected correctly, check the equipment that is connected to the cable.

BADHDXFSM, PQUICC, 
LOG_ALERT, 0, 
"PQUICC(%d%%d), Unexpected 
HDX state %d, event %d"

A bad event was detected in the state machine for half duplex transmission/reception.

Copy the error message exactly as it appears, and report it to your Cisco technical support representative.

TOOSMALL, PQUICC, LOG_ALERT, 
0, "PQUICC(%d/%d), packet 
was less than 2 bytes"

A small packet (less than 2 bytes) was queued up for transmission. The interface cannot process such small packets for transmission.

The system should recover. No action is required. If the message recurs, it might indicate a hardware error that is related to data traffic patterns. Copy the error message exactly as it appears, and report it to your Cisco technical support representative.

TOOBIG, PQUICC, LOG_ALERT, 
0, "PQUICC(%d/%d), packet 
too big";

A packet greater than the assigned MTU of this serial interface was queued for transmission.

The system should recover. No action is required. If the message recurs, it might indicate an error that is related to data traffic patterns. Copy the error message exactly as it appears, and report it to your Cisco technical support representative.

UNKNOWN_WIC, PQUICC, 
LOG_ALERT, 0,"PQUICC(%d), 
WIC card has an unknown ID 
of 0x%x"

The software does not recognize the WAN Interface Card (WIC) that is plugged into the port module.

Check part number on the WIC card to verify that it is supported in the Cisco IOS release operational on the router, or contact your Cisco technical support representative.

1 Clear to Send (CTS)

2 data terminal equipment (DTE)

3 channel service unit/data service unit (CSU/DSU)


If you need to contact your technical support representative for assistance, be prepared to provide the Cisco ITP-L and Cisco PGW 2200 Softswitch debug trace information. You should capture this information while the problem was occurring. In most cases, the backhaul trace and the LSC trace from the Cisco ITP-L are required. If the problem is associated with link alignment, you should also include the IAC trace output. Trace information can help the investigator identify the problem on the Cisco ITP-L or on the Cisco PGW 2200 Softswitch.

To enable MTP2 traces, enter the following commands:

debug ss7 mtp2 back channel 
debug ss7 mtp2 lsc channel 
debug ss7 mtp2 iac channel

To turn off debug trace, enter the un all command.

The output from a show version command provides explicit details about the image and branch information. This information specifically identifies which branch of code to investigate.

The output from a show run command indicates which Cisco ITP-L configuration is present. Knowing the configuration is important because improper configurations can cause many problems, such as timer durations.

Provide any information from show commands that you used to identify the problem; for example:

show ss7 sm stats

Include any other information that might be useful for understanding or reproducing the problem. This information helps your technical support representative verify a fix.