Cisco ICS 7750 Troubleshooting Guide, 2.5.0
Solving Serial Connection Problems
Downloads: This chapterpdf (PDF - 417.0KB) The complete bookPDF (PDF - 3.24MB) | Feedback

Solving Serial Connection Problems

Table Of Contents

Solving Serial Connection Problems

Using the show interface serial Command

Syntax

Understanding Command Output

Interface and Line Protocol Status

Output Drops

Input Drops

Input Errors

Interface Resets

Carrier Transitions

Using Extended ping Tests

Using debug Commands

Enabling debug Commands

Debugging Serial and WAN Problems

Troubleshooting Clocking Problems

Overview of Clocking

Causes of Clocking Problems

Detecting Clocking Problems

Isolating Clocking Problems

Solutions to Clocking Problems

Inverting the Transmit Clock

Adjusting Buffers

Tuning System Buffers

Implementing Hold Queue Limits

Using Priority Queuing to Reduce Bottlenecks

Using Loopback Tests

Local Loopback Tests for HDLC or PPP Links

Remote Loopback Tests for HDLC or PPP Links


Solving Serial Connection Problems


This chapter provides guidance on resolving problems associated with serial link connections. The chapter is organized as follows:

Using the show interface serial Command

Using Extended ping Tests

Using debug Commands

Troubleshooting Clocking Problems

Adjusting Buffers

Using Loopback Tests

Using the show interface serial Command

The output of the show interface serial EXEC command displays statistics related to the status of serial interfaces. You can use the show interface serial command to diagnose serial line connectivity problems in a WAN environment.

Syntax

The following is the syntax of the show interface serial command:

show interface serial [slot | interface] [accounting]

where:

slot (optional) identifies the slot of a particular card in the Cisco ICS 7750 (legal slot values are 1 through 8).

interface (optional) identifies a particular interface, such as serial 0.

accounting (optional) displays the number of packets of each protocol type that has been sent through the interface.

Understanding Command Output

The following sections describe the most important fields of the command output for troubleshooting serial connections.

Interface and Line Protocol Status

Output Drops

Input Drops

Input Errors

Interface Resets

Carrier Transitions

Example 6-1 shows an example of output from the show interface serial command for a synchronous serial interface. Table 6-1 describes the most significant fields, as indicated by the callouts (the numbers in square brackets).

Example 6-1 Output of show interface serial Command

Cisco ICS 7750#show interface serial
[1] Serial 0 is up, line protocol is up
   Hardware is MCI Serial
   Internet address is 150.136.190.203, subnet mask is 255.255.255.0
   MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,rely 255/255, load 1/255
   Encapsulation HDLC, loopback not set, keepalive set (10 sec)
   Last input 0:00:07, output 0:00:00, output hang never
   [2] Output queue 0/40, 0 drops; [3] input queue 0/75, 0 drops
   Five minute input rate 0 bits/sec, 0 packets/sec
   Five minute output rate 0 bits/sec, 0 packets/sec
       16263 packets input, 1347238 bytes, 0 no buffer
       Received 13983 broadcasts, 0 runts, 0 giants 
       [4] 2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 2 abort
[5] 1 carrier transitions 
     22146 packets output, 2383680 bytes, 0 underruns
     0 output errors, 0 collisions, [6] 2 interface resets, 0 restarts

Table 6-1 Descriptions for the show interface serial Field 

Callout
Field
Description

1

Interface and line protocol status

Indicates whether the interface hardware is currently active (whether CD1 is present) or that it has been disabled by an administrator.

2

Output drops

Number of packets in the output queue. Each number is followed by a slash, the maximum size of the queue, and the number of packets dropped because of a full queue.

3

Input drops

Number of packets in the input queue. Each number is followed by a slash, the maximum size of the queue, and the number of packets dropped because of a full queue.

4

Input errors, including CRC errors and framing errors

Total number of errors related to no buffer2 , runt3 , giant4 , CRC5 , frame6 , overrun7 , ignored8 , and abort. Other input-related errors can also increment the count, so this sum might not balance with the other counts.

5

Carrier transitions

Number of times the CD signal of a serial interface has changed state. For example, if DCD9 goes down and comes up, the carrier transition counter increments two times. This field also indicates modem or line problems if the CD line is changing state often.

6

Interface resets

Number of times an interface has been completely reset, which can happen if packets queued for transmission are not sent within several seconds. On a serial line, a reset can be caused by a malfunctioning modem that is not supplying the transmit clock signal or by a cable problem. If the system notices that the CD line of a serial interface is up, but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.

1 CD = carrier detect.

2 No buffer errors occur when a packet is discarded because there was no buffer space. Bursts of noise on serial lines are often responsible for no buffer events. Compare with ignored packets.

3 Runts are packets that are discarded because they are smaller than the medium's minimum packet size.

4 Giants are packets that are discarded because they are larger than the medium's maximum packet size.

5 CRC = cyclic redundancy check. A CRC error occurs when the CRC generated by the originating station or far-end device does not match the checksum calculated from the data received. On a serial link, CRCs usually indicate noise, gain hits, or other transmission problems on the data link.

6 Framing errors occur when packets are received with a CRC error and a noninteger number of octets. On a serial line, this is usually the result of noise or other transmission problems.

7 Overruns represent the number of times that the receiver hardware is unable to hand received data to a hardware buffer because the input rate exceeds the receiver's ability to handle the data.

8 Ignored packets are those that are discarded because the interface hardware does not have enough internal buffers. Compare with no buffer.

9 DCD = data carrier detect.


Interface and Line Protocol Status

Table 6-2 identifies six possible problem conditions in the interface status line of the show interface serial command display (see callout 1 in Example 6-1):

Serial x is down, line protocol is down (DTE mode)

Serial x is up, line protocol is down (DTE mode)

Serial x is up, line protocol is down (DCE mode)

Serial x is up, line protocol is up (looped)

Serial x is up, line protocol is down (disabled)

Serial x is administratively down, line protocol is down

Table 6-2 show interface serial Status Line Conditions 

Status Line Condition
Possible Problem
Solution
 

Serial x is down, line protocol is down (DTE1 mode)

Typically indicates that the router is not sensing a CD signal.

Telephone company problem—Line is down or not connected to CSU/DSU.

Faulty or incorrect cabling.

Hardware failure (CSU/DSU).

Step 1 

Check the LEDs on the CSU/DSU2 to see whether the CD is active, or insert a breakout box (see "System Troubleshooting Guidelines") on the line to check for CD signal.

 

Step 2 

Verify that you are using the proper cable and interface (refer to the Cisco ICS 7750 Installation and Configuration Guide).

 

Step 3 

Insert a breakout box, and check all control leads.

 

Step 4 

Contact your leased-line or other carrier service to see whether there is a problem.

   

Step 5 

Swap faulty parts.

   

Step 6 

If you suspect that an ASI3 or MRP4 card is faulty, change the serial line to another interface. If the connection comes up, there is a problem with the previously connected interface.

Serial x is up, line protocol is down (DTE mode)

Local or remote router is misconfigured.

Keepalives are not being sent by remote router.

Leased-line or other carrier service problem—noisy line or misconfigured or failed switch.

Step 1 

Put the modem, CSU, or DSU in local loopback mode, and use the show interface serial command to determine whether the line protocol comes up.

If the line protocol comes up, a telephone company problem or a failed remote router is probably the cause.

 

Timing problem on cable—SCTE5 not set on CSU/DSU.

Step 2 

If the problem appears to be on the remote end, repeat Step 1 on the remote modem, CSU, or DSU.

 

Failed local or remote CSU/DSU.

Failed local or remote hardware.

Step 3 

Verify all cabling. Make certain that the cable is attached to the correct interface, the correct CSU/DSU, and the correct telephone company network termination point.

   

Step 4 

Enable the debug serial interface EXEC command.

   

Step 5 

If the line protocol does not come up in local loopback mode and if the output of the debug serial interface command shows that the keepalive counter is not incrementing, an MRP hardware problem is likely. Swap the MRP.

   

Step 6 

If the line protocol comes up and the keepalive counter increments, the problem is not in the local ASI or MRP. Troubleshoot the serial line as described in the "Troubleshooting Clocking Problems" section and the "Using Loopback Tests" section.

   

Step 7 

If you suspect that the ASI or MRP hardware is faulty, change the serial line to an unused interface. If the connection comes up, there is a problem with the previously connected interface.

Serial x is up, line protocol is down (DCE6 mode)

Missing clockrate interface configuration command.

Step 1 

Add the clockrate interface configuration command on the serial interface.

 

DTE device does not support or is not set up for SCTE mode (terminal timing).

Failed remote CSU or DSU.

Step 2 

Set the DTE device to SCTE mode if possible. If your CSU/DSU does not support SCTE, you might have to disable SCTE on the Cisco router interface. (See the "Inverting the Transmit Clock" section.)

 

Failed or incorrect cable.

Step 3 

Verify that the correct cable is being used.

 

Router hardware failure.

Step 4 

If the line protocol is still down, there is a possible hardware failure or cabling problem. Insert a breakout box, and observe the leads. (See "System Troubleshooting Guidelines.")

   

Step 5 

Replace faulty parts as necessary.

Serial x is up, line protocol is up (looped)

Loop exists in circuit. The sequence number in the keepalive packet changes to a random number when a loop is detected initially.

Step 1 

Use the show running-config privileged EXEC command. This will enable you to look for any loopback interface configuration command entries.

 

If the same random number is returned over the link, a loop exists.

Step 2 

If you find a loopback interface configuration command entry, use the no loopback interface configuration command to remove the loop.

   

Step 3 

If you do not find the loopback interface configuration command, examine the CSU/DSU to determine whether it is configured in manual loopback mode. If it is, disable manual loopback.

   

Step 4 

Reset the CSU/DSU and inspect the line status. If the line protocol comes up, no other action is needed.

   

Step 5 

If the CSU/DSU is not configured in manual loopback mode, contact the leased-line or other carrier service for line troubleshooting assistance.

Serial x is up, line protocol is down (disabled)

High error rate because of telephone company service problem.

CSU/DSU hardware problem.

Step 1 

Troubleshoot the line with a serial analyzer and breakout box. (See "System Troubleshooting Guidelines.") Look for toggling CTS 37 and DSR 48 signals.

 

Bad router hardware.

Step 2 

Loop CSU/DSU (DTE loop). If the problem continues, there is likely a hardware problem. If the problem does not continue, the problem is likely with the telephone company.

   

Step 3 

Swap out bad hardware as required (CSU/DSU, switch, local ASI or MRP, or remote router).

Serial x is administratively down, line protocol is down

Router configuration includes the shutdown interface configuration command.

Step 1 

Check the ASI or MRP configuration for the shutdown command.

 

Duplicate IP address.

Step 2 

Use the no shutdown interface configuration command to remove the shutdown command.

   

Step 3 

Verify that there are no identical IP addresses using the show running-config privileged EXEC command or the show interface EXEC command.

   

Step 4 

If there are duplicate addresses, resolve the conflict by changing one of the IP addresses.

1 DTE = data terminal equipment. Computers and terminals are the most common DTE types.

2 CSU/DSU = channel service unit/data service unit.

3 ASI = analog station interface.

4 MRP = multiservice route processor.

5 SCTE = serial clock transmit external.

6 DCE = data communications equipment. Modems (analog) and CSU/DSUs (digital) are the most common DCE types.

7 CTS = Clear To Send.

8 DSR = Data Set Ready.


Output Drops

Output drops appear in the output of the show interface serial command when the system is attempting to hand off a packet to a transmit buffer but no buffers are available (see callout 2 in Example 6-1).

Symptom    Increasing number of output drops on serial link

Possible Cause    Input rate to serial interface exceeds bandwidth available on serial link

Recommended Action    The following steps are suggested when you encounter this symptom:


Step 1 Minimize periodic broadcast traffic such as routing and Service Advertising Protocol (SAP) updates by using access lists or other means. For example, to increase the delay between SAP updates, use the ipx sap-interval interface configuration command.

Step 2 Increase the output hold queue size in small increments by using the hold-queue out interface configuration command.

Step 3 On affected interfaces, turn off fast switching for heavily used protocols. For example, to turn off IP fast switching, enter the no ip route-cache interface configuration command. For the command syntax for other protocols, refer to the Cisco IOS IP Configuration Guide, Release 12.2, and the Cisco IOS IP Command Reference, Volume 2 of 3: Routing Protocols, Release 12.2.

Step 4 Implement priority queuing on slower serial links by configuring priority lists. For information on configuring priority lists, refer to the "Congestion Management" section in the Cisco IOS Quality of Service Solutions Configuration Guide, Release 12.2, and the Cisco IOS Quality of Service Solutions Command Reference, Release 12.2.


Input Drops

Input drops appear in the output of the show interface serial command when too many packets from that interface are still being processed in the system (see callout 3 in Example 6-1).

Symptom    Increasing number of input drops on serial link

Possible Cause    Input rate exceeds the capacity of the MRP or input queues exceed the size of output queues

Recommended Action    The following steps are suggested when you encounter this symptom:


Step 1 Increase the output queue size on common destination interfaces for the interface that is dropping packets. Use the hold-queue out interface configuration command.

Step 2 Reduce the input queue size by using the hold-queue in interface configuration command to force input drops to become output drops. Output drops have less impact on the performance of the router than input drops have.



Note Input drop problems typically occur when heavy traffic is being routed between Ethernet and serial interfaces. ASIs, MRPs, and routers may drop packets during these congested periods.


Input Errors

Possible sources for input errors that appear in the show interface serial command output (see callout 4 in Example 6-1) are as follows:

CRC errors

Framing errors

Aborted transmission


Note Any input error value for CRC errors, framing errors, or aborts above 1 percent of the total interface traffic suggests a link problem that you should isolate and repair.


The most likely sources are summarized in the list of possible problems that follows and in Table 6-3.

Symptom    Increasing number of input errors in excess of 1 percent of total interface traffic

Possible Cause    The following problems can result in this symptom:

Faulty telephone company equipment

Noisy serial line

Incorrect clocking configuration (SCTE not set)

Incorrect cable or cable too long

Bad cable or connection

Bad CSU or DSU

Bad ASI or MRP hardware

Data converter or other device being used between MRP and DSU


Note Cisco strongly discourages the use of data converters when you are connecting an ASI or MRP to a WAN or serial network.


Recommended Action    The following steps are suggested when you encounter this symptom:


Step 1 Use a serial analyzer to isolate the source of the input errors. If you detect errors, it is likely that there is a hardware problem or a clock mismatch in a device outside the ASI or MRP.

Step 2 Use the loopback and ping tests to isolate the specific problem source. For more information, see the "Using Extended ping Tests" section and the "Using Loopback Tests" section.

Step 3 Look for patterns. For example, if errors occur at a consistent interval, they could be related to a periodic function such as the sending of routing updates.


Table 6-3 Troubleshooting Serial Line Input Errors 

Input Error Type (Field Name)
Possible Problem
Solution
 

CRC errors

CRC errors occur when the CRC calculation does not pass, indicating that data is corrupted.

Step 1 

Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary.

 

Noisy serial line.

Serial cable too long.

Step 2 

Ensure that the cable is within the recommended length—no more than 50 ft (15.24 m), or 25 ft (7.62 m) for a T1 link.

 

SCTE mode is not enabled on DSU.

CSU line clock is incorrectly configured.

Step 3 

Ensure that all devices are properly configured for a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the "Inverting the Transmit Clock" section.

 

Ones density problem on T1 link (incorrect framing or coding specification).

Step 4 

Ensure that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service—for example, ESF1 /B8ZS2 .

Contact your leased-line or other carrier service, and have the service perform integrity tests on the line.

Framing errors

A framing error occurs when a packet does not end on an 8-bit byte boundary. Possible causes include:

Step 1 

Make sure that the physical cable meets the specified transmission requirements. Shield the cable if necessary. Ensure that you are using the correct cable and that it is firmly seated in its connectors.

 

The serial line has signal noise on it or other interference.

Step 2 

Ensure that the cable is within the recommended length—no more than 50 ft (15.24 m), or 25 ft (7.62 m) for a T1 link.

 

Improperly designed cable; serial cable is too long; the cable from the CSU or DSU to the router is not shielded.

Step 3 

Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the "Inverting the Transmit Clock" section.

 

SCTE mode is not enabled on the DSU; the CSU line clock is configured incorrectly; one of the clocks is configured for local clocking.

Step 4 

Ensure that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service—for example, ESF/B8ZS.

 

Ones density problem on T1 link (incorrect framing or coding specification).

Step 5 

Contact your leased-line or other carrier service, and have the service perform integrity tests on the line.

Aborted transmission

Aborts indicate an illegal sequence of one bits (more than 7 in a row).

SCTE mode is not enabled on DSU.

Step 1 

Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the "Inverting the Transmit Clock" section.

 

CSU line clock is incorrectly configured.

Serial cable is too long or cable from the CSU or DSU to the router is not shielded.

Step 2 

Shield the cable if necessary. Make sure the cable is within the recommended length—no more than 50 ft (15.24 m), or 25 ft (7.62 m) for a T1 link. (Refer to the
Cisco ICS 7750 Installation and Configuration Guide.) Ensure that all connections are good.

 

Ones density problem on T1 link (incorrect framing or coding specification).

Step 3 

Check the hardware at both ends of the link. Swap faulty equipment as necessary.

 

Packet terminated in the middle of the transmission (typical cause is an interface reset or a framing error).

Step 4 

Lower the data rates, and observe whether aborts decrease.

 

Hardware problem—bad circuit, bad CSU/DSU, or bad sending interface on remote router.

Step 5 

Use local and remote loopback tests to determine where aborts are occurring. (See the "Using Loopback Tests" section.)

   

Step 6 

Contact your leased-line or other carrier service, and have the service perform integrity tests on the line.

1 ESF = Extended Superframe

2 B8ZS = binary eight-zero substitution


Interface Resets

Interface resets that appear in the output of the show interface serial command are the result of missed keepalive packets (see callout 5 in Example 6-1).

Symptom    Increasing number of interface resets on serial link

Possible Cause    The following problems can result in these symptoms:

Congestion on link (typically associated with output drops)

Bad line causing CD transitions

Possible hardware problem at the CSU, DSU, or switch

Recommended Action    When interface resets are occurring, examine other fields of the show interface serial command output to determine the source of the problem. Assuming that an increase in interface resets is being recorded, examine the following fields:


Step 1 If there are a high number of output drops in the show interface serial output, see the "Output Drops" section.

Step 2 Check the carrier transitions field in the show interface serial display. If carrier transitions are numerous while interface resets are being registered, the problem is probably a bad link or CSU/DSU. Contact your leased-line or carrier service, and swap faulty equipment as necessary.

Step 3 Examine the input errors field in the show interface serial display. If input errors are numerous while interface resets are increasing, the problem is probably a bad link or CSU/DSU. Contact your leased-line or other carrier service, and swap faulty equipment as necessary.


Carrier Transitions

Carrier transitions appear in the output of the show interface serial command whenever there is an interruption in the carrier signal, such as an interface reset at the remote end of a link (see callout 6 in Example 6-1).

Symptom    Increasing carrier transitions count on serial link

Possible Cause    The following problems can result in this symptom:

Line interruptions by an external source, such as physical separation of cabling, T1 alarms, or lightning striking somewhere along the network

Faulty switch, DSU, ASI, or MRP hardware

Recommended Action    The following steps are suggested when you encounter this symptom:


Step 1 Check the hardware at both ends of the link by attaching a breakout box or a serial analyzer. Test to determine the source of the problems. (See "System Troubleshooting Guidelines.")

Step 2 If an analyzer or breakout box does not identify any external problems, check the ASI or MRP hardware.

Step 3 Swap faulty equipment as necessary.


Using Extended ping Tests

The ping command is particularly useful when numerous input errors are being registered in the output from the show interface serial command. (See the "Input Errors" section.)

Cisco internetworking devices provide a mechanism to automate the sending of many ping packets in sequence. Example 6-2 illustrates the menu you can use to specify extended ping options. This example specifies 20 successive pings (in the Repeat count field). However, when testing the components on your serial line, you should specify a much larger number, such as 1000 pings.

Example 6-2 Extended ping Specification Menu

Cisco ICS 7750#ping
Protocol [ip]:
Target IP address:129.44.12.7
Repeat count [5]:20
Datagram size [100]:64
Timeout in seconds [2]:
Extended commands [n]:yes
Source address:
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:0xffff
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 20, 64-byte ICMP Echos to 129.44.12.7, timeout is 2 seconds:
Packet has data pattern 0xFFFF
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent, round-trip min/avg/max = 1/3/4 ms

In general, perform serial line ping tests as follows:


Step 1 Put the CSU/DSU into local loopback mode.

Step 2 Configure the extended ping command to send different packet sizes and data patterns.

Example 6-3 and Example 6-4 illustrate two useful ping tests, an all-zeros 1500-byte ping and an all-ones 1500-byte ping, respectively.


Note The packet size in both examples (1500 bytes) is specified by the Datagram size field. The data pattern (all zeroes in Example 6-3, all ones in Example 6-4) is specified by a hexadecimal value in the Data pattern field.


Example 6-3 All-Zeros 1500-Byte ping Test

Cisco ICS 7750#ping
Protocol [ip]:
Target IP address: 192.169.51.22
Repeat count [5]: 100
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address: 192.169.51.14
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]: 0x0000
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 192.169.51.22, timeout is 2 
seconds:
Packet has data pattern 0x0000
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), 
round-trip min/avg/max = 4/6/8 ms

Example 6-4 All-Ones 1500-Byte ping Test

Cisco ICS 7750#ping
Protocol [ip]:
Target IP address: 192.169.51.22
Repeat count [5]: 100
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address: 192.169.51.14
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]: 0xffff
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 192.169.51.22, timeout is 2 
seconds:
Packet has data pattern 0xFFFF
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), 
round-trip min/avg/max = 4/6/8 ms

Step 3 Examine the show interface serial command output, and determine whether input errors have increased. (See the "Input Errors" section.) If input errors have not increased, the local hardware (DSU, cable, ASI, and MRP card) is probably in good condition. If this test sequence was prompted by the appearance of a large number of CRC and framing errors, then a clocking problem is likely. Check the CSU/DSU for a timing problem. (See the "Troubleshooting Clocking Problems" section.)

Step 4 If you determine that the clocking configuration is correct and is operating properly, put the CSU/DSU into remote loopback mode.

Step 5 Repeat the ping test, and look for changes in the input error statistics.

Step 6 If input errors increase, there is a problem either in the serial line or on the CSU/DSU. Contact the WAN service provider, and swap the CSU/DSU. If problems persist, contact your technical support representative.


Using debug Commands

This section includes information on using debug commands on the ICS 7750, and includes the following sections:

Enabling debug Commands

Debugging Serial and WAN Problems


Note For information on debugging VoIP problems, see the "Debugging VoIP Problems" section. For information on debugging H.323 signaling, see the "Debugging H.323 Signaling" section.


Enabling debug Commands

All debug commands are entered in privileged EXEC mode. The output of debug privileged EXEC commands provides useful troubleshooting information relating to protocol status and network activity for many internetworking events.

Most debug commands take no arguments; for example to turn on vtsp debugging, enter the command debug vtsp all. To turn off a debug command, you can enter the "no" form of the command. For example, to turn off the command debug vtsp all, enter the "no" form of the command at the command prompt: no debug vtsp all. Alternatively, you can enter the undebug form of the command: undebug vtsp all.

To enable all system diagnostics, enter debug all. The no debug all (or undebug all) command turns off all diagnostic output.


Caution Use debug commands with care. Debugging output is assigned high priority in the CPU process; therefore, enabling debugging can significantly disrupt the operation of your system. For this reason, use debug commands only for troubleshooting specific problems or during troubleshooting sessions. When you finish using a debug command, remember to disable it with its specific no debug or undebug command, or with the no debug all command.

To use the debug commands, you must first enable logging on the monitor.

To enable logging, access the MRP using Telnet or a HyperTerminal connection through the SAP. At the command prompt, enter the following commands in privileged EXEC mode (this example uses the debug vtsp all command):

Cisco ICS 7750#conf terminal
Cisco ICS 7750(config)#logging console debug
Cisco ICS 7750(config)#logging monitor
Cisco ICS 7750(config)#exit
Cisco ICS 7750#term monitor
Cisco ICS 7750#debug vtsp all

Debugging Serial and WAN Problems

The following are some debug commands that are useful when troubleshooting serial and WAN problems. For more information about the function and output of each command, refer to the Cisco IOS Debug Command Reference.

debug serial interface verifies whether HDLC keepalive packets are incrementing. If they are not, an ASI card, an MRP card, or the network might have a timing problem.

debug arp indicates whether an ASI or MRP is sending information about, or learning about, other cards or routers (with Address Resolution Protocol [ARP] packets) on the other side of the WAN. Use this command when some nodes on a TCP/IP network are responding but others are not.

debug ppp negotiation shows Point-to-Point Protocol (PPP) packets transmitted during PPP startup, when PPP options are negotiated.

debug ppp packet shows PPP packets being sent and received. This command displays low-level packet dumps.

debug ppp errors shows PPP errors (such as illegal or malformed frames) associated with PPP connection negotiation and operation.

debug ppp chap shows PPP Challenge Handshake Authentication Protocol (CHAP) and Password Authentication Protocol (PAP) packet exchanges.

debug serial packet shows Switched Multimegabit Data Service (SMDS) packets being sent and received. This display also prints out error messages that indicate why a packet was not sent or was received erroneously. For SMDS, the command dumps the entire SMDS header and some payload data when an SMDS packet is transmitted or received.

Troubleshooting Clocking Problems

Clocking conflicts in serial connections can cause chronic loss of connection service or degraded performance. The following sections discuss issues related to clocking problems:

Overview of Clocking

Causes of Clocking Problems

Detecting Clocking Problems

Isolating Clocking Problems

Solutions to Clocking Problems

Inverting the Transmit Clock

Overview of Clocking

The CSU/DSU derives the data clock from the data that passes through it. To recover the clock, the CSU/DSU hardware must receive at least one 1-bit value for every 8 bits of data that pass through it (this is known as ones density.) Maintaining ones density allows the hardware to recover the data clock reliably.

Newer T1 implementations commonly use Extended Superframe (ESF) format framing with binary 8-zero substitution (B8ZS) encoding. B8ZS provides a scheme by which a special code is substituted whenever eight consecutive zeros are sent through the serial link. This code is then interpreted at the remote end of the connection. This technique guarantees ones density independent of the data stream.

Older T1 implementations use D4 (also known as Super Frame [SF] format) framing and alternate mark inversion (AMI) encoding. AMI does not utilize an encoding scheme like B8ZS. This restricts the type of data that can be transmitted because ones density is not maintained independent of the data stream.

Another important element in serial communications is serial clock transmit external (SCTE) terminal timing. SCTE is the clock echoed back from the data terminal equipment (DTE) device (such as an ASI or MRP) to the data communications equipment (DCE) device (such as a modem or CSU/DSU).

When the DCE device uses SCTE instead of its internal clock to sample data from the DTE, it is better able to sample the data without error even if there is a phase shift in the cable between the CSU/DSU and the router. Using SCTE is highly recommended for serial transmissions faster than 64 kbps. If your CSU/DSU does not support SCTE, see the "Inverting the Transmit Clock" section.

Causes of Clocking Problems

In general, clocking problems in serial WAN interconnections can be attributed to one of the following causes:

Incorrect DSU configuration

Incorrect CSU configuration

Cables out of specification (longer than 50 ft [15.24 m] or unshielded)

Noisy or poor patch panel connections

Several cables connected together

Detecting Clocking Problems

To detect clocking conflicts on a serial interface, look for input errors as follows:


Step 1 Use the show interface serial command on the ASI, MRPs, or routers at both ends of the link.

Step 2 Examine the command output for CRC, framing errors, and aborts (see the "Input Errors" section).

Step 3 If either of the preceding steps indicates errors exceeding an approximate range of 0.5 to 2.0 percent of traffic on the interface, clocking problems are likely to exist somewhere in the WAN.

Step 4 Isolate the source of the clocking conflicts as outlined in the "Isolating Clocking Problems" section.

Step 5 Bypass or repair any faulty patch panels.


Isolating Clocking Problems

After you determine that clocking conflicts are the most likely cause of input errors, follow these steps to isolate the source of those errors:


Step 1 Perform a series of ping tests and loopback tests (both local and remote), as described in the "Using Extended ping Tests" section and "Using Loopback Tests" section.

Step 2 Determine which end of the connection is the source of the problem or whether the problem is in the line. In local loopback mode, use different patterns and sizes in the ping tests (for example, use 1500-byte datagrams). Using a single pattern and packet size may not force errors to materialize, particularly when a serial cable to the ASI, MRP, or CSU/DSU is the problem.

Step 3 Use the show interface serial EXEC command, and determine whether input error counts are increasing and where they are accumulating.


If input errors are accumulating on both ends of the connection, clocking of the CSU is the most likely problem.

If only one end is experiencing input errors, there is probably a DSU clocking or cabling problem.

Aborts on one end suggest that the other end is sending bad information or that there is a line problem.


Note Refer to the show interface serial command output. Note any changes in error counts.



Note Refer to Troubleshooting Serial Lines for additional information on troubleshooting serial problems and clocking problems, including information about using show commands and debug commands, special line tests, and tuning mechanisms.


Solutions to Clocking Problems

Possible causes of clocking problems are as follows:

Incorrect CSU configuration

Incorrect DSU configuration

Out-of-specification cable to ASI or MRP

Table 6-4 describes possible solutions for these clocking problems.

Table 6-4 Clocking Problems and Solutions 

Possible Problem
Solution
 

Incorrect CSU configuration

Step 1 

Determine whether the CSUs at both ends agree on the clock source (local or line).

 

Step 2 

If the CSUs do not agree, configure them so that they do. The line is usually the source.

 

Step 3 

Check the LBO1 setting on the CSU to ensure that the impedance matches that of the physical line. For information on configuring the CSU, consult your CSU hardware documentation.

Incorrect DSU configuration

Step 1 

Determine whether the CSUs at both ends have SCTE mode enabled.

 

Step 2 

If SCTE is not enabled on both ends of the connection, enable it.

For any interface that is connected to a line of 128 kbps or faster, SCTE must be enabled. If your DSU does not support SCTE, see the "Inverting the Transmit Clock" section.

 

Step 3 

Make sure that ones density is maintained—the DSU must use the same framing and coding schemes (such as ESF and B8ZS) that the leased-line or other carrier service uses.

Check with your leased-line provider for information on the provider's framing and coding schemes.

If your carrier service uses AMI coding, either invert the transmit clock on both sides of the link or run the DSU in bit-stuff mode. For information on configuring your DSU, consult the DSU hardware documentation that came with your equipment.

Out-of-specification cable to ASI or MRP

Step 1 

If the cable is longer than 50 ft (15.24 m), use a shorter cable.

 

Step 2 

If the cable is unshielded, replace it with shielded cable.

1 LBO = Line Build Out


Inverting the Transmit Clock

If you are attempting serial connections at speeds greater than 64 kbps with a CSU/DSU that does not support SCTE, you might have to invert the transmit clock on the router. Inverting the transmit clock compensates for phase shifts between the data and clock signals.

The specific command used to invert the transmit clock varies depending on the platform. To ensure that you are using the correct command syntax for an ASI, MRP, or router, refer to the "Interface Commands (interface fastethernet - loopback line)" chapter in the Cisco IOS Interface Command Reference,
Release 12.2
.

Adjusting Buffers

Excessively high bandwidth utilization reduces overall performance and can cause intermittent failures. However, increasing the bandwidth might not be necessary or immediately practical. One way to resolve marginal serial-line overutilization problems is to control how MRPs use data buffers.


Caution In general, do not adjust system buffers unless you are working closely with a Cisco technical support representative. You can severely affect the performance of your hardware and your network if you incorrectly adjust the system buffers on MRPs or routers.

Use one of the following three options to control how buffers are used:

Adjust parameters associated with system buffers.

Specify the number of packets held in input or output queues (hold queues).

Prioritize how traffic is queued for transmission (priority output queuing).

The configuration commands associated with these options are described in
Cisco IOS configuration guide and command reference publications.

The following section focuses on identifying situations in which you are likely to use these options to help resolve connectivity and performance problems in serial and WAN interconnections.

Tuning System Buffers

There are two general types of buffers on Cisco ASIs, MRPs, and routers: hardware buffers and system buffers. Only the system buffers can be directly configured by system administrators.

The hardware buffers are used as the receive and transmit buffers associated with each interface and, in the absence of any special configuration, are dynamically managed by the system software.

The system buffers are associated with the main system memory and are allocated to different sized memory blocks. A useful command for determining the status of the system buffers is the show buffers EXEC command. Example 6-5 shows output from the show buffers command.

Example 6-5 show buffers Command Output

Cisco ICS 7750>show buffers
Buffer elements:
401 in free list (500 max allowed)
87777499 hits, 0 misses, 0 created
Small buffers, 104 bytes (total 120, permanent 120):
114 in free list (20 min, 250 max allowed)
70005538 hits, 6 misses, 2 trims, 2 created
Middle buffers, 600 bytes (total 90, permanent 90):
88 in free list (10 min, 200 max allowed)
25696696 hits, 27 misses, 27 trims, 27 created
Big buffers, 1524 bytes (total 90, permanent 90):
90 in free list (5 min, 300 max allowed)
8214530 hits, 15 misses, 366 trims, 366 created
Large buffers, 5024 bytes [2](total 5, permanent 5):
5 in free list (0 min, 30 max allowed)
15017 hits, 12 misses, [1] 16354 trims, 16354 created
Huge buffers, 18024 bytes (total 3, permanent 0):
2 in free list (0 min, 4 max allowed)
297582 hits, 17 misses, 30 trims, 33 created
0 failures [3](0 no memory)

The show buffers command output in Example 6-5 indicates large numbers in the trims and created fields for large buffers. (See callout 1.) In cases such as this, you can increase the serial link performance by increasing the max-free value configured for the system buffers.

Use the buffers max-free number global configuration command to increase the number of free system buffers. The value you configure should be approximately 150 percent of the figure indicated in the total field of the show buffers command output. (See callout 2.) Repeat this process until the show buffers output no longer indicates trims and created buffers.

If the show buffers command output shows a large number of failures in the no memory field (callout 3), you must either reduce the usage of the system buffers or increase the amount of shared or main memory (physical RAM) on the ASI, MRP, or router (refer to Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750).

Implementing Hold Queue Limits

Hold queues are buffers used by each ASI, MRP, or router interface to store outgoing or incoming packets. Use the hold-queue interface configuration command to increase the number of data packets queued before the routing device begins to drop packets.

Use the hold-queue command to prevent packets from being dropped and to improve serial link performance in the following situations:

You have an application that cannot tolerate drops, and the protocol is able to tolerate longer delays.

The interface is very slow (bandwidth is low, or anticipated utilization is likely to sporadically exceed available bandwidth).


Note When you increase the number specified for an output hold queue, you might need to increase the number of system buffers. The value you use depends on the size of the packets associated with the traffic anticipated for the network.


Using Priority Queuing to Reduce Bottlenecks

Priority queuing is a list-based control mechanism that allows traffic to be prioritized for each interface. Priority queuing involves two steps:


Step 1 Create a priority list by protocol type and level of priority.

Step 2 Assign the priority list to a specific interface.

Both of these steps use versions of the priority-list global configuration command. In addition, you can apply further traffic control by referencing access-list global configuration commands from priority-list specifications. For examples of defining priority lists and for details about command syntax associated with priority queuing, refer to the "Congestion Management" section in Cisco IOS Quality of Service Solutions Configuration Guide, Release 12.2, and Cisco IOS Quality of Service Solutions Command Reference, Release 12.2.



Note Priority queuing automatically creates four hold queues of varying sizes, which override any hold queue specification included in your configuration.


Use priority queuing to prevent packets from being dropped and to improve serial link performance in the following situations:

When the interface is slow, many types of traffic are being transmitted, and you want to improve terminal traffic performance.

If you have a serial link that is intermittently experiencing very heavy loads (such as file transfers occurring at specific times), you can use priority lists to select types of traffic to discard during heavy traffic periods.

In general, start with the default number of queues when you implement priority queuing. After enabling priority queuing, monitor output drops with the show interface serial EXEC command. If you notice that output drops are occurring in the traffic queue that you specified to be high priority, increase the number of packets that can be queued (using the queue-limit keyword option of the priority-list global configuration command).

Using Loopback Tests

If the output of the show interface serial EXEC command indicates that the serial line is up but the line protocol is down (see the "Interface and Line Protocol Status" section), use CSU/DSU loopback tests to determine the source of the problem. Perform the local loop test first, and then perform the remote test.


Note To use these tests, the internetworking system must be attached to a CSU or DSU, or to a multiplexer with built-in CSU/DSU functionality. Because there is no concept of a loopback in X.25 or Frame Relay packet-switched network (PSN) environments, loopback tests do not apply to X.25 and Frame Relay networks.


Local Loopback Tests for HDLC or PPP Links

The following is a general procedure for performing loopback tests in conjunction with built-in system diagnostic capabilities.


Step 1 Place the CSU/DSU in local loop mode (refer to your vendor documentation). In local loop mode, the use of the line clock (from the T1 service) is terminated, and the DSU is forced to use the local clock.

Step 2 Use the show interface serial EXEC command (see the "Interface and Line Protocol Status" section) to determine whether the line status changes from "line protocol is down" to "line protocol is up (looped)," or whether it remains down.

Step 3 If the line protocol comes up when the CSU or DSU is in local loopback mode, the problem might be occurring on the remote end of the serial connection. If the status line does not change state, there might be a problem in the ASI, MRP, router, connecting cable, or CSU/DSU.

Step 4 If the problem appears to be local, use the debug serial interface privileged EXEC command. (See the "Using debug Commands" section.)

Step 5 Take the CSU/DSU out of local loop mode. When the line protocol is down, the debug serial interface command output indicates that keepalive counters are not incrementing.

Step 6 Place the CSU/DSU in local loop mode again. This should cause the keepalive packets to begin to increment. Specifically, the values for mineseen and yourseen keepalives will increment every 10 seconds. This information will appear in the debug serial interface output.

If the keepalives do not increment, there may be a timing problem on the MRP card or on the network. For information on correcting timing problems, see the "Troubleshooting Clocking Problems" section.

Step 7 Check the local router and CSU/DSU hardware and any attached cables. Ensure that the cables are within the recommended lengths (no more than 50 ft [15.24 m], or 25 ft [7.62 m] for a T1 link). Verify that the cables are attached to the proper ports. Swap faulty equipment as necessary.


Example 6-6 shows the output from the debug serial interface command for an HDLC serial connection, with missed keepalives (callouts 1 through 4) causing the line to go down (callout 5) and the interface to reset.

Example 6-6 debug serial interface Command Output

Cisco ICS 7750# debug serial interface
Serial1: HDLC myseq 636119, mineseen 636119, yourseen 515032, line up
Serial1: HDLC myseq 636120, mineseen 636120, yourseen 515033, line up
Serial1: HDLC myseq 636121, mineseen 636121, yourseen 515034, line up
Serial1: HDLC myseq 636122, mineseen 636122, yourseen 515035, line up
Serial1: HDLC myseq 636123, mineseen 636123, yourseen 515036, line up
Serial1: HDLC myseq 636124, mineseen 636124, yourseen 515037, line up
Serial1: HDLC myseq 636125, mineseen 636125, yourseen 515038, line up
Serial1: HDLC myseq 636126, mineseen 636126, yourseen 515039, line up
Serial1: HDLC myseq 636127, mineseen 636127, yourseen 515040, line up
Serial1: HDLC myseq 636128, [1] mineseen 636127, yourseen 515041, line up
Serial1: HDLC myseq 636129, mineseen 636129, yourseen 515042, line up
Serial1: HDLC myseq 636130, mineseen 636130, yourseen 515043, line up
Serial1: HDLC myseq 636131, [2] mineseen 636130, yourseen 515044, line up
Serial1: HDLC myseq 636132, [3] mineseen 636130, yourseen 515045, line up
Serial1: HDLC myseq 636133, [4] mineseen 636130, yourseen 515046, [5] line down

Remote Loopback Tests for HDLC or PPP Links

If you determine that the local hardware is functioning properly but you still encounter problems when attempting to establish connections over the serial link, try using the remote loopback test to isolate the problem.


Note This remote loopback test assumes that HDLC encapsulation is being used and that the preceding local loopback test (see the "Local Loopback Tests for HDLC or PPP Links" section) was performed immediately before this test.



Step 1 Put the remote CSU or DSU into loopback mode (refer to the vendor documentation).

Step 2 Using the show interface serial EXEC command, determine whether the line protocol remains up, with the status line indicating "Serial x is up, line protocol is up (looped)," or whether it goes down, with the status line indicating "line protocol is down."

Step 3 If the line protocol remains up (looped), the problem is probably at the remote end of the serial connection (between the remote CSU/DSU and the remote router). Perform both local and remote tests at the remote end to isolate the problem source.

Step 4 If the line status changes to "line protocol is down" when remote loopback mode is activated, make certain that ones density is being properly maintained. The CSU/DSU must be configured to use the same framing and encoding schemes (such as ESF and B8ZS) used by the leased-line or other carrier service.

Step 5 If problems persist, contact your WAN network manager or the WAN service organization.