Catalyst 6500 Series Switch SIP, SSC, and SPA Software Configuration Guide
Troubleshooting the Serial SPAs
Downloads: This chapterpdf (PDF - 231.0KB) The complete bookPDF (PDF - 22.81MB) | Feedback

Troubleshooting the Serial SPAs

Table Of Contents

Troubleshooting the Serial SPAs

General Troubleshooting Information

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Performing Basic Interface Troubleshooting

Serial Lines: show interfaces serial Status Line Conditions

Serial Lines: Increasing Output Drops on Serial Link

Serial Lines: Increasing Input Drops on Serial Link

Serial Lines: Increasing Input Errors in Excess of 1 Percent of Total Interface Traffic

Serial Lines: Troubleshooting Serial Line Input Errors

Serial Lines: Increasing Interface Resets on Serial Link

Serial Lines: Increasing Carrier Transitions Count on Serial Link

Using Bit Error Rate Tests

Configuring a BER Test

Viewing a BER Test

Interpreting BER Test Results

Using loopback Commands

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Preparing for Online Insertion and Removal of a SPA


Troubleshooting the Serial SPAs


This chapter describes techniques that you can use to troubleshoot the operation of your serial SPAs.

It includes the following sections:

General Troubleshooting Information

Performing Basic Interface Troubleshooting

Using Bit Error Rate Tests

Using loopback Commands

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Preparing for Online Insertion and Removal of a SPA

The first section provides information about basic interface troubleshooting. If you are having a problem with your SPA, use the steps in the "General Troubleshooting Information" section to begin your investigation of a possible interface configuration problem.

To perform more advanced troubleshooting, see the other sections in this chapter.

For more information about troubleshooting serial lines, see the Internetwork Troubleshooting Handbook at: http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1915.html.

General Troubleshooting Information

This section describes general information for troubleshooting SIPs and SPAs. It includes the following sections:

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Interpreting Console Error Messages

To view the explanations and recommended actions for Catalyst 6500 Series switch error messages, including messages related to Catalyst 6500 Series switch SIPs and SPAs, see the Catalyst 6500 Series Cisco IOS System Message Guide, 12.2SX.

System error messages are organized in the documentation according to the particular system facility that produces the messages. The SIP and SPA error messages use the following facility names:

Cisco 7600 SIP-200—C7600_SIP200

2-Port and 4-Port Channelized T3 SPA—SPA_CHOC_DSX

Using debug Commands

Along with the other debug commands supported on the Catalyst 6500 Series switch, you can obtain specific debug information for SPAs on the Catalyst 6500 Series switch using the debug hw-module subslot privileged exec command.

The debug hw-module subslot command is intended for use by Cisco Systems technical support personnel. For more information about the debug hw-module subslot command and other debug commands, see the Cisco IOS Debug Command Reference, Release 12.2.


Caution Because debugging output is assigned high priority in the CPU process, it can cause the system to be unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. You should use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For information about other debug commands supported on the Catalyst 6500 Series switch, see the Catalyst 6500 Series Cisco IOS Command Reference, 12.2SX and any related feature documents for Cisco IOS Release 12.2 SX.

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the SIPs and SPAs on the Catalyst 6500 Series switch. This chapter describes using the show interfaces and show controllers commands to perform troubleshooting of your SPA.

For more information about show commands to verify and monitor SIPs and SPAs, see the following chapters of this guide:

Chapter 16, "Configuring the 2-Port and 4-Port Clear Channel T3/E3 SPAs"

Chapter 15, "Configuring the 8-Port Channelized T1/E1 SPA"

Chapter 17, "Configuring the 2-Port and 4-Port Channelized T3 SPAs"

Performing Basic Interface Troubleshooting

You can perform most of the basic interface troubleshooting using the show interfaces serial command and examining several areas of the output to determine how the interface is operating.

The output of the show interfaces serial exec command displays information specific to serial interfaces.


Note The output of the show interfaces serial command will vary depending on the type of serial SPA. Other fields that may be shown in the display are described in detail in the Catalyst 6500 Series Cisco IOS Command Reference, 12.2SX.


This section describes how to use the show interfaces serial command to diagnose serial line connectivity problems in a wide-area network (WAN) environment. The following sections describe some of the important fields of the command output:

Serial Lines: show interfaces serial Status Line Conditions

Serial Lines: Increasing Output Drops on Serial Link

Serial Lines: Increasing Input Drops on Serial Link

Serial Lines: Increasing Input Errors in Excess of 1 Percent of Total Interface Traffic

Serial Lines: Troubleshooting Serial Line Input Errors

Serial Lines: Increasing Interface Resets on Serial Link

Serial Lines: Increasing Carrier Transitions Count on Serial Link

Serial Lines: show interfaces serial Status Line Conditions

You can identify five possible problem states in the interface status line of the show interfaces serial display:

Serial x is down, line protocol is down

Serial x is up, line protocol is down

Serial x is up, line protocol is up (looped)

Serial x is up, line protocol is down (disabled)

Serial x is administratively down, line protocol is down

The following example shows the interface statistics on the first port of a T3/E3 SPA installed in subslot 0 of the SIP located in chassis slot 5.

Router# show interfaces serial

Serial5/0/0 is up, line protocol is up 
  Hardware is SPA-4T3E3
  Internet address is 110.1.1.2/24
  MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, 
     reliability 255/255, txload 234/255, rxload 234/255
  Encapsulation HDLC, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:05, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 40685000 bits/sec, 115624 packets/sec
  5 minute output rate 40685000 bits/sec, 115627 packets/sec
     4653081241 packets input, 204735493724 bytes, 0 no buffer
     Received 4044 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     4652915555 packets output, 204728203520 bytes, 0 underruns
     0 output errors, 0 applique, 4 interface resets
     0 output buffer failures, 0 output buffers swapped out
     2 carrier transitions

Table 19-1 shows the interface status conditions, possible problems associated with the conditions, and solutions to those problems

Table 19-1 Serial Lines: show interfaces serial Status Line Conditions 

Status Line
Condition
Possible Problem
Solution

Serial x is up, line protocol is up

This is the proper status line condition. No action is required.

Serial x is down, line protocol is down

The switch is not sensing a carrier detect (CD) signal (that is, the CD is not active).

The line is down or is not connected on the far end.

Cabling is faulty or incorrect.

Hardware failure has occurred (CSU/DSU).

1. Check the CD LEDs to see whether the CD is active, or insert a breakout box on the line to check for the CD signal.

2. Verify that you are using the proper cable (see your hardware installation documentation).

3. Insert a breakout box and check all control leads.

4. Contact your leased-line or other carrier service to see whether there is a problem.

5. Swap faulty parts.

6. If you suspect faulty switch hardware, change the serial line to another port. If the connection comes up, the previously connected interface has a problem.

Serial x is up, line protocol is down

A local or remote switch is misconfigured.

Keepalives are not being sent by the remote router.

A leased-line or other carrier service problem has occurred (noisy line or misconfigured or failed switch).

A timing problem has occurred on the cable.

A local or remote CSU/DSU has failed.

Router hardware (local or remote) has failed.

1. Put the line in local loopback mode and use the show interfaces serial command to determine whether the line protocol comes up.

Note If the line protocol comes up, a failed remote device is the likely problem.

This solution will only work with HDLC encapsulation. For FR and PPP encapsulation a looped interface will always have the line protocol down. In addition, you may need to change the encapsulation to HDLC to debug this issues.

2. If the problem appears to be on the remote end, repeat Step 1 on the remote interface.

3. Verify all cabling. Make certain that the cable is attached to the correct interface, the correct CSU/DSU, and the correct remote termination point.

4. Enable the debug serial interface exec command.

Note First enable per- interface debugging using the debug interface serial x command. Depending on the encapsulation, also enable the corresponding debug.

HDLC: debug serial interface
PPP: debug ppp negotiation
FR: debug frame-relay lmi


Caution Because debugging output is assigned high priority in the CPU process, it can cause the system to become unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. You should use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.
   

5. If the line protocol does not come up in local loopback mode, and if the output of the debug serial interface exec command shows that the keepalive counter is not incrementing, a switch hardware problem is likely. Swap switch interface hardware.

   

6. If the line protocol comes up and the keepalive counter increments, the problem is not in the local switch. Troubleshoot the serial line, as described in the sections "Troubleshooting Clocking Problems" and "CSU and DSU Loopback Tests," later in this chapter.

7. If you suspect faulty switch hardware, change the serial line to an unused port. If the connection comes up, the previously connected interface has a problem.

Serial x is up, line protocol is up (looped)

A loop exists in the circuit. The sequence number in the keepalive packet changes to a random number when a loop is initially detected. If the same random number is returned over the link, a loop exists.

1. Use the show running-config privileged exec command to look for any loopback interface configuration command entries.

2. If you find a loopback interface configuration command entry, use the no loopback interface configuration command to remove the loop.

3. If you do not find the loopback interface configuration command, examine the CSU/DSU to determine whether they are configured in manual loopback mode. If they are, disable manual loopback.

4. Reset the CSU or DSU, and inspect the line status. If the line protocol comes up, no other action is needed.

5. If the CSU or DSU is not configured in manual loopback mode, contact the leased-line or other carrier service for line troubleshooting assistance.

Serial x is up, line protocol is down (disabled)

A high error rate has occurred due to a remote device problem.

A CSU or DSU hardware problem has occurred.

Switch hardware (interface) is bad.

1. Troubleshoot the line with a serial analyzer and breakout box.
Examine the output of
show controller T1 or
show controller T3 or
show controller serial x depending on whether the SPA is a T1/E1 or CT3 or T3/E3.

2. Loop CSU/DSU (DTE loop). If the problem continues, it is likely that there is a hardware problem. If the problem does not continue, it is likely that there is a telephone company problem.

3. Swap out bad hardware, as required (CSU, DSU, switch, local or remote switch).

Serial x is administratively down, line protocol is down

The switch configuration includes the shutdown interface configuration command.

A duplicate IP address exists.

1. Check the switch configuration for the shutdown command.

2. Use the no shutdown interface configuration command to remove the shutdown command.

3. Verify that there are no identical IP addresses using the show running-config privileged exec command or the show interfaces exec command.

4. If there are duplicate addresses, resolve the conflict by changing one of the IP addresses.


Serial Lines: Increasing Output Drops on Serial Link

Output drops appear in the output of the show interfaces serial command when the system is attempting to hand off a packet to a transmit buffer but no buffers are available.

Symptom: Increasing output drops on serial link.

Table 19-2 shows the possible problem that might cause this symptom and describes solutions to that problem.

Table 19-2 Serial Lines: Increasing Output Drops on Serial Link 

Possible Problem
Solution

Input rate to serial interface exceeds bandwidth available on serial link

1. Minimize periodic broadcast traffic, such as routing and SAP1 updates, by using access lists or by other means. For example, to increase the delay between SAP updates, use the ipx sap-interval interface configuration command.

2. Increase the output hold queue size in small increments (for instance, 25 percent), using the hold-queue out interface configuration command.

3. Implement priority queuing on slower serial links by configuring priority lists. For information on configuring priority lists, see the Cisco IOS configuration guides and command references.

Note: Output drops are acceptable under certain conditions. For instance, if a link is known to be overused (with no way to remedy the situation), it is often considered preferable to drop packets than to hold them. This is true for protocols that support flow control and can retransmit data (such as TCP/IP and Novell IPX2 ). However, some protocols, such as DECnet and local-area transport, are sensitive to dropped packets and accommodate retransmission poorly, if at all.

1 SAP = Service Advertising Protocol

2 IPX = Internetwork Packet Exchange


Serial Lines: Increasing Input Drops on Serial Link

Input drops appear in the output of the show interfaces serial exec command when too many packets from that interface are still being processed in the system.

Symptom: Increasing number of input drops on serial link.

Table 19-3 shows the possible problems that might cause this symptom and describes solutions to that problem.

Table 19-3 Serial Lines: Increasing Input Drops on Serial Link 

Possible Problem
Solution

Input rate exceeds the capacity of the switch, or input queues exceed the size of output queues

Note: Input drop problems are typically seen when traffic is being routed between faster interfaces (such as Ethernet, Token Ring, and FDDI1 ) and serial interfaces. When traffic is light, there is no problem. As traffic rates increase, backups start occurring. Switches drop packets during these congested periods.

1. Increase the output queue size on common destination interfaces for the interface that is dropping packets. Use the hold-queue number out interface configuration command. Increase these queues by small increments (for instance, 25 percent) until you no longer see drops in the show interfaces output. The default output hold queue limit is 40 packets.

2. Reduce the input queue size, using the hold-queue number in interface configuration command, to force input drops to become output drops. Output drops have less impact on the performance of the switch than do input drops. The default input hold queue is 75 packets.

1 FDDI = Fiber Distributed Data Interface


Serial Lines: Increasing Input Errors in Excess of 1 Percent of Total Interface Traffic

If input errors appear in the show interfaces serial output, there are several possible sources of those errors. The most likely sources are summarized in Table 19-4.


Note Any input error value for cyclic redundancy check (CRC) errors, framing errors, or aborts above 1 percent of the total interface traffic suggests some kind of link problem that should be isolated and repaired.


Symptom: Increasing number of input errors in excess of 1 percent of total interface traffic.

Table 19-4 Serial Lines: Increasing Input Errors in Excess of 1 Percent of Total Interface Traffic 

Possible Problem
Solution

The following problems can result in this symptom:

Faulty telephone company equipment

Noisy serial line

Incorrect clocking configuration

Incorrect cable or cable that is too long

Bad cable or connection

Bad CSU or DSU

Bad switch hardware

Data converter or other device being used between switch and DSU

Note: Cisco strongly recommends against the use of data converters when you are connecting a switch to a WAN or a serial network.

1. Use a serial analyzer to isolate the source of the input errors. If you detect errors, there likely is a hardware problem or a clock mismatch in a device that is external to the switch.

2. Use the loopback and ping tests to isolate the specific problem source. For more information, see the sections "Using Extended ping Tests" and "CSU and DSU Loopback Tests," later in this chapter.

3. Look for patterns. For example, if errors occur at a consistent interval, they could be related to a periodic function, such as the sending of routing updates.


Serial Lines: Troubleshooting Serial Line Input Errors

Table 19-5 describes the various types of input errors displayed by the show interfaces serial command, possible problems that might be causing the errors, and solutions to those problems.

Table 19-5 Serial Lines: Troubleshooting Serial Line Input Errors 

Input Error Type
(Field Name)
Possible Problem
Solution

CRC errors (CRC)

CRC errors occur when the CRC calculation does not pass (indicating that data is corrupted) for one of the following reasons:

The serial line is noisy.

The serial cable is too long, or the cable from the CSU/DSU to the switch is not shielded.

SCTE mode is not enabled on DSU.

The CSU line clock is incorrectly configured.

A ones density problem has occurred on the T1 link (incorrect framing or coding specification).

1. Ensure that the line is clean enough for transmission requirements. Shield the cable, if necessary.

2. Make sure that the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for a T1 link).

3. Ensure that all devices are properly configured for a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the section "Inverting the Transmit Clock," later in this chapter.

4. Make certain that the local and remote CSU/DSU are configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS).

5. Contact your leased-line or other carrier service, and have it perform integrity tests on the line.

Framing errors (frame)

A framing error occurs when a packet does not end on an 8-bit byte boundary for one of the following reasons:

The serial line is noisy.

The cable is improperly designed; the serial cable is too long; the cable from the CSU or DSU to the switch is not shielded.

SCTE mode is not enabled on the DSU; the CSU line clock is incorrectly configured; one of the clocks is configured for local clocking.

A ones density problem has occurred on the T1 link (incorrect framing or coding specification).

1. Ensure that the line is clean enough for transmission requirements. Shield the cable, if necessary. Make certain that you are using the correct cable.

2. Make sure that the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for a T1 link).

3. Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the section "Inverting the Transmit Clock," later in this chapter.

4. Make certain that the local and remote CSU/DSU is configured for the same framing and coding scheme as that used by the leased-line or other carrier service (for example, ESF1 /B8ZS2 ).

5. Contact your leased-line or other carrier service, and have it perform integrity tests on the line.

Aborted transmission (abort)

Aborts indicate an illegal sequence of 1 bit (more than seven in a row)

The following are possible reasons for this to occur:

SCTE mode is not enabled on DSU.

The CSU line clock is incorrectly configured.

The serial cable is too long, or the cable from the CSU or DSU to the switch is not shielded.

A ones density problem has occurred on the T1 link (incorrect framing or coding specification).

A packet terminated in middle of transmission (typical cause is an interface reset or a framing error or a buffer overrun).

A hardware problem has occurred (bad circuit, bad CSU/DSU, or bad sending interface on remote switch).

1. Ensure that all devices are properly configured to use a common line clock. Set SCTE on the local and remote DSU. If your CSU/DSU does not support SCTE, see the section "Inverting the Transmit Clock," later in this chapter.

2. Shield the cable, if necessary. Make certain that the cable is within the recommended length (no more than 50 feet [15.24 meters], or 25 feet [7.62 meters] for a T1 link). Ensure that all connections are good.

3. Check the hardware at both ends of the link. Swap faulty equipment, as necessary.

4. Lower data rates and determine whether aborts decrease.

5. Use local and remote loopback tests to determine where aborts are occurring (see the section "Special Serial Line Tests," later in this chapter).

6. Contact your leased-line or other carrier service, and have it perform integrity tests on the line.

1 ESF = Extended Superframe Format

2 B8ZS = binary eight-zero substitution


Serial Lines: Increasing Interface Resets on Serial Link

Interface resets that appear in the output of the show interfaces serial exec command are the result of missed keepalive packets.

Symptom: Increasing interface resets on serial link.

Table 19-6 shows the possible problems that might cause this symptom and describes solutions to those problems.

Table 19-6 Serial Lines: Increasing Interface Resets on Serial Link

Possible Problem
Solution

The following problems can result in this symptom:

Congestion on link (typically associated with output drops)

Bad line causing CD transitions

Possible hardware problem at the CSU, DSU, or switch

When interface resets are occurring, examine other fields of the show interfaces serial command output to determine the source of the problem. Assuming that an increase in interface resets is being recorded, examine the following fields:

1. If there is a high number of output drops in the show interfaces serial output, see the section "Serial Lines: Increasing Output Drops on Serial Link," earlier in this chapter.

2. Check the Carrier Transitions field in the show interfaces serial display. If carrier transitions are high while interface resets are being registered, the problem is likely to be a bad link or a bad CSU or DSU. Contact your leased-line or carrier service, and swap faulty equipment, as necessary.

3. Examine the Input Errors field in the show interfaces serial display. If input errors are high while interface resets are increasing, the problem is probably a bad link or a bad CSU/DSU. Contact your leased-line or other carrier service, and swap faulty equipment, as necessary.


Serial Lines: Increasing Carrier Transitions Count on Serial Link

Carrier transitions appear in the output of the show interfaces serial exec command whenever there is an interruption in the carrier signal (such as an interface reset at the remote end of a link).

Symptom: Increasing carrier transitions count on serial link.

Table 19-7 shows the possible problems that might cause this symptom and describes solutions to those problems.

Table 19-7 Serial Lines: Increasing Carrier Transitions Count on Serial Link

Possible Problem
Solution

The following problems can result in this symptom:

Line interruptions due to an external source (such as physical separation of cabling, red or yellow T1 alarms, or lightning striking somewhere along the network)

Faulty switch, DSU, or switch hardware

1. Check hardware at both ends of the link (attach a breakout box or a serial analyzer, and test to determine the source of problems).

2. If an analyzer or breakout box is incapable of identifying any external problems, check the switch hardware.

3. Swap faulty equipment, as necessary.


Using Bit Error Rate Tests

Bit Error Rate (BER) test circuitry is built into the 4-Port Clear Channel T3/E3 SPA. With BER tests, you can test cables and signal problems in the field. You can configure individual T1 lines to run BER tests, but only one BER test circuit exists for all 28 T1 lines. Only one BER test can be run on a single T3 port at any given time.

There are two categories of test patterns that can be generated by the onboard BER test circuitry: pseudorandom and repetitive. Pseudorandom test patterns are exponential numbers and conform to the CCITT/ITU O.151 and O.153 specifications; repetitive test patterns are all zeros, all ones, or alternating zeros and ones.

A description of each type of test pattern follows:

Pseudorandom test patterns:

2^15 (per CCITT/ITU O.151)

2^20 (per CCITT/ITU O.153)

2^23 (per CCITT/ITU O.151)

Repetitive test patterns:

All zeros (0s)

All ones (1s)

Alternating zeros (0s) and ones (1s)

Both the total number of error bits received and the total number of bits received are available for analysis. You can set the testing period from 1 minute to 14,400 minutes (240 hours), and you can also retrieve the error statistics anytime during the BER test.

When running a BER test, your system expects to receive the same pattern that it is transmitting. To help ensure this:

Use a loopback at a location of your choice in the link or network. To see how to configure a loopback, go to the "Using loopback Commands" section.

Configure remote testing equipment to transmit the same BER test pattern at the same time.

Configuring a BER Test

To send a BER test pattern on an interface, use the bert pattern command as described in the Cisco IOS Interface and Hardware Component Command Reference at this URL:

http://www.cisco.com/en/US/docs/ios/interface/command/reference/ir_a1.html#wp1013940

Viewing a BER Test

You can view the results of a BER test with the show controllers command.

You can view the results of a BER test at the following times:

After you terminate the test using the no t1 bert command.

After the test runs completely.

Anytime during the test (in real time).


Router# show controllers serial T3 1/0/0
T3 1/0/0 is up.
C2T3 H/W Version : 3, C2T3 ROM Version : 0.79, C2T3 F/W Version : 0.29.0
T3 1/0/0 T1 1
No alarms detected.
Clock Source is internal.
BERT test result (running)
   Test Pattern : 2^15, Status : Sync, Sync Detected : 1
   Interval : 5 minute(s), Time Remain : 5 minute(s)
   Bit Errors(Since BERT Started): 6 bits,
   Bits Received(Since BERT start): 8113 Kbits
   Bit Errors(Since last sync): 6 bits
   Bits Received(Since last sync): 8113 Kbits 

Interpreting BER Test Results

Table 19-8 explains the output of the preceding command.

Table 19-8 Interpreting BER Test Results 

Field
Description

BERT test result (running)

Indicates the current state of the test. In this case, "running" indicates that the BER test is still in progress. After a test is completed, "done" is displayed.

Test Pattern : 2^15, Status : Sync, Sync Detected : 1

Indicates the test pattern you selected for the test (2^15), the current synchronization state (sync), and the number of times synchronization has been detected during this test (1).

Interval : 5 minute(s), Time Remain : 5 minute(s)

Indicates the time the test takes to run and the time remaining for the test to run.

If you terminate a BER test, you receive a message similar to the following:

Interval : 5 minute(s), Time Remain : 2 minute(s) (unable to complete)

"Interval: 5 minutes" indicates the configured run time for the test. "Time Remain : 2 minutes" indicates the time remaining in the test prior to termination. "(Unable to complete)" signifies that you interrupted the test.

Bit Errors(Since BERT Started): 6 bits, Bits Received(Since BERT start): 8113 Kbits
Bit Errors(Since last sync): 6 bits
Bits Received(Since last sync): 8113 Kbits

These four lines show the bit errors that have been detected versus the total number of test bits that have been received since the test started and since the last synchronization was detected.


Using loopback Commands

Loopback support is useful for testing the interface without connectivity to the network, or for diagnosing equipment malfunctions between the interface and a device. The 2-Port and 4-Port Clear Channel T3/E3 SPA supports both an internal and an external loopback mode. The external loopback mode requires the use of a loopback cable and implements a loopback through the transceiver on the SPA.

You can also configure an internal loopback without the use of a loopback cable that implements a loopback at the PHY device internally. By default, loopback is disabled.

To configure local loopback, perform this task:

Command
Purpose

Router# configure terminal

Enters global configuration mode.

T3/E3

Router(config)# interface serial slot/subslot/port

T1/E1

Router(config)# controller slot/subslot/port

Selects the interface to configure.

slot/subslot/portSpecifies the location of the interface.

slot/subslot/portSpecifies the location of the T1/E1 controller.

T3/E3

Router(config-if)# loopback {local | dte | network {line | payload} | remote}

T1/E1

Router(config-controller)# loopback {local [line | payload] | diag}

Specifies the loopback mode.

local—Loopback after going through the framer toward the terminal.

dte—Loopback after the LIU towards the terminal.

network—Loopback towards the network.

remote—Send FEAC to set remote in loopback.

line—Loopback toward network before going through framer.

payload—Loopback toward network after going through framer.

diag—Loopback after going through the HDLC controller towards the terminal.


Verifying Loopback Mode

This example shows how to verify loopback mode:

Router# show interfaces serial 6/0/0
Serial6/0/0 is up, line protocol is up (looped)
  Hardware is SPA-4T3E3
  MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, 
     reliability 255/255, txload 225/255, rxload 221/255
  Encapsulation FRAME-RELAY, crc 16, loopback set (local)
  Keepalive set (10 sec)
  LMI enq sent  13281, LMI stat recvd 13280, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 1, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/256, broadcasts sent/dropped 0/0, interface broadcasts 0
  Last input 00:00:07, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d12h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 612756
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 38446000 bits/sec, 109217 packets/sec
  5 minute output rate 39023000 bits/sec, 110854 packets/sec
     14601577951 packets input, 642478074437 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicast)
     0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     14545044296 packets output, 639982568049 bytes, 0 underruns
     0 output errors, 0 applique, 1 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
   rxLOS inactive, rxLOF inactive, rxAIS inactive
   txAIS inactive, rxRAI inactive, txRAI inactive

Using the Cisco IOS Event Tracer to Troubleshoot Problems


Note This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.


The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the "spa" component to trace SPA OIR-related events.

For more information about using the Event Tracer feature, refer to the following URL:

http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_event_tracer_ps6441_TSD_Products_Configuration_Guide_Chapter.html

Preparing for Online Insertion and Removal of a SPA

The Catalyst 6500 Series switch supports online insertion and removal (OIR) of the SIP, in addition to each of the SPAs. Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the switch.

This means that a SIP can remain installed in the switch with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

For more information about activating and deactivating SPAs in preparation for OIR, see the "Preparing for Online Insertion and Removal of SIPs and SPAs" topic in the "Troubleshooting a SIP" chapter in this guide.