BXM Trunk Troubleshooting [Cisco BPX 8600 Series Switches]

Document ID: 40540

Introduction
Prerequisites
      Requirements
      Components Used
      Conventions
Error Descriptions
Troubleshooting Causes of Errors
Related Information

Introduction

This document describes how to troubleshoot errors seen on a BPX BXM trunk in order to narrow down the source of the errors. Many of the error definitions apply to transmission lines generally, and can be seen on both IGX trunks as well as BPX trunks. The troubleshooting described in this document can generally also be applied to IGX trunk troubleshooting when you attempt to narrow down the source of errors.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Error Descriptions

This list describes possible BPX BXM trunk errors:

Out of Frms

The Out of Frames (OOF) or Loss of Frame (LOF) condition occurs when errors in the framing pattern are detected by the BXM backcard receiver. This error represents consecutive errors in the terminal framing bits. For more information, refer to BTM Out of Frame Errors.
Loss of Sig

This statistical trunk error indicates that a loss of signal (LOS) has occurred. The BXM logs an LOS error when the signal level at the receiver falls below an acceptable level. LOS is a physical layer error and typically results in an integrated alarm. Refer to BTM Loss of Signal Errors for more information.
Line Code Errs (E3 and T3 back cards only)

The Line Code Err indicates a bipolar violation (BPV) in the line coding for the BXM trunk. For T3 trunks, bipolar with three zero substitution (B3ZS) line coding is used. For E3 trunks, high density bipolar of order three (HDB3) line coding is used.

A BPV error event for a B3ZS- or HDB3-coded signal is the occurrence of a pulse of the same polarity as the previous pulse that is not a part of the zero substitution code. A BPV typically indicates that a data error has occurred. BPVs cannot pass through higher order transmissions systems. As a result, BPVs cannot be used to determine end-to-end performance. BPVs indicate problems in near-end facilities.

A Line Code Err typically involves a local BXM receiver, cable, or network termination (NT) transmitter. Multiplexers on the path do not relay the errors to the next cable segment.

For more information, refer to BTM Line Code Errors.
P-bit Parity (E3 and T3 back cards only)

The P-bit Parity Errs counter indicates that in-service bit errors were received during transmission. There are two P bits (parity bits) that contain parity information in the DS3 frame. The P bits are located in the first bit position in block 1 of subframes 3 and 4. The DS3 source computes parity over all DS3 information bits after the first X bit in a DS3 frame. The resulting parity information is inserted in the two P bits of the the next frame. The value of both P bits is always the same. Both P bits equal 1 if the previous DS3 frame contained an odd number of ones. Both P bits equal 0 if the previous DS3 frame contained an even number of ones. Because P bits are recomputed by each facility section of the DS3 path, they do not provide end-to-end path monitoring. For more information, refer to BTM P-bit Parity Errors.
C-bit Parity (T3 back cards only)

The C-bit Parity Errs counter indicates that in-service, end-to-end bit errors have been received during transmission. There are three C bits (control bits) that control bit stuffing in each DS3 subframe. The BXM only supports the DS3 C bit parity-frame format, which does not require the 21 C bits for bit-stuffing control. The DS3-level C bits are used for in-service, end-to-end path performance monitoring and in-band data links. The three C bits in subframe 3 are called CP (C bit Parity) bits and are used for DS3 path parity. At the DS3 transmitter, the CP bits are set to the same value as the two P bits and are not recomputed in the network. Because CP bits are not changed by transmission equipment, they provide end-to-end path monitoring when evaluated at the receiver. DS3 C bit parity frame-format offers significant advantages over the M13 frame format. M13 only supports P-bit parity for local segment error detection. Refer to BTM C-bit Parity Errors for more information.
BIP-8 Code (E3 and T3 back cards only)

Bit-interleaved parity with eight bit errors (BIP-8) indicates errors in the Physical Layer Convergence Protocol (PLCP) framing. PLCP framing is used to map asynchronous transfer mode (ATM) cells into E3 or DS3 frames. The BXM does not support direct mapping of ATM cells into the DS3 frame. PLCP framing is generated and used by the terminating BXMs. PLCP errors can occur along the circuit path as well as at the terminating BXM card sets. For more information, refer to UXM/BTM BIP-8 Code Errors.
Comm Fails

The BXM logs a Comm Fail error when the communication failure test (initiated depending on node parameter settings by SWSW) fails. The BPX BCC card uses a Comm Fail test to verify whether a trunk is usable for communication between nodes using the same trunk. A good explanation of how these work can be found with reference to the UXM Comm Fail Errors. The Comm Fail test works in exactly the same way for the BPX and IGX. Refer to BTM Comm Fail Errors for more information.
Loss of SIG (RED)

This integrated trunk error indicates that there has been an LOS. The BXM declares an LOS alarm when the signal level at the receiver falls below a minimum acceptable level. An LOS is a Physical Layer error and typically results in an integrated alarm. For more information, refer to BTM Loss of Sig (Red) Error.
AIS (BLUE)

The Blue Signal, also known as an Alarm Indication Signal (AIS), indicates one of these problems:
- Equipment upstream from the trunk interface is in alarm.
- Equipment upstream from the trunk interface is functional, but an intermediate device is in alarm.
The term upstream refers to the relative position of a piece of transmission equipment in the network. The location of the BXM is:
- Downstream from the nearest piece of transmission equipment that transmits towards the BXM.
- Upstream from the nearest piece of transmission equipment the BXM transmits to.
Transmission equipment generates an AIS in the downstream direction if it cannot recover from a problem that occurs with the upstream signal. Transmission equipment includes multiplexers, channel service units (CSUs), and digital cross-connect systems (DCS).

Error conditions such as an LOS or LOF do not allow the trunk to deliver the signal received from the equipment upstream to the downstream equipment.

For more information, refer to BTM AIS (Blue) Errors.
Out of Frms (RED)

The OOF or LOF condition occurs when the BXM backcard receiver detects a loss of frame synchronization. This error represents consecutive errors in terminal framing bits. For more information, refer to BTM Out of Frms (RED) Error.
Remote OOF (YEL)

The yellow alarm also is known as a remote alarm indication (RAI) signal as well as FERF (Far End Receive Fail) or 'distant'. The alarm transmits in the opposite direction when a terminal determines the incoming signal is lost. The yellow alarm returns to the transmitting terminal to indicate the receiving terminal has lost the signal.

If the local dsptrks screen indicates a trunk is in Rmt Alarms (YEL), the remote node dsptrks screen for the trunk typically has one of these alarms:
- Loss of Sig (Red)
- AIS (Blu)
- Out of Frms (Red)
For more information, refer to BTM Rmt E3 FERF / Rmt OOF (YEL) Alarm.
Loss of Cell

A Loss of Cell error message indicates that the trunk receiver is unable to recognize cell boundaries.

On OC3, E3, and T3 direct mapping ATM interfaces, the ATM cells are not aligned to the physical layer framing. The receiver has to match the header error control (HEC) field value with the previous four bytes received to find the beginning of the ATM cells. Those four bytes are a potential ATM cell header. If this HEC value is valid again 53 bytes further downstream, and continues to be, the mechanism locks in and the receiver has achieved cell synchronization.

If the mechanism does not work, a Loss of Cell error occurs. Possible causes include these:
- High rates of transmission errors
- Hardware malfunction
- Clocking misconfiguration
HCS Errors

ATM interfaces protect against changes to the cell header with a header error checksum (HCS) field. The HCS detects errors only in the header and not in the 48-byte payload. HCS errors indicate that the source, destination, or ATM network corrupted the cell header in some way.
PLCP Out of Frame (T3 Back cards only)

The BXM T3 transports ATM cells with Physical Layer Convergence Protocol (PLCP) framing. The BXM T3 does not map ATM cells directly onto the T3 frame. PLCP framing only achieves a cell transfer rate of 40.704 Mbps. The PLCP Out of Frame Errs occur after the receiver loses synchronization with the PLCP framing. Because PLCP framing is generated and used by the terminating BXMs, the problem can occur anywhere on the link between and as well as on the two trunk cards.
Loss of Pointer (OC3 and OC12 back cards only)

A pointer supports asynchronously timed, lower-rate signals and timing variations between synchronously timed network equipment in the Synchronous Optical Network (SONET)/Synchronous Digital Hierarchy (SDH). The pointer is contained in H1 and H2 bytes of the overhead line. If the local trunk has received an invalid path pointer, it will declare a Loss of Pointer condition. The pointer enables the receiver to locate the payload within the signal. Because the receiver cannot determine where the payload starts, all downstream traffic is lost as long as the condition persists.
Path AIS (OC3 and OC12 back cards only)

One of the SONET/SDH systems on the path from the remote UXM has a problem from which it cannot recover. It informs downstream systems of this situation with a path AIS. For more information, refer to UXM Path AIS Errors.
Path Yellow (OC3 and OC12 back cards only)

The Path Yellow error indicates that the remote end of the SONET/SDH path is in alarm. The alarm is generated by the BXM SONET Receive Framers by way of the G1 Path status byte. The G1 Path status byte conveys the path terminating status and performance back to the originating path terminating equipment. The duplex path in its entirety can be monitored from either end, or from any point along the path. For more information, refer to UXM Path Yellow Errors.

Troubleshooting Causes of Errors

For trunk errors or alarms seen on BXM trunk cards, you first need to prove if the errors or alarms are caused by intermediate equipment such as Telco multiplexers, cable cross-connect points, or by the cards themselves. In order to narrow down the source of the errors or alarms, it is necessary to try and prove that the source is external to the BPX. In order to do this you need to take these steps, during a maintenance window only, as this work is totally intrusive:

Let us use this scenario as an example—the dsptrkerrs 1.1 command on a BPX 'B' that shows a presence of errors.

This means that any location—from BXM card 10 at BPX 'A', and the entire transmission path (from BPX 'A' to BPX 'B'), and the interconnecting cables at both ends, as well as the BXM back card at BPX 'B'—could be responsible.

To begin to troubleshoot, disconnect the Tx and Rx cable from port 1.1 on card 1 at BPX 'B' and apply a physical loopback (this usually takes the form of a very short length of cable) to port 1.1 so that the Tx port is directly connected to the Rx port. BPX 'B' now shows the trunk as looped. For example:

B    StrataCom BPX 8620 9.2.40 Aug. 21 2002 13:59 GMT

TRK         Type         Current Line Alarm Status                     Other End
1.1         OC3          Major - Looped Back                           A/10.1
1.3         OC3          Clear - OK                                    A/10.3

Last Command: dsptrks

If you have the connectors available, it is also good idea at this point to apply a loopback to the cable tails you have just disconnected from the Tx and Rx ports on the BXM card, so that the end of the Tx cable attaches directly to the end of the Rx cable (which requires a loopback connector, or short piece of cable with a relevant connector type for whatever cable medium you currently use). Do not touch the fiber cable ends to ensure that new faults are not introduced by dirt and grease from your fingers.

BPX 'A' now shows the trunk as looped:

A    StrataCom BPX 8620 9.2.40 Aug. 21 2002 14:01 GMT

TRK         Type         Current Line Alarm Status                     Other End
10.1         OC3          Major - Looped Back                           B/1.1
10.3         OC3          Clear - OK                                    B/1.3

Last Command: dsptrks

Issue the clrtrkerrs 10.1 command on BPX 'A' and the clrtrkerrs 1.1 command on BPX 'B'. Then, monitor the looped trunks for errors with the dsptrkerrs 10.1 command on BPX 'A' and the dsptrkerrs 1.1 command on BPX 'B'. There is a second page of errors monitored, but if you want to watch the first page only, type n when asked 'Continue ?'—the screen then dynamically updates approximately every 10 seconds with the latest count of errors.

A    StrataCom BPX 8620 9.2.40 Aug. 21 2002 14:04 GMT

Trunk 10.1 Status: Major - Looped Back               Clrd: 08/21/02 13:29:41
Type             Count     ETS     Status       Type     Count        ETS     Status
Out of Frms    0         0                      Comm Fails             7         -
Loss of Sig     1         1                     Loss of Sig (RED)      7         -
                                                AIS(BLU)               0         -
                                                Out of Frms (RED)      0         -
                                                Remote (YEL)           0         -
                                                Loss of Cell           0         -
                                                Loss of Pointer        0         -
                                                Path AIS               0         -
                                                Path Yellow            1         -

This Command: dsptrkerrs 10.1

Continue? y

A    StrataCom BPX 8620 9.2.40 Aug. 21 2002 14:04 GMT

Trunk 10.1 Status: Major - Looped Back                 Clrd: 08/21/02 13:29:41
Type               Count     ETS     Status    Type  Count     ETS     Status
HCS Errors         79         1
BXM:Tx NTS CDs     0          0
BXM:Tx HP CDs      0          0
BXM:Tx V CDs       0          0
BXM:Tx TS CDs      0          0
BXM:Tx BDA CDs     0          0
BXM:Tx BDB CDs     0          0
BXM:Tx CBR CDs     0          0
BXM:Tx ABR CDs     0          0
BXM:Tx VBR CDs     0          0
Line Unavail Secs  24         6
Path Unavail Secs  24         6

This Command: dsptrkerrs 10.1

If no errors are displayed for BPX 'A' 10.1 and errors are displayed for BPX 'B' 1.1, you must suspect BXM card 1 on BPX node 'B' and arrange for replacement as necessary.

If errors are displayed for BPX 'A' 10.1 and errors are not displayed for BPX 'B' 1.1, you must continue to troubleshoot to find the source of the errors. You can run the above loops in reverse—apply a loop directly to the BPX 'A' 10.1 Tx and Rx ports, and a loopback on the cable ends you have just disconnected back to the BPX 'B' 1.1 Tx and Rx ports.

If no errors are now displayed for BPX 'A' 10.1 and errors are displayed when you issue the dsptrkerrs 1.1 command on BPX 'B', then you know for certain that you have proved both BXM cards at BPX nodes 'A' and 'B' to be good error-free cards and that the transmission path between the two BPXs is responsible for the errors.

Work with your Telco to continue to troubleshoot. You could provide assistance to them if you reconnect the trunk connectors back to the BXM cards at both ends and wait to see them apply loops at various stages along their network, and issue the clrtrkerrs command, and then the dsptrkerrs command to see at which point the errors cease or continue in much the same manner as described. Alternatively, you can apply physical loops to the Telco point of interconnect, and allow them to run tests around your loops.

Note: When you investigate physical layer problems such as AIS and LOS, this is associated with BXM back cards and the transmission path between the two, so if the troubleshooting steps have proven that a BXM card is at fault, it is the back card (which provides the physical interface) that you need to replace. When you investigate any other errors, and if it is proven that a BXM card is responsible, it is more likely to be a front card (which is responsible for cell and frame processing) that you need to replace. For information on how and where this takes place, refer to BXM Functional Description.

Related Information

Updated: Oct 04, 2005

Document ID: 40540

Hierarchical Navigation

Cisco BPX 8600 Series Switches