This chapter provides information about isolating faults in the Cisco 6130 with NI-2 system. Most problems in a Cisco 6130 with NI-2 system can be traced to one of the system's field-replaceable units (FRUs), which include the following hardware components:
Primary or secondary NI-2 card
Line cards
Quad-port Discrete Multitone (DMT) ATU-C line card (4xDMT)
Quad-port flexi ATU-C line card (4xflexi)
Quad-port STU-C line card (4xSDSL)
System I/O card
Fan tray
Air filter
Warning Only trained and qualified personnel should be allowed to install, replace, or service this equipment.
Alarms: Descriptions of alarms that can signal problems or help with troubleshooting
Hot-Swappable FRUs
All Cisco 6130 FRUs are hot swappable, except the system I/O card. However, hot swapping some FRUs causes an interruption in service. See Table 5-1 for more detailed information.
Table 5-1 Service Interruptions Caused by the Replacing of FRUs
FRU
Does Hot Swap Interrupt Service?
Notes
NI-2 card
Yes
If a secondary NI-2 card is installed (redundancy), it will automatically take over service and become the active NI-2 card. Otherwise, service is interrupted until the NI-2 card is replaced.
Line card
Yes
Service is interrupted only for subscribers served by that line card.
System I/O card
Yes
You must turn off system power to replace the system I/O card. Note that the MAC address is stored on the system I/O card.
Fan tray
No
Air filter
No
Basic Checks
Before using the troubleshooting tables in this chapter, make the following basic checks:
Are the ports properly configured? Refer to these sources for configuration instructions:
Are power leads and data cables firmly connected at both ends?
Are all cards firmly seated and locked (as necessary) in the chassis?
Is the system I/O card securely screwed to the chassis backplane?
Are the Cisco IOS images up-to-date? Refer to these sources for update instructions:
Configuration Guide for Cisco DSLAMs with NI-2
Command Reference for Cisco DSLAMs with NI-2
Contacting the Cisco TAC for Help
In certain situations, the troubleshooting tables in this chapter direct you to contact the Cisco Technical Assistance Center (TAC) for help. If you have a maintenance contract or if your hardware is under warranty, call the TAC at 1 800 553 2447 (North America only), 1 408 526 7209, or visit this URL for a worldwide list of TAC regional telephone numbers:
Use the following table to diagnose and troubleshoot any problems that affect the entire Cisco 6130 system.
Symptoms
Steps to Take
System fails to come up.
1. Check the POWER LED on the NI-2 card and the STATUS LEDs on the line cards. If all LEDs are off, troubleshoot the power source.
2. If any green LEDs are on, the system has power. Check the STATUS LED on the NI-2 card. If the STATUS LED is off, refer to the "NI-2 Card Problems" section.
3. If the Cisco 6130 is set to boot from a remote device over the network, make sure the remote device is up, that its network connection is solid, and that it contains the boot file. (The Cisco 6130 tries to boot over the network for a configured period, usually 5 to 15 minutes. If it is unable to boot over the network, it will eventually boot from bootflash.)
Examine the command prompt. If the prompt says rommon 1>, the problem could be in flash memory, in bootflash, in an incorrectly set boot configuration register, or in an incorrect file name in a boot system command in the startup-config file. Refer to the Configuration Guide for Cisco DSLAMs with NI-2 for information on setting and interpreting configuration registers, configuring flash memory, and editing the startup-config file.
To select an image to boot the system from flash, enter dir flash:. In the resulting display, find the name of the software image. Then enter boot flash:imagename, replacing imagename with the name of the software image. The system boots from flash.
If you see a normal Cisco IOS prompt, which usually contains the name of the system (default is DSLAM>), enter show oir status. If the results indicate that the card is loading software, wait a few minutes for the port to come back up.
You cannot establish a console or Telnet connection to the system.
If you are connecting to the Cisco 6130 through the Ethernet interface, check the configuration of your LAN for both the Cisco 6130 and the Telnet source.
If you are connecting to the Cisco 6130 through an ATM interface, make sure that PVCs are set up between the two devices and that the map-list is correctly configured. Enter the command show running-config to display this information.
If the Telnet source and the Cisco 6130 are on different networks, make sure static routes are configured at both ends so that the two devices can communicate. To check, use the ping command to ping each device from the other (that is, ping the Cisco 6130 from the Telnet source, and ping the Telnet source from the Cisco 6130). Alternatively, ping your default gateways from each end of the connection.
3. For both console and Telnet problems:
Check the cabling and connectors between the terminal or Telnet source and the Cisco 6130. See "Connector and Pinout Specifications" to check pinouts.
Press the Reset button on the NI-2 card faceplate to reset the card.
If the problem persists, replace the NI-2 card.
System experiences a critical, major, or minor alarm.
1. Enter the command show facility-alarm status. Note the affected slot and port, if any, and the description of the problem.
2. If no slot number is indicated, enter show environment all and examine the results for an indication of which FRU is at fault. Refer to the "Cisco 6130 Component-Specific Problems" section for instructions on troubleshooting that FRU.
3. If ATM0/0, ATM0/1, ATM0/2, or ATM0/3 is indicated:
Enter a show int command for the interface (for example, show int atm0/1). Results may indicate a SONET problem (Loss of Signal, for example). Refer to the "NI-2 Card Problems" section for instructions on troubleshooting the NI-2 card.
Enter show controllers commands for all trunk and subtending ports. (The ports are atm0/1, atm0/2, and, in a DS3 system, atm0/3.) For example, show controllers atm0/1. In the resulting display, check that the framing mode is set to the same value on this interface as at the other end of the connection. Also check that cell payload scrambling is on (on DS3 interfaces only).
2. To check the interface status and configuration, enter show interface atmslot#/port#. Check the following in the resulting display:
If the port Admin Status is down, enter the commands below to correct the problem, replacing slot/port ID atm 0/1 with your slot/port ID:
DSLAM> configure terminal
Enter configuration commands, one per line.
End with CNTL/Z.
DSLAM(config-if)# int atm 0/1
DSLAM(config-if)# no shutdown
DSLAM(config-if)#
If the port IF Status is down, check for disconnected or faulty cables. (Optical cables connect to the NI-2 card; DS3 coaxial cables connect to the system I/O card on the back of the chassis.)
If the Line Protocol is down, the line protocol software processes might have determined that the line is unusable; try swapping the cable. Another possibility is that clocking might be misconfigured, or the clocking source might have failed.
Check the CRC field. The presence of many CRC errors but not many collisions is an indication of excessive noise. If the number is too high (greater than 0.5 to 2 percent of total traffic on the interface), check the cables to determine if any are damaged.
If you need more information on interface configuration, refer to the Configuration Guide for Cisco DSLAMs with NI-2 and the ATM Switch Router Software Configuration Guide.
3. For a DS3 interface: enter show controllers atmslot#/port#. Check the following in the resulting display:
Framing mode must be the same at both ends of the connection.
Cell payload scrambling must be On at both ends of the connection.
4. Check the status and configuration of the interface at the far end.
5. If you need to run a loopback test, do the following:
In interface configuration mode, enter loopback diagnostic or loopback line.
Set the external test equipment to loop data through the Cisco 6130 port.
Obtain loopback results from your external test equipment.
Enter no loopback diagnostic to take the port out of loopback mode.
6. In interface configuration mode, reset the trunk port by executing the shutdown command followed by the no shutdown command.
The following sections describe symptoms that might occur and the steps that you need to take if you experience problems with any of the following Cisco 6130 components:
1. Check the STATUS LEDs on the line cards. If all LEDs are off, troubleshoot the power source.
2. If the line card STATUS LEDs are lit, remove the NI-2 card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.
STATUS LED is off, indicating that the NI-2 card failed to boot or failed its power-on self test.
Press the Reset button on the NI-2 card. If the problem persists, replace the card.
Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.
NI-2 Card Cold Redundancy Problems
Use the following table to diagnose and troubleshoot any problems with an NI-2 cold redundancy configuration.
The software version on the primary and secondary NI-2 cards may be mismatched. Use the Cisco IOS command show version to determine the Cisco IOS release currently installed on each of the NI-2 cards. See the "System-Wide Problems" section for more information on upgrading software images.
Line Card Problems
Use the following table to diagnose and troubleshoot any problems with the line cards (4xDMT, 4xflexi, or 4xSDSL).
1. Check the POWER LED on the NI-2 card. If all LEDs are off, troubleshoot the power source.
2. If the NI-2 card POWER LED is off, remove the card from its slot and check for bent or broken pins on both the card and the backplane. If you find damaged pins on the card, replace it. If you find damaged pins on the backplane, contact the Cisco TAC.
All ports on a card fail to come up (modems do not train). STATUS LED might be red, indicating that the line card failed to boot or failed its power-on self test.
1. Enter the show ipc nodes command to find out whether there is a communication problem between the line card and the NI-2 card. There should be an entry in the resulting display for each line card ("SMB IP Slot n") and for the NI-2 card ("IPC Master"). If there is an entry for each card, go to Step 4.
2. If one or more cards are not listed, enter show oir status. If the resulting display indicates that the card in question is loading new code, wait a few minutes and reenter the command. (Usually, 2 to 3 minutes is long enough to wait.) The card's status should change to running.
3. Enter the command show dsl int atmslot#/port#. If the status says Microcode downloading, wait 10 minutes. When the download is complete, the card will reboot and come up normally.
4. Check the ALARM LEDs on the NI-2 card or enter show facility-alarm status to determine the alarm status of the system. If any alarms are indicated, see the "Alarms" section for instructions on how to troubleshoot alarms.
5. Enter the command show dsl status and examine the results to ensure that the card is configured to be in its current slot. (In the Names column, the slot number appears as part of the port ID. For example, in ATM19/2, the slot number is 19. Nothing is displayed for slots that are not configured.) If necessary, use the slot command to update the configuration, or move the card to the correct slot. If the line card is a 4xflexi, it will not function unless you use the slot command to configure either CAP or DMT operation.
6. Reset the line card by disconnecting it from the backplane and reseating it in its slot.
7. Install the card in another slot.
8. If the problem persists, replace the line card.
Port fails to come up (modems do not train), or port LED flashes continuously.
1. Enter the command show dsl int atmslot#/port# to display the port configuration. Check the display to ensure that the port is properly provisioned. Make sure the port is configured to be running ("no shutdown" or IOS admin state = up). Also check the line status; if this reads "No CPE detected," troubleshoot the CPE device and the loop as described below under the symptom "You suspect a problem with the CPE or the subscriber loop."
2. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command.
3. Connect the subscriber to another port on the Cisco 6130. If the modems do not train, troubleshoot the CPE device and the loop as described below under the symptom "You suspect a problem with the CPE or the subscriber loop." If the modems train, go to the next step to troubleshoot the port that failed.
4. To test the modem on the line card, enter dsl test atmslot#/port#self. Results are displayed automatically after a few seconds. If the modem fails the test, replace the line card.
You suspect a problem with the CPE or the subscriber loop.
1. Disconnect the local loop and replace it with a test setup that consists of a modem or CPE that is known to work and a few thousand feet of wire. If the modems train over the shorter distance, the problem lies in the local loop or in the CPEsee the steps that follow. If the modems do not train, the port is probably at fault. Replace the line card.
2. Make sure that the CPE at the subscriber site is powered up.
3. Make sure that the CPE is compatible with the Cisco 6130, and that the software version running on the CPE is compatible with the software version running on the Cisco 6130.
4. Power cycle the CPE. When you turn the power on, the WAN Link LED on the CPE should blink. If the LED does not blink, check the CPE's configurationif the interface is shut down, bring it up.
5. Perform a continuity check to find out if the cabling to the CPE is connected and intact. For example, if there is a phone on the line, check for a dial tone.
6. If there is no POTS, check DC resistance by shorting tip and ring at the remote end.
7. Make sure there are no load coils on the local loop.
8. Is the local loop too long? The maximum length range is 15,000 to 25,000 feet (4572 to 7620 meters). Within that range, wire gauge, crosstalk, and multiple bridge taps reduce the distance over which the modems can train.
9. Is the local loop too short? DMT modems train best over loops of several thousand feet. In a test situation with a loop just a few feet in length, the modems may fail to train. Add wire to the loop.
10. Replace the CPE.
Modems train at a low bit rate, or modems retrain continuously, or the line experiences too many errors.
1. Enter the command show dsl int atmslot#/port# to display the port configuration. Check the display to ensure that the port is properly provisioned. Look in particular for these statistics:
Attenuation: typically this is 20 to 50 dB. If the attenuation value is higher than 50 dB, it might be depressing the bit rate. Repair or replace the cables and connectors in the loop.
SNR margin: 3 to 6 dB is optimum. Use the dmt margin command to adjust SNR margin.
Correction ratio: under DSL Statistics, look at the Received Superframes and Corrected Superframes values. A ratio of more than 1 corrected superframe for every 10 superframes received is too high. One or more of these adjustments might correct the problem: increase the SNR margin using the dmt margin command; increase error correction using the dmt check-bytes command; or increase interleaving using the dmt interleaving-delay command.
Errored seconds: a rate of 10 to 20 errored seconds per minute or more is likely to cause retraining. (1 or 2 errored seconds every 15 minutes is a good rate.) See the steps that follow on crosstalk and impulse noise for suggestions on how to compensate.
CRC errors: normal rates vary system by system. If the CRC error rate is higher than usual, it might cause excessive retraining. See the steps that follow on crosstalk and impulse noise for suggestions on how to compensate.
2. Crosstalk is caused by interference between services in adjacent cables. It affects random bits rather than chunks of data; upstream and downstream traffic can be affected differently. If crosstalk is increasing the bit error rate (BER), you can compensate in several ways:
Reduce the bit rate using the dmt bitrate command. (See the Command Reference for Cisco DSLAMs with NI-2 for information on the dmt bitrate command.)
Increase the SNR margin using the dmt margin command. (See the Command Reference for Cisco DSLAMs with NI-2 for information on the dmt margin command.)
Turn on or increase error correction. Use the dmt check-bytes command. If the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting. (See the Command Reference for Cisco DSLAMs with NI-2 for information on these commands.)
3. If you experience impulse noise or clipping, both of which affect chunks of data rather than random bits, you can compensate by turning on or increasing interleaving and error correction. (However, note that this approach adds delay.) Use the dmt interleaving-delay command for interleaving. Use the dmt encoding-trellis command or the dmt check-bytes command for error correction. If you use dmt check-bytes and if the codeword size is not set to auto, you might need to use the dmt codeword-size command to adjust the codeword setting.
4. If errors or retraining occur while the line is ringing, use the dmt interleaving-delay command to turn on interleaving.
5. In interface configuration mode, reset the port by executing the shutdown command followed by the no shutdown command.
6. Connect the customer to a different port on the Cisco 6130. If the modems train, replace the line card with the faulty port.
1. Remove the card and reinsert it, pushing firmly on both the top and the bottom of the faceplate. (The card might jam in the slot if you apply pressure to the top only.) If the chassis is full, you might need to push sideways to insert the card.
2. Inspect connectors on both the card and the backplane, looking for obstructions, bent pins, or other damage. If you find damage to a line card connector, replace the line card. If you find damage to a backplane connector, contact the Cisco TAC.
Card experiences problems in one slot but operates normally in another.
There may be a fault in your backplane. Contact the Cisco TAC.
System I/O Card Problems
Use the following table to diagnose and troubleshoot any problems with the system I/O card.
1. Check the connection at the wire-wrap connector on the system I/O card. See Table 1-11 for the pin assignments of the wire wrap connector.
2. Check the connection at the other end of the circuit.
3. For a BITS problem, troubleshoot the T1 line at the other end of the circuit.
4. Replace the system I/O card (except in the case of a circuit breaker alarm problem; the system I/O card does not provide the circuit breaker trip alarm).
This message appears on the console or in the system log file:
00:00:28:%C6100-4-COOKIE:
Corrupt or missing MAC
address cookie using random
base 007e.eb7d.e700
1. Make sure the software release running on your system is compatible with the hardware.
2. Enter the command show hardware chassis. In the resulting display, look for data from the system I/O card EEPROM. If the system can read the contents of the EEPROM, the system I/O card is likely to be healthy.
3. If the system I/O card is missing, install it.
4. If the system I/O card is present, reseat it. First, turn off system power. Use a screwdriver to unfasten the screws that secure the system I/O card cover, and pull the card off the connectors. Push it firmly back into place and tighten the screws.
5. If the problem persists, replace the system I/O card.
Fan Tray Problems
Use the following table to diagnose and troubleshoot any problems with the fan tray.
1. Make sure the fan module is fully inserted into the fan tray and screwed in place.
2. Replace the fan module.
Fan modules run but the system overheats.
1. Make sure that the air intake vents at the bottom front of the chassis and the exhaust vents on the top of the chassis are free of blockages.
2. Make sure that the ambient temperature and other environmental factors affecting the system are within the ranges specified in the "Environmental Requirements" section.
3. Make sure that all cards, blank faceplates, and covers are in place. The cooling system cannot operate effectively unless the chassis is fully enclosed.
4. Check the LED on the fan module. If the LED is not green, the fan module has failed. Replace the fan module.
5. Check the air filter at the bottom of the fan tray, and if necessary clean or replace it. See the "Replacing the Air Filter in the Fan Tray" section for instructions on cleaning or replacing the air filter.
6. Reduce the ambient temperature.
Fan module status LED is not green.
The fan module has failed or is not receiving power. If other components in the system are receiving power (look for green LEDs), replace the fan module.
Tables 5-2 through 5-8 describe alarms for the Cisco 6130. In each table, the text in the Alarm column is the text that appears in the description field of the alarm message. Alarm messages appear on the console screen as the alarms occur; to see a list of current alarms, enter show facility-alarm status.
Refer to the problem tables in this chapter for more detailed troubleshooting instructions.
Table 5-2 Chassis Alarm
Alarm
Keyword
Severity
Description
Chassis temperature too high
OVER_TEMP
Major
An overtemperature condition has been detected. (Temperature is measured on the NI-2 card.)
Table 5-3 Line Card Slot Alarms
Alarm
Keyword
Severity
Description
Provisioned slot is empty
MODULE_MISMATCH
Major
The line card in this slot does not match the type configured for this slot.
Line card not equal to provisioning
MODULE_MISSING
Minor
This slot is configured for a line card, but no line card is present.
Invalid line card for this slot
MODULE_INVALID
Minor
The line card detected in this slot cannot operate in this slot or is incompatible with the system configuration.
Flexi line card not provisioned
FLEX_MODULE_NOT_PROVISIONED
Info
You must use the slot command to configure a 4xflexi as either CAP or DMT.
Table 5-4 NI-2 Redundancy Alarms
Alarm
Keyword
Severity
Description
Active/Standby NI2 type mismatch
C6100_CONTROLLER_SLOT_MISMATCH_ALARM
Major
The NI-2 cards installed in slot 10 and slot 11 are different types; for example, one is an OC-3c/OC-3c NI-2 card and the other is an OC-3c/2xDS3 NI-2 card.
Standby NI2 missing
C6100_CONTROLLER_SLOT_MISSING_ALARM
Major
The secondary NI-2 card slot has been provisioned for a standby NI-2 card, but the card is not installed in the chassis.
Redundancy process failed
NI2_RF_PROCESS_FAILED
Major
An internal redundancy software process has failed and redundancy may not be available.
Peer APS mode mismatch
APS_MODE_MISMATCH_ALARM
Minor
The OC3 trunk is connected to a device not configured for nonrevertive, unidirectional APS.
Table 5-5 IOS Controller Alarms
Alarm
Keyword
Severity
Description
Loss of active clock sync
LOSS_OF_SYNC
Major
Loss of timing reference. The configured clock source is not available, so the system is using its internal clock.
BITS clock failure
BITS_FAILED
Major
BITS clock failure (LOS or AIS). The configured clock source is not available, so the system is using its internal clock.
The source of the OC-3c alarms in Table 5-6 is one of the following interfaces: ATM0/1 (the trunk) or ATM0/2 (the subtending interface).
Table 5-6 OC-3c/STM-1 Network Interface Alarms
Alarm
Keyword
Severity
Description
Loss of Cell Delineation
SONET_LOCD
Critical
Loss of cell delineation on a SONET line.
Path RDI Received
SONET_PRDI
Critical
Path Remote Defect Indication was received on a SONET line. This is equivalent to Path Far End Receive Failure (FERF).
Path AIS Received
SONET_PAIS
Critical
Path Alarm Indication Signal was received on a SONET line.
Loss of Pointer
SONET_LOP
Critical
Loss of pointer condition on a SONET line.
Line RDI
SONET_LRDI
Critical
Line Remote Defect Indication received on a SONET line. This is equivalent to line Far End Receive Failure (FERF).
Line AIS Received
SONET_LAIS
Critical
Line Alarm Indication Signal received on a SONET line.
Loss of Frame
SONET_LOF
Critical
LOF1 condition on a SONET line.
Loss of Signal
SONET_LOS
Critical
LOS2 detected on the SONET line.
Signal Label Mismatch
SONET_SIGNAL_
LABEL
Minor
Incorrect payload type signal label mismatch on a SONET line.
The source of the DS3 alarms in Table 5-7 is one of the following interfaces: ATM0/1 (the trunk), ATM0/2 (subtending interface), or ATM0/2 (subtending interface).
Table 5-7 DS3/E3 Network Interface Alarms
Alarm
Keyword
Severity
Description
Loss of Cell Delineation
DS3/E3_LOCD
Critical
The DS3 line is experiencing a loss of cell delineation.
RAI Received
DS3/E3_RAI
Critical
The DS3 line is receiving a remote alarm indication.
Yellow Alarm Received
DS3/E3_YELLOW
Critical
The DS3 line is receiving a yellow alarm, indicating that another device has detected a failure that might be in this device.
AIS Received
DS3/E3_AIS
Critical
The DS3 line is receiving an Alarm Indication Signal.
OOF Received
DS3/E3_OOF
Critical
The DS3 line has detected an Out of Frame condition.
LOS Detected
DS3/E3_LOS
Critical
The DS3 line has detected Loss of Signal at the framer.
PLCP LOF Detected
DS3_PLCP_LOF
Critical
The DS3 line has detected a Physical Layer Convergence Procedure Loss of Frame error.
Table 5-8 Fan Tray Alarms
Alarm
Keyword
Severity
Description
Not detected or missing
FAN_TRAY_REMOVED
Major
The fan tray has been disconnected from the chassis or is missing.
Multiple fan failures
FAN_FAIL_MULTI
Major
Two or more fan modules (out of three) in the fan tray have failed. When fan modules fail, the remaining fan modules run at full speed until the fault is corrected.
Single fan failure
FAN_FAIL_SING
Minor
A single fan module (one of three) in the fan module has failed. When fan modules fail, the remaining fan modules run at full speed until the fault is corrected.