Guest

Cisco 6400 Series Broadband Aggregators

Cisco 6400 Hardware Troubleshooting Guide

Cisco - Cisco 6400 Hardware Troubleshooting Guide

Document ID: 17822

Updated: Dec 28, 2005

   Print

Introduction

The Cisco 6400 is a broadband concentrator that features Cisco end-to-end Asynchronous Transfer Mode (ATM) services, Point-to-Point Protocol (PPP) termination, and tunneling. The Cisco 6400 combines Cisco ATM switching and routing capabilities in a modular, scalable, redundant chassis. This document does not discuss any architectural overview of the 6400 chassis or its parts. It discusses basic dispalyed LED indications, how to troubleshoot IOS reboot problems, and console hangs; but not hardware architecture details. If you want a more detailed discussion about 6400 (configuration, installation, release notes, command line), please refer to Cisco 6400 Carrier-Class Broadband Aggregator . Also For quick referencing for error messages and their meaning on 6400, please read Error Messages on the Cisco 6400 UAC Descriptions and Resolutions. For more detailed descriptions and technical discussions, see the useful links below under Related Information

Related Cisco Support Community Discussions section.

The Cisco 6400 includes the following hardware components:

  • Chassis

    • Module compartment

    • Blower compartment

    • Power Entry Module (PEM) compartment

    • Blackplane

  • Node Switch Processor (NSP)

  • Node Route Processor (NRP)

  • Node Line Card (NLC)

The following diagram shows a fully-loaded chassis with Blower compartment, PEMs, NSPs, NRPs and NLCs. The Module compartment in the chassis has ten slots—two central slots that are reserved for NSP cards and eight slots that can accommodate a mixture of full-height NRP cards and full-height and half-height NLCs.

6400_hw_1.gif

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the Cisco 6400 Universal Access Gateway.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Identifying Cisco 6400 Hardware Issues - Chassis

Blower Module

6400_hw_2.gif

The Blower module is located on the top of the chassis and is connected to the backplane. The module provides airflow throughout the system. The air intake is located at the bottom of the chassis, and there is an exhaust in the top rear of the chassis through the Blower module.

The Blower module supports hot-swapping and can be replaced without interrupting system operation. It is important to note that the system will shut down after two minutes if the Blower module is removed and not replaced.

Use the show environment command on the NSP to display information about fan operation and temperature levels.

Switch#show environment
Temperature normal: chassis inlet measured at 27C/80F
Temperature normal: chassis core measured at 33C/91F
Fan:                                            OK
AC Power Entry Module 0 status:                 OK
Switch#

When a single fan fails, replacing the Blower module is recommended. If several fans fail, replacing the Blower module is required. The following table shows blower module LEDs.

Table 1

LED Status Condition
Fans OK Steady green Fans are operational.
Single Fan Failure Steady yellow One fan has failed and alarms are triggered.
Multiple Fan Failure Steady yellow Two or more fans have failed and the system will shut down.

Power Entry Module

6400_hw_3.gif

The DC PEM shown above provides filtering and supplies DC power to the chassis electronics. Two PEM bays are in the chassis but only one PEM is required. An additional PEM can be used for redundancy. The PEMs are located on the left side of the chassis and are installed from the front. Input source power (-48 VDC from a building centralized power source) enters the chassis through screw terminals on the rear of the chassis and is supplied to the PEMs when they are plugged into the backplane.

6400_hw_4.gif

The AC PEM shown above provides power conversion directly from the facility VAC input power to the -48 VDC used internally in the Cisco 6400 chassis. AC power goes into the AC PEM through a power cord attached to the front faceplate.

Issue the show environment command on the NSP to display information about the PEM operation.

Switch#show environment
Temperature normal: chassis inlet measured at 27C/80F
Temperature normal: chassis core measured at 33C/91F
Fan:                                            OK
AC Power Entry Module 0 status:                 OK
Switch#

When a PEM fails and a redundant PEM is available in the chassis, the redundant PEM will start and continuously provide power to the Cisco 6400. Replacing the PEM is recommended. If there is not a redundant PEM, replacing the PEM is required.

Table 2 shows DC power entry module LEDs.

Table 2

LED Status Condition
Power Steady green Power is available.
Fault Steady yellow The PEM has failed or is turned off.
Miswire Steady yellow Cables are attached incorrectly and should be reversed.

Table 3 shows AC power entry module LEDs.

Table 3

LED Status Condition
Power Steady green Power is available.
Fault Steady yellow The PEM has failed or is turned off.

Identifying 6400 Hardware Issues - Node Switch Processor

6400_hw_5.gif

The NSP is the main system processor card in the Cisco 6400 and resides in chassis slot 0A or 0B. The NSP contains the ATM switch engine and processor, and most memory components. The NSP runs the system software that maintains and executes the management functions that control the system.

Table 4 shows NSP indicators.

Table 4

LED Status Condition
STATUS Steady yellow Blinking yellow Steady green Blinking green Off Cisco IOS® software is not running. System is booting. NSP is active (primary). NSP is standby (secondary). NSP has no power.
FAIL Yellow Off NSP has failed. NSP has not failed.

Useful Information

The following information is essential for determining the cause of most NSP problems.

  • Outputs of the show tech-support command.

  • Console logs.

  • Syslog information.

  • Crashinfo file.

  • Complete bootup sequence if the router experiences boot errors.

Capturing information from the console of the NSP is essential for troubleshooting NSP problems. Log the console output in a file for later analysis or for use by the Cisco Technical Assistance Center (TAC), if a TAC case is opened.

NSP Bootup Issues

If the STATUS or the FAIL LED on the NSP remains yellow (does not change to green), the first step in determining the cause of the problem is to connect to the console and see if you can get a router prompt. For information about terminal settings of a console connection, see the Troubleshooting section of the Cisco 6400 UAC Hardware Installation and Maintenance Guide.

If the NSP is locked in ROM monitor (ROMMON) mode with a rommon prompt, set the configuration register to 0x2102 and reload the NSP.

rommon 1>confreg 0x2102
rommon 2>reset

If the NSP is still in ROMMON mode after it was reloaded, use the following recovery procedure.

  1. Issue the dev command to determine which devices are available on the NSP.

    rommon1>dev 
    Devices in device table: 
             id  name 
     bootflash:  boot flash 
         slot0:  PCMCIA slot 0
         slot1:  PCMCIA slot 1
         disk0:  PCMCIA slot 0
         disk1:  PCMCIA slot 1
        eprom:   eprom
     rommon 2>
  2. Issue the dir <device_id> command to search for a valid Cisco IOS software image.

    rommon 2>dir disk0:
    Directory of disk0:
    
    3     0         drw-     images
    4     0         drw-     slot1
    5     0         drw-     slot2
    6     0         drw-     slot3
    7     0         drw-     slot4
    8     0         drw-     slot5
    9     0         drw-     slot6
    10    0         drw-     slot7
    11    0         drw-     slot8
    8283  5025140   -rw-     c6400s-wp-mz_121-5_DB.bin
    13298 5289460   -rw-     c6400s-wp-mz.122-2.B1.bin
    4287  0         drw-     non-mirror
    rommon 3>
  3. Boot from the image you selected.

    rommon3>boot disk0:c6400s-wp-mz.122-2.B1.bin 
    Self decompressing the image : ################################################ 
    ###############################################################################
    ###############################################################################
    ############################################################################### 
    ############################################################################### 
    ###############################################################################
    ############################################################################### 
    [OK]

NSP Reboot/Reload

Spontaneous NSP reboots/reloads or failures can be caused by either software or hardware. Only failures caused by hardware are discussed in this document. The crash info and console logs are required to understand whether the problem is hardware or software related.

Parity errors and bus errors are examples of failures that can be caused by hardware:

  • Parity Errors – These can be caused by hardware failures or single event upsets. Please review Processor Memory Parity Errors (PMPEs) to help determine whether the problem is caused by a single event upset or a hardware failure.

  • Bus Errors – These can be caused by software or hardware. Please review Troubleshooting Bus Error Crashes to help determine whether the problem is caused by hardware or software.

NSP Stops Responding

The console will not respond if the NSP stops responding. This can be caused by a failure in hardware, memory, software, or by a high CPU utilization. Please see Troubleshooting Router Hangs for more information.

Identifying 6400 Hardware Issues - Node Route Processor

The NRP sends and receives traffic between one or more NSP ATM switch ports. The Cisco 6400 can contain multiple NRP modules configured to operate independently or as redundant pairs (1+1). You can insert NRPs into slots 1 through 8 in the Cisco 6400 chassis.

The Cisco 6400 supports two NRPs, designated as NRP-2SV and NRP-1. These can be used together in any combination in a single Cisco 6400 chassis.

NRP-2SV Module

6400_hw_6.gif

The NRP-2SV provides a Gigabit Ethernet interface and sufficient processing capability for handling an OC-12 rate of user traffic.

Table 5 shows NRP-2SV LED indicators.

Table 5

LED Status Condition
STATUS Steady yellow Blinking yellow Steady green Steady yellow Off NRP-2SV is active. System is booting. Cisco IOS software is not running. NRP-2SV has no power.
FAIL Steady yellow Off NRP-2SV has failed. Normal operation.

Methods Available for Gaining Access to the NRP-2

There are two methods available for accessing the NRP-2:

You can also configure the NRP-2 with the Cisco 6400 Service Connection Manager, Release 2.2(1) and later. For more information, refer to Cisco 6400 SCM .

Accessing the NRP-2 Console Through the NSP

The NSP is equipped with an internal communication server for accessing the NRP-2 console line. To access the NRP-2 console line, use Telnet to connect to the NSP as a communication server, using the port numbers shown in Table 6 below to select the NRP-2.

Table 6

NSP Communication Server Port Numbers Associated Cisco 6400 Chassis Slot
2001, 4001, 6001 Slot 1
2002, 4002, 6002 Slot 2
2003, 4003, 6003 Slot 3
2004, 4004, 6004 Slot 4
2005, 4005, 6005 Slot 5
2006, 4006, 6006 Slot 6
2007, 4007, 6007 Slot 7
2008, 4008, 6008 Slot 8

To exit the NRP-2 console line without closing the console connection, use the escape sequence Ctrl-Shift-6. To close the NRP-2 console line connection, use the exit command.

The example procedure below illustrates this further.

  1. Suppose the NSP in your Cisco 6400 system has the management IP address 10.1.5.4. To access the console line of the NRP-2 in slot 6 of the same Cisco 6400 chassis, use the telnet command from another router:

    device#telnet 10.1.5.4 2006 
    Trying 10.1.5.4, 2006 ... Open 
    NRP-2# 
  2. To return to the device prompt without closing the NRP-2 console line connection, enter the escape sequence Ctrl-Shift-6 at the NRP-2 prompt. Notice that the full escape sequence does not appear as you enter it in the command-line interface (CLI):

    NRP-2# Ctrl^ x 
    device# 
    
  3. To return to the connected NRP-2 console line, hit Enter at the device prompt.

    device# 
    [Resuming connection 1 to 10.1.5.4 ... ] 
    NRP-2# 
  4. To close the NRP-2 console line connection, use the escape sequence to return to the device prompt, and then use the exit command.

    NRP-2#Ctrl^ 
    device#exit 
    (You have open connections) [confirm] 
    Closing:10.1.5.4 ! 
    device con0 is now available 
    Press RETURN to get started. 
    device> 

Using Telnet to Connect to the NRP-2 from the NSP

The NSP is equipped with command aliases for using Telnet to connect to an NRP-2 in the same Cisco 6400 chassis. To use Telnet to connect to the NRP-2, use the following NSP command alias in EXEC mode.

Command Purpose
Switch#nrpsslot 
Uses Telnet to connect to the NRP-2 in the specified slot.

Note: Set the enable password for the NSP before you use Telnet to connect to the NRP-2.

To exit the NRP-2 VTY line without closing the Telnet session, use the escape sequence Ctrl-Shift-6. To close the NRP-2 Telnet session, use the exit command.

The example procedure below illustrates this further.

  1. Suppose you want to use Telnet to connect to the NRP-2 from a device outside your Cisco 6400 system, and the NSP in the Cisco 6400 has the management IP address 10.1.5.4. To use Telnet to connect to the NRP-2, first connect to the NSP, and then use the nrps command alias to connect to the NRP-2.

    device#telnet 10.1.5.4 
    Trying 10.1.5.4 ... Open 
    User Access Verification 
    Password: 
    NSP> 
    NSP>nrps6 
    Trying 10.6.0.2 ... Open 
    NRP-2> 
  2. To close the Telnet session to the NRP-2 and return to the NSP prompt, use the exit command.

    NRP-2>exit 
    [Connection to 10.6.0.2 closed by foreign host] 
    NSP>

NRP-1 Module

6400_hw_7.gif

The NRP-1 incorporates a 100 Mbps Fast Ethernet interface for connecting into an IP network and has processing capability for an OC-3 rate of user traffic.

Table 7 shows NRP-1 LED indicators.

Table 7

LED Status Condition
STATUS Steady green Blinking green Steady yellow Blinking yellow Off NRP is active (primary). NRP is standby (secondary). Cisco IOS software is not running. System is booting. NRP has no power.
FAIL Steady yellow Off NRP has failed. Normal operation.

Useful Information

The following information is essential for determining the cause of most NRP problems.

  • Outputs of the show tech-support command.

  • Console logs.

  • Syslog information.

  • Crashinfo file.

  • Complete bootup sequence if the router experiences boot errors.

Capturing information from the console of the NRP is essential for troubleshooting NRP problems. Log the console output in a file for later analysis or for use by the Cisco TAC, if a TAC case is opened.

NRP Bootup Issues

If the STATUS or the FAIL LED on the NSP remains yellow (does not change to green), the first step in determining the cause of the problem is to connect to the console and see if you can get a router prompt. For information about terminal settings of a console connection, see the Troubleshooting section of the Cisco 6400 UAC Hardware Installation and Maintenance Guide.

If the NRP is locked in ROMMON mode with a rommon prompt, set the configuration register to 0x2102 and reload the NRP.

rommon 1>confreg 0x2102
rommon 2>reset

If the NRP is still in ROMMON mode after it has been reloaded, use the following recovery procedure.

  1. Issue the dev command to determine which devices are available on the NRP.

    rommon 1 >dev
    Devices in device table:
            id  name
        flash:  flash 
    bootflash:  boot flash
        eprom:  eprom
    rommon 2>
  2. Issue the dir <device_id> command to look for a valid Cisco IOS software image.

    rommon 2>dir flash:
            File size           Checksum      File name
    5545532 bytes (0x549e3c)    0xf1cbd97e    c6400r-g4p5-mz.121-5.DC1
    rommon 3>
  3. Boot from the image you selected.

    rommon 3>boot flash:c6400r-g4p5-mz.121-5.DC1 
    Self decompressing the image : ################################################ 
    ###############################################################################
    ###############################################################################
    ############################################################################### 
    ############################################################################### 
    ###############################################################################
    ############################################################################### 
    [OK]

NRP Reboot/Reload

Spontaneous NRP reboots/reloads or failures can be caused by either software or hardware. Only failures caused by hardware are discussed in this document. The crashinfo and console logs are required to understand whether the problem is hardware or software related.

Parity errors and bus errors are examples of failures that can be caused by hardware.

  • Parity Errors – These can be caused by hardware failures or single event upsets. Please review Processor Memory Parity Errors (PMPEs) to help determine whether the problem is caused by a single event upset or a hardware failure.

  • Bus Errors – These can be caused by software or hardware. Please review Troubleshooting Bus Error Crashes to help determine whether the problem is caused by hardware or software.

NRP Stops Responding

The console will not respond if the NRP stops responding. This can be caused by a failure in hardware, memory, software, or by a high CPU utilization. Please see Troubleshooting Router Hangs for more information.

Identifying 6400 Hardware Issues - Node Line Cards

6400_hw_8.gif

The OC-3/STM-1 SM half-height NLC has two 155 Mbps fiber-optic ports for a single-mode intermediate reach connection of uplink and downlink interfaces.

6400_hw_9.gif

The OC-3/STM-1 MM half-height NLC has two 155 Mbps fiber-optic ports for a multimode connection on the front of each NLC.

6400_hw_10.gif

The DS3 half-height NLC has two 45 Mbps bidirectional ports for a connection to network services using coaxial cable.

Table 8 shows OC-3 and DS3 NLC LED indicators.

Table 8

LED Status Condition
FAIL Steady yellow Off NLC has failed. NLC is operational.
PORT 0 (top connector)
TX (transmit) Green Off Steady yellow Flashing yellow Transmit activity. No traffic. Far-end alarm. Local loopback.
RX (receive) Green Off Steady yellow Receive activity. No traffic. Loss of Signal.
STATUS Green Blinking green Off Active (primary). Standby mode (secondary). No power.
PORT 1 (bottom connector)
TX (transmit) Green Off Steady yellow Flashing yellow Transmit activity. No traffic. Far-end alarm. Local loopback.
RX (receive) Green Off Steady yellow Receive activity. No traffic. Loss of Signal.
STATUS Steady green Blinking green Off Active. Standby mode. No power.

6400_hw_11.gif

The OC-12/STM-4 full height NLC has one 622 Mbps fiber-optic port for the connection of uplink and downlink interfaces; single mode, intermediate reach.

Table 9 shows OC-12 NLC LED indicators.

Table 9

LED Status Condition
FAIL Steady yellow Off OC-12/STM-4 NLC has failed. OC-12/STM-4 NLC is operational.
TX (transmit) Green Off Steady yellow Flashing yellow Transmit activity. No traffic. Far-end alarm. Local loopback.
RX (receive) Green Off Steady yellow Receive activity. No traffic. Loss of Signal.
STATUS Green Blinking green Off Active (primary). Standby mode (secondary). No power.

The Cisco 6400 can contain multiple NLC modules, configured to operate independently or as redundant pairs. NLCs can be inserted into slots 1 through 8 (subslots 0 and 1) in the Cisco 6400 chassis.

To show the NLC modules in the Cisco 6400, issue the show hardware command on the NSP.

Switch#show hardware
6400 named Switch, Date: 12:41:06 UTC Wed Dec 19 2001
Feature Card's FPGA Download Version: 0
Slot  Ctrlr-Type    Part No.   Rev  Ser No  Mfg Date   RMA No.  Hw Vrs Tst EEP
----- ------------  ---------- --  -------- ---------  -------- ------ --- ---
1/0  NRP            73-3082-02 ??  09694065 Jan 00 00  00-00-00   2.20  FF   0
2/0  NRP            73-2342-03 F0  11667429 Feb 26 99  00-00-00   1.0    0   2
3/0  NRP            73-2342-03 F0  11667428 Feb 27 99  00-00-00   1.0    0   2
4/0  NRP2           73-4902-02 C0  25389496 Feb 28 01  00-00-00   1.0    0   2
5/0 155SM NLC       73-2892-03 01  09156343 Sep 09 98  00-00-00   0.0    0  FF
6/0 155SM NLC       73-2892-03 01  09156410 Sep 09 98  00-00-00   3.1    0  FF
7/0 155MM NLC       73-5824-02 A0  25388608 Feb 24 01  00-00-00   1.0    0   2
8/0 622SM NLC       73-3868-02 A0  14342212 Jul 15 99  00-00-00   1.0    0   2
5/1 DS3 NLC         73-3696-02 02  11555742 Feb 04 99  00-00-00   1.0    0   2
6/1 DS3 NLC         73-3696-02 A0  16088970 Dec 23 99  00-00-00   1.0    0   2
7/1 155SM NLC       73-2892-03 01  11671065 Feb 09 99  00-00-00   1.0    0   2
0A/PC NSP-PC        73-2996-06 A0  15792098 Dec 30 99  00-00-00   1.0    0   2
0A/FC FC-PFQ        73-2281-04 B0  15792035 Dec 30 99  00-00-00   4.1    0   2
0A/SC NSP-SC        73-2997-06 A0  15791346 Dec 30 99  00-00-00   1.0    0   2
0B/PC NSP-PC        73-2996-06 A0  15792204 Dec 28 99  00-00-00   1.0    0   2
0B/FC FC-PFQ        73-2281-04 B0  15791847 Dec 28 99  00-00-00   4.1    0   2
0B/SC NSP-SC        73-2997-06 A0  15791341 Dec 29 99  00-00-00   1.0    0   2

Primary NSP:Slot 0A

DS1201 Backplane EEPROM:

 Model   Ver.  Serial  MAC-Address  MAC-Size  RMA  RMA-Number   MFG-Date
------- ----  -------- ------------ --------  ---  ----------  -----------
C6400    2    10036146 00107BB9BD00   128      0   0           Sep 16 1998

Switch#

To determine if a NLC module is operational, issue the show hw-module subslot <slot>/<subslot> command on the NSP.

Switch#show hw-module subslot 8/0

Slot  Ctrlr-Type            Comment
----  ----------  ---------------------------------
8/0    622SM NLC   State = Operational

Switch#

Related Information

Updated: Dec 28, 2005
Document ID: 17822