Guest

Cisco Catalyst 8500 Series Multiservice Switch Routers

Troubleshooting ASP Red Status Light and Power-on Diagnostic Problems on the LightStream 1010 and Catalyst 8510-MSR

Document ID: 10464



Contents

Introduction
Prerequisites
      Requirements
      Conventions
Typical Output on the ASP, PCQ (Also Known as FC1)
Typical Output on the ASP, PFQ (Also Known as FC3)
Field Definitions
SNAKE Test Failures on FC-PFQ
Recommendations
Conclusion
NetPro Discussion Forums - Featured Conversations
Related Information

Introduction

The purpose of this document is to explain the various reasons that can cause an ASP Status LED to turn red. It also describes the power-on diagnostics (PoD), as well the various show diag power-on command output fields.

Observe the LEDs of all ports at bootup to see that online tests are conducted one at a time. The ATM Switch Processor (ASP) conducts various diagnostic tests. If an error is detected within the tests, the Status LED under ASP turns red. The Status LED usually changes color after bootup.

If the ASP Status LED turned red, you can use the show diag power-on command to determine the precise reason. The output slightly varies for each feature card type.

Note: Individual line cards on your ATM switch also use LEDs to indicate status information. A red LED on line cards is not discussed in this document. Refer to the ATM and Layer 3 Module Installation Guides to troubleshoot a red status LED on a specific line card.

The test is not performed systematically when the LS1010 boots. It is done when the box is powered up, but not when it is re-loaded afterwards.

Prerequisites

Requirements

There are no specific requirements for this document.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Typical Output on the ASP, PCQ (Also Known as FC1)

Check the feature card (FC) type with the show hardware command as shown here:

Gambrinus#show hardware
 
LS1010 named Gambrinus, Date: 01:01:48 UTC Wed Apr 26 2000
Feature Card's FPGA Download Version: 10
 
Slot Ctrlr-Type      Part No.  Rev  Ser No  Mfg Date   RMA No. Hw Vrs  Tst EEP
---- --------------  ---------- -- -------- --------- -------- ------- --- ---
4/0  155MM PAM       73-1496-03 A0 10989369 Dec 08 98 00-00-00   3.2     0   2
4/1  155MM PAM       73-1496-03 A0 10989016 Dec 08 98 00-00-00   3.2     0   2
2/0  ATM Swi/Proc    73-1402-04 A1 03654393 Oct 25 96 00-00-00   3.2     0   2
2/1  FeatureCard1    73-1405-03 D0 03654637 Oct 25 96 00-00-00   3.2     0   2
 
DS1201 Backplane EEPROM:
Model  Ver.  Serial  MAC-Address  MAC-Size  RMA  RMA-Number   MFG-Date
------ ---- -------- ------------ --------  ---  ----------  -----------
LS1010  2   68003437 006047508400   256      0        0      Nov 11 1996

Check the result of the PoD with the show diag power-on command:

Gambrinus#show diag power-on
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Diags  Date: 00/03/04   Time: 09:34:02   By: V 3.44
 
   BOOTFLASH:  .   PCMCIA-Slot0: N   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .
 
   PS1:        .   PS2:          N   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .
 
   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .
 
   Feature-Card Access: .
   ICC: .   OCC: .   OQP: .   OQE: .   CC:  .   RT:  .
   TM0: .   TM1: .   TMC: .   IT:  .   LT:  .   RR:  .   ABR: .

Access/Interrupt/Loopback/CPU-MCast/Port-MCast/FC-MCast/FC-TMCC Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 4/0 (155MM)         .......   .......   .......   .......   
PAM 4/1 (155MM)         .......   .......   .......   .......   
 
   Ethernet-port Access:   .         Ethernet-port CAM-Access: .
   Ethernet-port Loopback: .         Ethernet-port Loadgen:    .
 
Power-on Diagnostics Passed.

Typical Output on the ASP, PFQ (Also Known as FC3)

Like the PCQ, you can check the FC type with the show hardware command:

Droopie#show hardware
Slot Ctrlr-Type      Part No.  Rev  Ser No  Mfg Date   RMA No. Hw Vrs  Tst EEP
---- --------------  ---------- -- -------- --------- -------- ------- --- ---
0/0  155MM PAM       73-1496-03 A2 07197796 Nov 28 97 00-00-00   3.1     0   2
0/1  1CT3 FR-PAM     73-2972-03 A0 12344254 May 17 99 00-00-00   3.0     0   2
4/0  622MM PAM       73-1864-02 A0 12347208 Apr 22 99 00-00-00   1.3     0   2
4/1  155MM PAM       73-1496-03 A6 03203786 Aug 01 95 00-00-00   3.1     0   2
2/0  ATM Swi/Proc    73-1402-07 B0 09002771 Oct 05 98 00-00-00   5.1     0   2
2/1  FC-PFQ          73-2281-04 A0 09002498 Oct 05 98 00-00-00   4.1     0   2
 
DS1201 Backplane EEPROM:
Model Ver.  Serial  MAC-Address  MAC-Size  RMA  RMA-Number   MFG-Date
----- ---- -------- ------------ --------  ---  ----------  -----------
LS1010  2   68002747 00604799FD00   256      0        0      Nov 23 1996

Display the PFQ diagnostics with the show diag power command:

Droopie#show diag power
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Diags  Date: 00/03/05   Time: 02:31:24   By: V 4.51
 
   BOOTFLASH:  .   PCMCIA-Slot0: N   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .
 
   PS1:        .   PS2:          N   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .
 
   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .
 
   FC-PFQ
    Access: .
     RST: .    REG: .    IVC: .    IFILL: .    OVC: .    OFILL: .
 
    TEST:
     CELL: .   SNAKE: .   RATE: .   MCAST: .   SCHED: .
     TGRP: .   UPC  : .   ABR : .   RSTQ : .
 
Access/Interrupt/Loopback/CPU-MCast/Port-MCast/FC-MCast/FC-TMCC Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 0/0 (155MM)         .....NN   .....NN   .....NN   .....NN   
PAM 0/1 (FR1CT3)        .....NN   .....NN   .....NN   .....NN   
PAM 4/0 (622MM)         .....NN      N         N         N
PAM 4/1 (155MM)         .....NN   .....NN   .....NN   .....NN   
 
 
FRPAM#          ING-SSRAM ING-SDRAM EGR-SSRAM EGR-SDRAM LOOPBACK
------------------------------------------------------------------
PAM 0/1 (FR1CT3)  .        .         .         .         .
   Ethernet-port Access:    .        Ethernet-port CAM-Access: .
   Ethernet-port Loopback:  .        Ethernet-port Loadgen:    .
   GEPAM Microcode:         .        GEPAM Access:             .
   GEPAM CAM Access:        .
 
Power-on Diagnostics Passed.

You can also see that Frame Relay/ATM cards are displayed in different ways, which can also be seen with the PCQ.

Whatever the output type, the main indication is that the PoD passed. If it had failed, the ASP Status LED would be red. If recoverable errors are seen in the tests, the tests perform normally, but a warning displays as in this example:

Miles#show diag power
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Diags  Date: 00/04/11   Time: 02:14:57   By: V 3.44
 
   BOOTFLASH:  .   PCMCIA-Slot0: N   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .
 
   PS1:        .   PS2:          N   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .
 
   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .
 
   FC-PFQ
    Access: .
     RST: .    REG: .    IVC: .    IFILL: .    OVC: .    OFILL: .
 
    TEST:
     CELL: .   SNAKE: .   RATE: .   MCAST: .   SCHED: .
     TGRP: .   UPC  : .   ABR : .   RSTQ : .
 
Access/Interrupt/Loopback/CPU-MCast/Port-MCast/FC-MCast/FC-TMCC Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 0/0 (25M)           .....NN   .....NN   .....NN   .....NN   
    Port 4  to 7 :      .....NN   .....NN   .....NN   .....NN   
    Port 8  to 11:      .....NN   .....NN   .....NN   .....NN   
PAM 0/1 (155MM)         .....NN   .....NN   .....NN   .....NN   
PAM 4/0 (155MM)         .....NN   .....NN   .....NN   .....NN   
PAM 4/1 (E3)            .....NN   .....NN      N         N
 
   Ethernet-port Access:   .         Ethernet-port CAM-Access: .
   Ethernet-port Loopback: .         Ethernet-port Loadgen:    .
 
M4:Non-Volatile Memory Read/Write Test []
 *** MEMDIAG_NVRAM_MAGIC_PATTERN_DATA_ERROR *** [Addr:BE001008, exp:0000ABCD, act:00000000]
power-on Diagnostics Passed.

Field Definitions

These tables explain only the fields related to the ports or memory. You can assume that any other test failures involve the replacement of the ASP. This includes diagnostics under the MMC, FC-PFQ, or feature card.

Chassis-Specific Fields

Field

Definition

BOOTFLASH

This performs validation on the files present in the CPU-board-resident flash file system. This checks the presence of file system and checksum validation for the bootflash resident files. If it fails, the bootflash is bad. Reformat it with the LS1010 and recopy the files with the copy tftp command. Refer to Recovering from a Corrupted or Lost Cisco IOS Image on LS1010 and Catalyst 8500 Switches.

PCMCIA-Slot[0 or 1]

This is the same as the bootflash test.

[CPU or Fcard]IDPROM

This performs validation of the CPU/Feature Card IDPROMS. If it fails, you will need to Return Material Authorization (RMA) the ASP.

NVRAM-Config

This performs validation of the NVRAM. If it fails, try to configure the LS1010 with the IOS config mode commands. If it still fails, replace the ASP.

SRAM

This performs the Read/Write test on Static memory, which is 128K in size. If it fails, replace the ASP.

DRAM

This performs the Read/Write test on Dynamic memory. Replace DRAM; if it still fails, replace ASP.

PS [1 or 2]

This is the power supply.

FAN

This is self-explanatory.

Temperature

This is self-explanatory.

Bkp-IDPROM

This performs validation on the backplane IDPROM. Replace the chassis.

Card-Specific Fields (FC-specific Test Skipped)

Field

Definition

Access

This test makes sure that the PHY-layer HW resident on the various port adapter module (PAM) cards in the system are accessible. If it fails, replace the PAM.

Interrupt

This test makes sure that the PHY-layer HW resident on the various PAM cards in the system can interrupt the CPU under alarm condition. If it fails, replace the PAM.

Loopback

This test sources the unicast cells to the port and validates the received cells in loopback mode. If it fails, the connectivity is broken in the cell path. Try to replace the PAM first.

CPU-MCast

This is the same as the loopback test for multicast cells. In other words, the CPU acts as the root of the point-to-multipoint connection.

Port-MCast

The CPU sends a unicast cell to the first port in the list, which in turn multicasts it to the rest of the port and validates the result in the loopback mode.

Ethernet-Specific Counters

Field

Definition

Ethernet-port Access

This test ensures that the Ethernet port HW resident on the Ethernet controller present on the ASP is accessible. If it fails, the Ethernet controller seems to be broken, and the ASP needs to be replaced.

Ethernet-port CAM-Access

This performs Read/Write on the built-in Content Addressable Memory (CAM) on the Ethernet controller. If it fails, the Ethernet controller seems to be broken, and the ASP needs to be replaced.

Ethernet-port Loopback

This performs the loopback test on the Ethernet port. If it fails, the Ethernet controller seems to be broken, and the ASP needs to be replaced.

Ethernet-port Loadgen

This artificially loads the Ethernet port. If it fails, the Ethernet controller seems to be broken, and the ASP needs to be replaced..

SNAKE Test Failures on FC-PFQ

The SNAKE test sends a cell through all interfaces on the switch. This test ensures that all interfaces and associated fabric interfaces are functional. Cisco bug ID CSCdk54678 resolves a problem that causes the SNAKE test to fail on an LightStream 1010, runs Cisco IOS® Software Release 11.3WA4, and uses an FC-PFQ.

Recommendations

If you see a failed PoD caused by a PAM (from the show diag power-on output), we highly recommend that you perform these steps in order until the problem is resolved:

  1. Upgrade the Cisco IOS software to a more recent release (Version 12.0 or higher) since some bugs have been resolved.

  2. Turn the LS1010 off, re-seat the PAM, and turn the LS1010 on again because an improperly-inserted PAM can definitely cause the tests to fail.

  3. If all else fails, contact Cisco TAC.

Conclusion

This is a typical example of a Pod that fails. It was solved by an RMA of the PAM. Similar issues have been solved with the PAM reseated.

show diag power-on
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Date: 98/09/19   Time: 05:15:33

   BOOTFLASH:  .   PCMCIA-Slot0: N   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .

   PS1:        .   PS2:          .   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .

   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .

   Feature-Card Access: .
   ICC: .   OCC: .   OQP: .   OQE: .   CC:  .   RT:  .
   TM0: .   TM1: .   TMC: .   IT:  .   LT:  .   RR:  .   ABR: .

Access/Interrupt/Loopback/CPU-MCast/Port-MCast/FC-MCast/FC-TMCC Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 10/0(155MM)         .......   .......   ..F....   .......   
PAM 10/1(155MM)         .......   .......   .......   .......   
PAM 11/0(155MM)         .......   .......   .......   .......   
PAM 11/1(155MM)         .......   .......   .......   .......   
PAM 12/1(155MM)         .......   .......   .......   .......   

   Ethernet-port Access:   .         Ethernet-port CAM-Access: .
   Ethernet-port Loopback: .         Ethernet-port Loadgen:    .

A4:ATM Layer Loopback Test [PM2P2,VP,Q,PHY,ASP_OSC]
 *** ATMDIAG_PIF_STAT_HEC_ERROR *** [Addr:08000001, exp:00000000, act:00000002]

Power-on Diagnostics Failed.

NetPro Discussion Forums - Featured Conversations

Networking Professionals Connection is a forum for networking professionals to share questions, suggestions, and information about networking solutions, products, and technologies. The featured links are some of the most recent conversations available in this technology.
NetPro Discussion Forums - Featured Conversations for Router and IOS Architecture
Network Infrastructure: LAN Routing and Switching
Network Infrastructure: WAN Routing and Switching

Related Information



Updated: Jun 05, 2005Document ID: 10464