Guest

Cisco Catalyst 8500 Series Multiservice Switch Routers

Hardware Troubleshooting for Catalyst 8540/8510 MSRs and LightStream 1010 ATM Switch: ATM Switch Processor (ASP) Hangs and Crashes

Document ID: 21454

Updated: Jun 05, 2005

   Print



ATM Switch Processor Power-on Diagnostics


Contents


<<<Previous Section      Next Section>>>


ATM Switch Processor Shows Red Status LED

This section explains the various reasons that can cause an ATM Switch Processor (ASP) Status LED to turn red. It also describes the power-on diagnostics (PoD), as well the various show diag power-on command output fields.

Note: Individual line cards on your ATM switch also use LEDs to indicate status information. A red LED on line cards is not discussed in this document. Please refer to the ATM and Layer 3 Module Installation Guides to troubleshoot a red status LED on a specific line card.

By observing the LEDs of all ports during bootup, you can see that online tests are being conducted one at a time. The ATM Switch Processor (ASP) conducts various diagnostic tests. If an error is detected during the tests, the Status LED turns red. The Status LED usually changes color after bootup.

If the ASP Status LED turned red, then you can use the show diag power-on command to determine the precise reason. The output slightly varies depending on the feature card type.

The test is not performed systematically when the LightStream 1010 is booting. It is done when the box is powered up, but not when it is reloaded afterwards.

Typical Output on the ASP, PCQ (also known as FC1)

You can check the feature card (FC) type by using the show hardware command as shown below:
LS1010# show hardware

LS1010 named LS1010, Date: 19:00:42 UTC Tue Mar 5 2002
Feature Card's FPGA Download Version: 10

Slot Ctrlr-Type      Part No.  Rev  Ser No  Mfg Date   RMA No. Hw Vrs  Tst EEP
---- ------------    ---------- -- -------- --------- -------- ------- --- ---
1/0  155MM PAM       73-1496-03 06 02180444 Jan 17 96 00-00-00   3.0     0   2
1/1  155MM PAM       73-1496-03 06 02202228 Jan 11 96 00-00-00   3.0     0   2
3/0  CE-T1 PAM       73-2176-02 A0 03669320 Feb 15 97 00-00-00   1.0     0   2
3/1  QUAD DS3 PAM    73-2197-02 A0 03816513 Jan 30 97 00-00-00   2.0     0   2
4/0  4CE1 FR-PAM     73-3040-02 A0 11667127 Feb 20 99 00-00-00   2.0     0   2
4/1  T1 PAM          73-2133-02 00 03669217 Feb 12 97 00-00-00   1.0     0   2
2/0  ATM Swi/Proc    73-1402-06 D0 07202996 Dec 20 97 00-00-00   4.1     0   2
2/1  FeatureCard1    73-1405-05 B0 07202788 Dec 20 97 00-00-00   3.2     0   2

DS1201 Backplane EEPROM:
Model Ver.  Serial  MAC-Address  MAC-Size  RMA  RMA-Number   MFG-Date
----- ---- -------- ------------ --------  ---  ----------  -----------
LS1010  2   68018639 003080CE3A00   256      0        0      Sep 16 1999

LS1010#
You can also check the result of the PoD by using the show diag power-on command:
LS1010# show diag power-on 
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Diags  Date: 01/11/20   Time: 10:01:07   By: V 4.54

   BOOTFLASH:  .   PCMCIA-Slot0: .   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .

   PS1:        .   PS2:          N   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .

   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .

   Feature-Card Access: .
   ICC: .   OCC: .   OQP: .   OQE: .   CC:  .   RT:  .
   TM0: .   TM1: .   TMC: .   IT:  .   LT:  .   RR:  .   ABR: .

Access/Interrupt/Loopback Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 0/0 (IMA8T1)        ...   ...   ...   ...
    Port 4  to 7 :      ...   ...   ...   ...
PAM 1/0 (155MM)         ...   ...   ...   ...
PAM 1/1 (155MM)         ...   ...   ...   ...
PAM 3/0 (T1CE)          ...   ...   ...   ...
PAM 3/1 (DS3Q)          ...   ...   ...   ...
PAM 4/0 (FR4CE1)       ...   ...   ...   ...
PAM 4/1 (T1)            ...   ...   ...   ...


FRPAM#          ING-SSRAM ING-SDRAM EGR-SSRAM EGR-SDRAM LOOPBACK
------------------------------------------------------------------
PAM 4/0 (FR4CE1)  .        .         .         .         .
   Ethernet-port Access:    .        Ethernet-port CAM-Access: .
   Ethernet-port Loopback:  .        Ethernet-port Loadgen:    .
   GEPAM Microcode:         .        GEPAM Access:            .
   GEPAM CAM Access:        .

Power-on Diagnostics Passed.

LS1010#

Typical Output on the ASP, PFQ (also known as FC3)

Like the PCQ, you can check the FC type by using the show hardware command:
NewLs1010# show hardware

LS1010 named NewLs1010, Date: 16:43:51 UTC Tue Mar 5 2002
Feature Card's FPGA Download Version: 0

Slot Ctrlr-Type      Part No.  Rev  Ser No  Mfg Date   RMA No. Hw Vrs  Tst EEP
---- ------------    ---------- -- -------- --------- -------- ------- --- ---
0/*  TS CAM          73-5659-01 01 76543210 Oct 25 00 00-00-00   1.0     0   2
0/0  155MM PAM       73-1496-03 00 02180455 Jan 17 96 00-00-00   3.0     0   2
0/1  155MM PAM       73-1496-03 06 02180424 Jan 16 96 00-00-00   3.0     0   2
1/0  4CE1 FR-PAM     73-3040-02 A0 11667078 Mar 03 99 00-00-00   2.0     0   2
1/1  155UTP PAM      73-1572-03 A0 09005188 Sep 28 98 00-00-00   3.2     0   2
3/0  DS3 PAM         73-2345-02 B0 07192680 Nov 06 97 00-00-00   1.5     0   2
3/1  CE-E1120 PAM    73-2177-03 A0 08782763 Apr 06 98 00-00-00   1.1     0   2
4/0  ARM CONTROLLER  73-4774-01 01 16104033 Nov 10 99 00-00-00   4.1     0   2
2/0  ATM Swi/Proc    68-0732-01 C0 17807077 Mar 23 00 00-00-00   6.0     0   2
2/1  FC-PFQ          73-2281-04 B0 17806810 Mar 23 00 00-00-00   4.2     0   2

DS1201 Backplane EEPROM:
Model Ver.  Serial  MAC-Address  MAC-Size  RMA  RMA-Number   MFG-Date
----- ---- -------- ------------ --------  ---  ----------  -----------
LS1010  2   68003772 00E0F75D0400   256      0        0      Dec 17 1996

NewLs1010#
You can display the PFQ diagnostics by using the show diag power command:
NewLs1010# show diag power-on 
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Diags  Date: 01/11/15   Time: 08:37:13   By: V 4.54

   BOOTFLASH:  .   PCMCIA-Slot0: .   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .

   PS1:        .   PS2:          N   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .

   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .

   FC-PFQ
    Access: .
     RST: .    REG: .    IVC: .    IFILL: .    OVC: .    OFILL: .

    TEST:
     CELL: .
Access/Interrupt/Loopback Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 0/0 (155MM)         ...   ...   ...   ...
PAM 0/1 (155MM)         ...   ...   ...   ...
PAM 1/0 (FR4CE1)       ...   ...   ...   ...
PAM 1/1 (155UTP)        ...   ...   ...   ...
PAM 3/0 (DS3)           ...   ...      N         N
PAM 3/1 (E1CEUTP)       ...   ...   ...   ...
PAM 4/0 (GEPAM)         ...      N         N         N
PAM 4/1 (GEPAM)         ...      N         N         N


FRPAM#          ING-SSRAM ING-SDRAM EGR-SSRAM EGR-SDRAM LOOPBACK
------------------------------------------------------------------
PAM 1/0 (FR4CE1)  .        .         .         .         .
   Ethernet-port Access:    .        Ethernet-port CAM-Access: .
   Ethernet-port Loopback:  .        Ethernet-port Loadgen:    .
   GEPAM Microcode:         .        GEPAM Access:            .
   GEPAM CAM Access:        .

Power-on Diagnostics Passed.

NewLs1010#
You can also see that Frame Relay/ATM cards are displayed in a different way, which would also be seen with the PCQ.

Whatever the output type, the main indication is that the PoD passed. If it failed, the ASP Status LED will be red. If you see some recoverable errors during the tests, the tests will perform normally, but a warning will be displayed. For example:

LS1010# show diag power
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Diags  Date: 00/04/11   Time: 02:14:57   By: V 3.44
 
   BOOTFLASH:  .   PCMCIA-Slot0: N   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .
 
   PS1:        .   PS2:          N   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .
 
   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .
 
   FC-PFQ
    Access: .
     RST: .    REG: .    IVC: .    IFILL: .    OVC: .    OFILL: .
 
    TEST:
     CELL: .   SNAKE: .   RATE: .   MCAST: .   SCHED: .
     TGRP: .   UPC  : .   ABR : .   RSTQ : .
 
Access/Interrupt/Loopback/CPU-MCast/Port-MCast/FC-MCast/FC-TMCC Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 0/0 (25M)           .....NN   .....NN   .....NN   .....NN   
    Port 4  to 7 :      .....NN   .....NN   .....NN   .....NN   
    Port 8  to 11:      .....NN   .....NN   .....NN   .....NN   
PAM 0/1 (155MM)         .....NN   .....NN   .....NN   .....NN   
PAM 4/0 (155MM)         .....NN   .....NN   .....NN   .....NN   
PAM 4/1 (E3)            .....NN   .....NN      N         N
 
   Ethernet-port Access:   .         Ethernet-port CAM-Access: .
   Ethernet-port Loopback: .         Ethernet-port Loadgen:    .
 
M4:Non-Volatile Memory Read/Write Test []
 *** MEMDIAG_NVRAM_MAGIC_PATTERN_DATA_ERROR *** [Addr:BE001008, exp:0000ABCD, act:00000000]
power-on Diagnostics Passed.

Field Definitions

The following tables explain only the fields related to the ports or memory. You can assume that any other test failures involve replacing the ASP. This includes diagnostics under the MMC, FC-PFQ, or feature card.

Chassis-Specific Fields
Field Definition
BOOTFLASH: Performs validation on the files present in the CPU-board-resident flash file system. This includes checking the presence of file system and checksum validation for the bootflash resident files. If it fails, the bootflash is bad. Reformat it using the LS1010 and recopy the files using the copy tftp command. 
PCMCIA-Slot[0 or 1] Same as the bootflash test.
[CPU or Fcard]IDPROM Performs validation of the CPU/Feature Card IDPROMS. If it fails, you will need to Return Material Authorization (RMA) the ASP.
NVRAM-Config Performs validation of the NVRAM. If it fails, try to configure the LS1010 using the IOS config mode commands. If it still fails, replace the ASP.
SRAM Performs Read/Write test on Static memory, which is 128K in size. If it fails, replace the ASP
DRAM Performs Read/Write test on Dynamic memory. Replace DRAM; if it still fails, replace ASP.
PS [1 or 2] Power supply
FAN Self-explanatory
Temperature Self-explanatory
Bkp-IDPROM Performs validation on the backplane IDPROM. Replace the chassis.

Card-Specific Fields (FC-specific test skipped)
Field Definition
Access This test makes sure that the PHY-layer HW resident on the various port adapter module (PAM) cards in the system are accessible. If it fails, replace the PAM.
Interrupt This test makes sure that the PHY-layer HW resident on the various PAM cards in the system are capable of interrupting the CPU under alarm condition. If it fails, replace the PAM.
Loopback Performs sourcing of unicast cells to the port and validates the received cells in loopback mode. If it fails, the connectivity is broken in the cell path. Try replacing the PAM first.
CPU-MCast Same as loopback test for multicast cells. In other words, the CPU acts as the root of the point-to-multipoint connection.
Port-MCast The CPU sends a unicast cell to the first port in the list, which in turn multicasts it to the rest of the port and validates the result in loopback mode.

Ethernet-specific Counters
Field Definition
Ethernet-port Access This test ensures the Ethernet port HW resident on the Ethernet controller present on the ASP is accessible. If it fails, the Ethernet controller is probably broken and the ASP needs to be replaced.
Ethernet-port CAM-Access Performs Read/Write on the built-in Content Addressable Memory (CAM) on the Ethernet controller. If it fails, the Ethernet controller is probably broken and the ASP needs to be replaced.
Ethernet-port Loopback Performs loopback test on the Ethernet port. If it fails, the Ethernet controller is probably broken and the ASP needs to be replaced.
Ethernet-port Loadgen Artificially loads the Ethernet port. If it fails, the Ethernet controller is probably broken and the ASP needs to be replaced.

SNAKE Test Failures on FC-PFQ

The SNAKE test sends a cell through all interfaces on the switch. This test ensures that all interfaces and associated fabric interfaces are functional. Cisco bug ID CSCdk54678 resolves a problem that causes the SNAKE test to fail on an LightStream 1010 running Cisco IOS® Software Release 11.3WA4 and using an FC-PFQ.

Recommendations

If you see a failed PoD caused by a PAM (from the show diag power-on output), you should  perform the following steps until the problem is resolved:
  1. Upgrade the Cisco IOS software to a more recent release (version 12.0 or higher) since some bugs have been resolved.
  2. Turn the LightStream 1010 off, re-seat the module, and turn the LightStream 1010 on again because an improperly-inserted PAM can definitely cause the tests to fail.
  3. RMA the PAM.
  4. RMA the ASP.

Conclusion

Finally, here is a typical example of a PoD that fails. It was solved by an RMA of the PAM. Similar issues have been solved by re-seating the PAM.
Switch# show diag power-on
LS1010 Power-on Diagnostics Status (.=Pass,F=Fail,U=Unknown,N=Not Applicable)
-----------------------------------------------------------------------------
   Last Power-on Date: 98/09/19   Time: 05:15:33

   BOOTFLASH:  .   PCMCIA-Slot0: N   PCMCIA-Slot1: N
   CPU-IDPROM: .   FCard-IDPROM: .   NVRAM-Config: .
   SRAM:       .   DRAM:         .

   PS1:        .   PS2:          .   PS (12V):     .
   FAN:        .   Temperature:  .   Bkp-IDPROM:   .

   MMC-Switch Access: .              Accordian Access: .
   LUT: .   ITT: .   OPT: .   OTT: .   STK: .   LNK: .   ATTR: .   Queue: .
   Cell-Memory:  .

   Feature-Card Access: .
   ICC: .   OCC: .   OQP: .   OQE: .   CC:  .   RT:  .
   TM0: .   TM1: .   TMC: .   IT:  .   LT:  .   RR:  .   ABR: .

Access/Interrupt/Loopback/CPU-MCast/Port-MCast/FC-MCast/FC-TMCC Test Status:
Ports                      0         1         2         3
----------------------------------------------------------------------------
PAM 10/0(155MM)         .......   .......   ..F....   .......   
PAM 10/1(155MM)         .......   .......   .......   .......   
PAM 11/0(155MM)         .......   .......   .......   .......   
PAM 11/1(155MM)         .......   .......   .......   .......   
PAM 12/1(155MM)         .......   .......   .......   .......   

   Ethernet-port Access:   .         Ethernet-port CAM-Access: .
   Ethernet-port Loopback: .         Ethernet-port Loadgen:    .

A4:ATM Layer Loopback Test [PM2P2,VP,Q,PHY,ASP_OSC]
 *** ATMDIAG_PIF_STAT_HEC_ERROR *** [Addr:08000001, exp:00000000, act:00000002]

Power-on Diagnostics Failed.

Related Information


Updated: Jun 05, 2005
Document ID: 21454