Guest

Cisco AS5800 Series Universal Gateways

Hardware Troubleshooting for the AS5800

Cisco - Hardware Troubleshooting for the AS5800

Document ID: 18899

Updated: Jul 14, 2005

   Print

Introduction

Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps you to troubleshoot common hardware issues on Cisco AS5800 Series Routers, and provides pointers for identifying whether or not there is a hardware fault.

Note: This document does not cover any software-related failures.

Prerequisites

Requirements

Readers of this document should have knowledge of these topics:

  • It is recommended that you read Troubleshooting Router Crashes before proceeding with this document.

  • You should be familiar with the various hardware parts on the AS5800. For more information on the components of the AS5800, refer to the document AS5800 Introduction. The Cisco AS5800 Product Overview also includes diagrams for the various components.

  • You should also be knowledgeable of using Cisco IOS® software on the AS5800, and executing commands on the router-shelf (RS) and dial-shelf (DS).

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco AS5800 Series Routers

  • Cisco IOS Software (all versions)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Background Information

Whenever you install a new card/module or Cisco IOS software image, it is important to verify that the access server has enough memory, and that the hardware and software are compatible with the features you wish to use.

Perform these recommended steps to check for hardware-software compatibility and memory requirements:

Note: To use the tools referenced in this document, you must be a registered customer, and you must be logged in.

  1. Use the Software Advisor tool (registered customers only) to choose software for your network device.

    Tips:

  2. Use the Download Software Area (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. To determine the amount of memory (RAM and Flash) installed on your router, refer to How to Choose a Cisco IOS Software Release - Memory Requirements.

    Tips:

    • If you want to keep the same features as the version that is currently running on your router, but do not know which feature set you are using, issue the show version command on your router and paste the output in the Output Interpreter tool (registered customers only) to find out. It is important to check for feature support, especially if you plan to use recent software features.

    • If you need to upgrade the Cisco IOS software image to a new version or feature set, refer to How to Choose a Cisco IOS Software Release for more information.

  3. If you determine that a Cisco IOS software upgrade is required, see Upgrading IOS Software on Cisco AS5800 Access Servers - Practical Example.

Troubleshooting Flowchart

This document includes some flowcharts to assist in troubleshooting. Click on a particular box to go to the section describing the procedure. Also, proceed to the next flowchart by clicking on the numbers.

5800_flow1.gif

To Verify System Health Using the Onboard LEDs

The LEDs on various parts of the AS5800 platform can provide a wealth of information regarding the status of the different components. This section guides you through the procedure of using the LEDs to verify system health.

This diagram shows the various components of the AS5800:

5800_hw_ts2.gif

Power-On the Router

Power-on these components in the order described below:

  1. AC-Input Power Shelf: If you are using the AC-input power shelf, power-on ( | ) the two power switches located on the AC-input power shelf front panel. Four green power OK LEDs should light.

  2. Dial-Shelf: Power-on ( | ) the power switches located on each dial shelf Power Entry Module (PEM) front panel. A green power LED on each PEM should light. You should hear the fans on the dial shelf power up.

    Note: Always power-on the dial shelf before the router shelf. The Dial Shelf Controller (DSC) card checks the dial shelf's inventory, which requires extra time. After two minutes, power-on the router shelf. The router shelf depends on the DSC card for the dial shelf's inventory report.

  3. Router-Shelf: Power-on ( | ) the two power switches located on the rear of the router shelf (7200). Both green power supply OK LEDs should light. You should hear the fans on the router shelf power up.

For more information on this procedure, or for diagrams identifying the various power switches described above, refer to the section Powering On the Cisco AS5800 in the document Powering On the Cisco AS5800 and Observing Initial Startup Conditions.

To Identify Hardware Subsystem Failures Using LEDs

You can use the various LEDs to determine whether any of the hardware subsystems failed during the boot sequence. The general flow of reasoning looks for the following (answering "Yes" advances to the next question):

  1. Do the green Power OK LEDs illuminate on the AC-Input Power Shelf, Dial-Shelf and Router-Shelf?

    To identify the various LEDs, refer to Powering On the Cisco AS5800. If any of the green Power OK LEDs do not come on, refer to the Troubleshooting the Power Subsystems section of the AS5800 Hardware Troubleshooting document.

  2. Are the fans operating?

    If the fans do not operate, refer to the Troubleshooting the Cooling Subsystems section of the AS5800 Hardware Troubleshooting document.

  3. Do the Input/Output (I/O) LEDs on the router shelf (7206) light in the proper sequence?

    The I/O LEDs should light in the following sequence. If they do not light as described, proceed to the Troubleshooting the Processor Subsystems section of the AS5800 Hardware Troubleshooting document.

    1. The I/O power OK LED lights up immediately and indicates that the I/O controller card is receiving DC power from the router midplane. This LED lights up during a successful router boot and remains green during normal operation of the router.

    2. The enabled LED lights up after the I/O power OK LED and indicates that the network processor card and the I/O controller card are enabled for operation by the system; however, that the enabled LED lights up does not mean that the Fast Ethernet port on the I/O controller card is functional or enabled. This LED lights up during a successful router boot and remains green during normal router operation.

    3. The FE enabled LED lights up after the enabled LED and indicates that the Fast Ethernet port on the I/O controller card is initialized and enabled for operation by the system. This LED lights up during a successful router boot and remains green during normal router operation.

    4. The FE link LED lights green only when the Fast Ethernet port on the I/O controller card is receiving a carrier signal from the network. (This LED remains off during normal router operation unless there is an incoming carrier signal and does not indicate startup problems.)

    5. The slot 1 and slot 2 LEDs light up after the enabled LED and indicate which PCMCIA slot is in use.

      During a successful router boot, these LEDs light up and remain on if a Flash memory card is present in the slot. These LEDs also blink when either slot is being accessed by the system. These LEDs do not indicate startup problems.

      If the LEDs do not light in the above sequence, try reseating the I/O Controller Card and restarting. If this does not help, refer to the Troubleshooting the Processor Subsystems section of the AS5800 Hardware Troubleshooting document.

  4. Do the port adapter-enabled LEDs (on the router shelf) come on?

    The port adapter-enabled LEDs each light up when the network processor card completes its initialization of the port adapter for operation. The enabled LED indicates that the port adapter is receiving power and has been recognized by the network processor card; it does not indicate the state of the individual port adapter interfaces.

    If an enabled LED does not light up, first try reseating the port adapters. If this does not help, refer to the Troubleshooting the Port Adapters section of the AS5800 Hardware Troubleshooting document.

  5. Does the Dial Shelf Interconnect (DSI) port adapter Link LED (on the router-shelf) stay on?

    The link status LED indicates an active connection to the dial shelf and lights up when the interconnect port adapter is receiving a carrier signal from the dial shelf controller card. This LED should light up and remain green when the access server is operating.

    5800_hw_ts3.gif

    If the interconnect port adapter link LED does not light up, refer to the Troubleshooting the Dial Shelf Interconnect Port Adapter section of the AS5800 Hardware Troubleshooting document.

  6. Do the LEDs on the Dial-Shelf light in the proper sequence?

    The LEDs on the dial shelf controller card, modem cards, and trunk cards should light as follows:

    1. Dial shelf controller card—Powers on first, followed by remaining logic cards. The PWR and MBus LEDs should light green. The remaining four alarm LEDs should all briefly flash on and then turn off.

      If the MBus LED remains off, but the power LED is green, there might be a problem with the MBus power supply on the card. Replace the card.

    2. Trunk card—At initial power on, all LEDs light up while the system runs a series of diagnostics. After the system passes initial diagnostics, all LEDs go off. The power, host CPU, and framer CPU LEDs then light green to indicate that the trunk card is operating properly.

    3. Modem card—During normal operation, all five LEDs light at system power up. When the modem card CPU software image starts running, it shuts off the modem LED. The power, host CPU, and modem LEDs then light green to indicate that the modem card has passed diagnostics and is operating properly.

    If the dial shelf processor component LEDs do not light as described, refer to theTroubleshooting the Dial Shelf Processor Subsystem section of the AS5800 Hardware Troubleshooting document.

The above steps address most common AS5800 hardware issues that are identifiable using LEDs. Continue to the next section for further troubleshooting.

Troubleshooting Flowchart (continued)

5800_flow2.gif

To Verify that the Dial-Shelf and Router-Shelf are running the same Cisco IOS Software Release

The dial shelf (DS) and router shelf (RS) run separate Cisco IOS images:

  • Both images MUST be from the same Cisco IOS software release. They must match.

  • The RS image is located in the Flash memory of the Cisco 7206. The filename begins with "c5800". The DS image is in the DSC card. The filename begins with "dsc."

Perform a show version on the RS to determine the Cisco IOS software version currently running. On the DS, you can also issue the show version command to check the Cisco IOS software version; however, if you do not have a physical console connection to the DS, issue the command execute-on slot [12 | 13] show version on the RS. The DSC can be in slot 12 or 13. Verify that the IOS versions on the DS and RS are the same. For more information, refer to section 1.2, "Matching the Cisco IOS Images", in the document Commissioning the Cisco AS5800 Hardware.

If the RS and DS run different image versions, some of the cards on the DS may not come up properly, or the RS may completely lose connectivity with the DS.

This example demonstrates a situation where the Cisco IOS software versions on the DS and RS are different. The router shelf is running Cisco IOS Software Release 12.2(23), and the dial shelf is running Cisco IOS Software Release 12.1(8). Begin by checking the Cisco IOS software version running on the Router-Shelf. Connect to the router shelf (using Console, Telnet, and so on) and issue these commands:

AS5800#show version
Cisco Internetwork Operating System Software
IOS (tm) 5800 Software (C5800-P4-M), Version 12.2(23), RELEASE SOFTWARE (fc2)

!--- The router-shelf is running IOS Version 12.2(23)

TAC:Home:SW:IOS:Specials for info
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Tue 04-Sep-01 17:45 by samuel
Image text-base: 0x60008960, data-base: 0x61B40000
   
ROM: System Bootstrap, Version 12.0(19990210:195103) [12.0XE 105], DEVELOPMENT SOFTWARE
BOOTFLASH: 7200 Software (C7200-BOOT-M), Version 12.0(5)T1, RELEASE SOFTWARE (fc1)

c5800 uptime is 15 minutes
System returned to ROM by power-on
System image file is "slot0:c5800-p4-mz.122-23.bin"

!--- The router-shelf image filename.  Note that the filename begins with c5800.

cisco 7206VXR (NPE300) processor (revision B) with 122880K/40960K bytes of memory.
Processor board ID 13251981
R7000 CPU at 262Mhz, Implementation 39, Rev 1.0, 256KB L2, 2048KB L3 Cache
6 slot VXR midplane, Version 2.0

!--- Output suppressed

Determine whether the Dial-Shelf is located in slot 12 or 13. Issue the show dial-shelf command on the router-shelf.

AS5800#show dial-shelf
Slot    Board     CPU       DRAM          I/O Memory   State         Elapsed
         Type     Util    Total (free)   Total (free)                Time
 0        CT1                  0(  0%)        0(  0%)  Unknown       00:15:19
 1        CT3                  0(  0%)        0(  0%)  Unknown       00:15:19
 2        CT1                  0(  0%)        0(  0%)  Unknown       00:15:19
 3 Modem(HMM)                  0(  0%)        0(  0%)  Unknown       00:15:19
 5 Modem(DMM)                  0(  0%)        0(  0%)  Unknown       00:15:19
11      UP324                  0(  0%)        0(  0%)  Unknown       00:15:19
12        DSC    0%/0%  18470752( 82%)  8388608( 66%)  Up            00:15:03

!--- The Dial Shelf Controller is in slot 12

AS5800#

Issue the show version command on the DS. You can either connect directly to the console port on the DS or you can issue the command execute-on slot <dsc_slot_number> show version on the RS. In this case, you have previously determined that the DSC is in slot 12.

c5800#execute-on slot 12 show version
User Access Verification
   
Password:
Cisco Internetwork Operating System Software
IOS (tm) 5800 Software (C5800-DSC-M), Version 12.1(8), RELEASE SOFTWARE (fc1)

!--- The Dial-Shelf is running IOS Version 12.1(8)


Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Tue 17-Apr-01 03:45 by kellythw
Image text-base: 0x60008940, data-base: 0x605AC000

ROM: System Bootstrap, Version 11.2(19980113:222207) [tkam-nitro_ios_rommon_011398 555], INTERIM SOFTWARE
ROM: 5800 Software (C5800-DSC-M), Version 11.3(2)AA, EARLY DEPLOYMENT, RELEASE SOFTWARE (fc1)

DA-Slot12 uptime is 16 minutes
System returned to ROM by power-on
System image file is "slot0:dsc-c5800-mz.121-8"

!--- The Dial-Shelf image filename. Note that the filename begins with dsc.


!--- Output suppressed

Note: The image versions on the RS and the DS are not the same.

Now issue the show dsip version command on the router-shelf to check whether the router-shelf has proper connectivity to the dial-shelf using DSIP.

AS5800#show dsip version

DSIP version information:
------------------------
Local DSIP major version = 5, minor version = 2

All feature boards are running DSIP versions compatible with router shelf

Local clients registered versions:
------------------------------------
Client Name Major Version Minor Version


!--- Output suppressed

DSIP Test   No version     No version
   
Mismatched remote client versions: 

!--- Remote client version mismatches exist

   -----------------------------------
   Client Name  Slot  Major Version   Minor Version
   Split DS     12     No version       No version 
   NextPort     12     No version       No version

Note: The show dsip version command indicates that there are mismatched remote client versions. Ideally, you should not see any mismatches.

To Upgrade the IOS Image on the Router-Shelf, Dial-Shelf, or Both

You can upgrade the Cisco IOS image on the RS, the DS, or both. Follow the procedure described in these documents:

To Troubleshoot the Dial-Shelf Controller and Feature Boards

5800_flow3.gif

From the router-shelf, issue the show dial-shelf command to determine whether the DSC and associated feature boards are Up. In the example shown here, all feature boards are Up.

Router#show dial-shelf

Slot    Board     CPU       DRAM          I/O Memory   State         Elapsed
         Type     Util    Total (free)   Total (free)                Time
 0        CT3    0%/0%  21598976( 81%)  8388608( 41%)  Up            00:01:35
 2 Modem(DMM)  20%/20%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 3 Modem(DMM)    0%/0%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 4 Modem(DMM)  20%/20%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 5 Modem(DMM)  20%/20%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 6 Modem(DMM)  40%/40%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 7 Modem(DMM)  40%/40%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 8 Modem(DMM)  35%/35%  46764800( 86%) 16777216( 74%)  Up            00:01:35
 9 Modem(DMM)    0%/0%  46764800( 86%) 16777216( 74%)  Up            00:01:35
10 Modem(DMM)  20%/20%  46764800( 86%) 16777216( 74%)  Up            00:01:34
12        DSC    0%/0%  19097792( 79%)  8388608( 66%)  Up            00:02:49

Dial shelf set for auto boot
Router#

If all feature boards are Up, then any hardware issue is not the result of a bootup problem. Proceed to the section Common Hardware Error Messages.

  • Dial-Shelf Controller is not Up

  • One or more Dial-Shelf feature boards are not Up

    • MICA/NextPort feature board is not Up

    • T1/E1/T3/E3 trunk line card is not coming Up

The Dial Shelf Controller is Not Up

Note: Follow the procedure described in the section Troubleshooting the Dial Shelf Controller Card in the document AS5800 Hardware Troubleshooting before continuing with this section.

If the process described in that document does not solve the problem, continue below:

  1. Reload the DSC by issuing the command execute-on slot {12 | 13} reload.

  2. Look for LED lights on the DSC card. If the lights are off, try re-seating the card. Refer to the section Troubleshooting the Dial Shelf Controller Card in the document AS5800 Hardware Troubleshooting for more information. Also, if the other DSC slot (12 or 13) is available, then move the DSC to the free slot.

  3. Verify that the DSI port adapter on the router-shelf is inserted correctly. Refer to the section Identifying Hardware Subsystem Failures Using LEDs for more information.

  4. Verify that the cable between the DSI port adapter and the DSC card is connected correctly. You can try replacing the DSI cable as well. For more information on attaching the DSI cable refer to the section Attaching the Dial Shelf Interconnect Cable in the document Cisco AS5800 Universal Access Server Dial Shelf Interconnect Port Adapter Installation and Replacement.

  5. From the router-shelf (Cisco 7206), verify that the DSI-Fast Ethernet interface and line protocol are up:

    AS5800>show dsi
    DSI-Fastethernet0/2/0 is up, line protocol is up 
    Hardware is DEC21140A, address is 0030.f2f5.1438 (bia 0030.f2f5.1438)
    
    
    !--- Output suppressed
    
    
  6. For advanced troubleshooting measures after the DSC card is up, open a virtual-console session to the DSC card (DA-Slot12) from the router shelf by issuing the command dsip console slave 12|13 . To end the session, press Ctrl+C three times:

    AS5800#dsip console slave 12
    
    Trying Dial shelf slot 12 ...
    Entering CONSOLE for slot 12
    Type "^C^C^C" to end this session
       
       
    DA-Slot12>
    DA-Slot12#
    Terminate NIP IO session? [confirm]
       
    [Connection to Dial shelf slot 12 closed by local host]
    Router#
  7. You can also try a dsip ping from the router shelf and see if the DSC responds. Issue the dsip ping destination slot_number command:

    AS5800#dsip ping destination ?
    <0-13>  remote slot number
    rs      destination is RS
    AS5800#dsip ping destination 12
    AS5800#!!!!!!!!
    

    Tip: Once a dsip ping has been activated, you cannot stop it as you would a normal ICMP ping. Since the router prompt does not appear, you must blindly type dsip ping off to turn it off (as displayed in the example below):

    AS5800#!!!!!!!d!!si!!p!! ping !!off
    

One or More Feature Boards are not Up

If one or more feature boards on the dial-shelf do not come up during the boot sequence, then you must delve into the specific module for further troubleshooting. However, you can perform some troubleshooting procedures regardless of the type of feature board not coming up. Follow the procedure below:

  1. Verify that each feature board's MAC addresses are registered by DSIP. Unregistered boards cannot communicate with the system. Shelf 0 is the router shelf (master). Shelf 1 is the default dial shelf (slave). Issue the show dsip transport command to verify that the boards are registered by the system. For example:

    c5800# show dsip transport
    
    DSIP transport statistics:
     IPC  : input msgs=2753859, bytes=406835215; output msgs=2753733, bytes=156251079
            total consumed ipc msgs=15316;  total freed ipc msgs = 15316
            transmit contexts in use = 18, free = 238, zombie = 0, invalid = 0
            ipc getmsg failures = 0, ipc timeouts=0
            core getbuffer failures=0, api getbuffer failures=0
            dsip test msgs rcvd = 53, sent = 53
     CNTL : input msgs=1040705, bytes=50803786; output msgs=679316, bytes=64974836
            getbuffer failures=0
     DATA : input msgs=27811, bytes=891245; output msgs=43985, bytes=675187
    DSIP Private  Buffer Pool Hits  = 0
    DSIP registered addresses:
     Shelf0 : Master: 00d0.06d3.0870, Status=local
     Shelf1 : Slot0 : 0010.7b03.a700, Status=remote
     Shelf1 : Slot1 : 0010.7b03.a708, Status=remote
     Shelf1 : Slot2 : 0010.7b03.a710, Status=remote
     Shelf1 : Slot3 : 0010.7b03.a718, Status=remote
     Shelf1 : Slot5 : 0010.7b03.a728, Status=remote
     Shelf1 : Slot11: 0010.7b03.a758, Status=remote
     Shelf1 : Slot12: 0010.7b03.a760, Status=remote
    c5800#

    In the above example, note that dial-shelf slots 0,1,2,3,5,11, and 12 have been registered by DSIP.

  2. Perform an Online Insertion and Removal (OIR)/Hotswap using the command line. For more information, use the OIR procedure described in the section %DSIPPF-5-DS_KEEPALIVE_LOSS: DSIP Keepalive Loss from shelf <x> slot <y>.

  3. Reseat the card or try the same card in another slot. If the card comes up in another slot, then the card is fine and the slot has gone bad.

  4. Verify that all feature boards (cards) are running DSIP versions that are compatible with the router shelf. Issue the show dsip version command to determine this.

  5. If the card does not come up for a long period, activate these debugs: debug dsip transport, debug dsip trace and debug dsip boot . The debug dsip transport command shows the registered MAC address sent from each feature board. The debug dsip trace command displays detailed DSIP hello and keepalive messages. The debug dsip boot command shows if the router shelf is sending the boot image to the feature boards.

  6. Proceed to the troubleshooting section corresponding to the feature board in question.

MICA/NextPort Feature Board is not Up

If the MICA/NextPort feature board does not come up for an extended period of time, then follow the procedure described below:

  1. Make sure you are running a Cisco IOS software version that supports Nextport or MICA cards. Use the Software Advisor tool (registered customers only) located on the Support Tools & Resources page to verify this.

  2. The MICA HMM Modem (144 Modems) feature card must have at least 64 MB of DRAM if running Cisco IOS Software Release 12.0 or later. For more information, see Field Notice: AS5800 - 64MB DMM DRAM is Required for Running all Cisco IOS 12.0-Based Releases.

    MICA HMM feature cards (72 Modems) require 32 MB of DRAM for Cisco IOS Software Release 12.0 and later. This is described in Field Notice: AS5800 - 32MB HMM DRAM Required to Run all Cisco IOS 12.0- Based Releases.

  3. Perform OIR/Hotswap through the command line. Use the OIR procedure described in the section %DSIPPF-5-DS_KEEPALIVE_LOSS: DSIP Keepalive Loss from shelf <x> slot <y> for more information.

Issues Related to the MICA or NextPort Modem Card

If the MICA or NextPort Modem card is not recognized by the dial-shelf (using the command show dial-shelf) then the problem could be on the Modem Card or it could be a software or configuration issue related to the Modem Card. Consult these documents depending on your needs:

Note: Ensure that MICA modems are running portware version 2.7.3.0. If you are not running 2.7.3.0, upgrade the modem code.

T1/E1/T3 Trunk Cards are not Booting Up

Verify that the trunk card LEDs operate properly by observing these LED states:

  • The power LED is ON.

    If the power LED remains OFF, verify that the card is seated properly.

    If the power LED lights up on other trunk cards in the dial shelf, try inserting the trunk card in a different slot. If none of the power LEDs light up, check your dial shelf power connections, power entry modules, and AC-input power supplies (if present). Perform an OIR.

  • The HCPU LED is ON.

    If the Host CPU (HCPU) LED is OFF but the power LED is ON, the software image might have failed to load onto the card. The dial shelf controller attempts to reload the software automatically. After a programmed number of attempts to reload the software image fails, the dial shelf controller powers OFF the trunk card and all LEDs shut off.

    If this happens, assume that the failure is due to defective hardware. Contact Cisco Technical Support for further troubleshooting.

  • The FCPU LED is ON.

    If the Framer CPU (FCPU) LED is OFF while the HCPU LED is ON, either the hardware is defective or the framer processor software has crashed. To determine if the failure is software related, wait while the auto-reload feature on the dial shelf controller card attempts to reload the software image. If the software fails to reload after the programmed number of times, assume that the failure is due to defective hardware. Contact the Cisco Technical Support for further troubleshooting.

References:

Issues Related to the T1/E1/T3 Trunk Card

If the T1/E1/T3 trunk line card is not recognized by the dial-shelf (using the show dial-shelf command), then the problem could be on the trunk card, or it could be a software or configuration issue related to the trunk card. Consult these documents depending on your needs:

Common Hardware Error Messages

This section provides information on common error messages associated with hardware issues. A troubleshooting procedure for each error is also included.

The Error Message Decoder tool (registered customers only) allows you to check the meaning of an error message. Error messages appear on the console of Cisco products, usually in this form:

%XXX-n-YYYY : [text]

This is an example error message:

%TRUNK-3-NOMSGS: Cannot send message to framer; no more messages available

Some error messages are informational only, while others indicate hardware or software failures and require action. The Error Message Decoder tool (registered customers only) provides an explanation of the message, a recommended action (if needed), and if available, a link to a document that provides extensive troubleshooting information about that error message.

For a comprehensive list of error messages, refer to the document System Error Messages for the Cisco AS5800 Universal Access Server.

5800_flow4.gif

%DEC21140-5-NOCARR: DSI-Fastethernet 0/2/0 Cable/Connector problem?

This error message indicates that the router-shelf has lost connectivity to the dial-shelf, through the Dial Shelf Interconnect (DSI) cable, connected to port 0/2/0 on the router-shelf. Issue the show dsi command to verify that the connection between the router-shelf and dial-shelf is up. Verify that both the interface and the line protocol are in an UP state. For example:

AS5800#show dsi
DSI-Fastethernet0/2/0 is up, line protocol is up 

!--- Interface is UP/UP

Hardware is DEC21140A, address is 00d0.d342.4c38 (bia 00d0.d342.4c38)

!--- Output suppressed

Perform these steps if the connection between the router-shelf and dial-shelf is not in the UP state:

  1. If the DSI-Fastethernet interface is not UP, perform a no shutdown on the interface.

  2. Check whether the dial-shelf interconnect port adapter-enabled LED is on.

    If it is not, verify that the interconnect port adapter is seated properly and that the port adapter levers are in the locked position. You can also try to reseat the port adapter.

    If power is supplied to the router-shelf, but the enabled LED remains off, the system has detected a hardware failure. You need to replace the interconnect port adapter.

  3. Verify that the link LED on the DSI Port Adapter is on.

    The link LED indicates an active connection to the dial-shelf. This LED lights when the dial-shelf interconnect port adapter is receiving a carrier signal from the dial-shelf.

    If the link LED remains off, check the Dial-Shelf Interconnect Cable (DSIC) connection and tighten the jack screws at both ends of the cable.

  4. Replace the DSI Cable (Part number CAB-DSIC-5= or CAB-DSIC-20=). Make sure that the new cable is properly secured.

    The dial-shelf interconnect port adapter contains a single RJ-45 receptacle, which is used to connect the router-shelf to the dial-shelf. The cable used for this connection is a Cisco proprietary cable, customized with jackscrews to secure the connection. You must use this specially designed cable that shipped with your dial-shelf interconnect port adapter to connect the dial-shelf to the router-shelf.

    5800_hw_ts4.gif

    For more information on attaching the DSI Cable, refer to the section Attaching the Dial Shelf Interconnect Cable in the document Cisco AS5800 Universal Access Server Dial Shelf Interconnect Port Adapter Installation and Replacement.

  5. Troubleshoot the dial-shelf. Proceed to the section Troubleshooting the Dial Shelf Controller Card in the document Hardware Troubleshooting.

%DSCCLOCK-3-FAIL: The System Primary Clock is Down. Moving to HOLDOVER STATE and Waiting to See if it Comes Up

The DSC Clock Synchronization Circuit (DCSC), normally locks on to a clock derived from one of the incoming trunk ports either based on user configuration or using a default mechanism. This selected clock acts as the TDM primary clock. If this primary clock fails, the DCSC moves from NORMAL to HOLDOVER state.

In HOLDOVER mode, DCSC still provides timing and synchronization signals, which are not locked to an external reference signal, but are based on storage techniques. The storage value is determined while the device is in Normal mode and locked to an external reference signal. Note that the accuracy of the TDM clock derived while the DCSC is in HOLDOVER mode decreases, and slips occur eventually if the problem is not rectified.

Normally, on the AS5800, with two trunks installed and at least two ports connected to an external network, there exist one primary clock and three backup clocks. So, when a primary fails, the DCSC immediately selects a backup as a new primary.

The DSC primary can enter holdover mode for one of these reasons:

  • The trunk card providing the primary has crashed.

  • The port providing the primary, has gone bad or cable has been removed.

  • The clock that is being derived from the trunk port has gone bad.

  • When the router-shelf reloads, it reloads all feature boards, and a message appears on the DSC console indicating that the primary clock has gone into holdover mode.

Use this procedure to isolate the issue:

  1. Issue the show controllers {t1 | e1} command to check whether the T1s are experiencing errors. If the T1 is not clean, refer to the T1 Troubleshooting Flowchart. You may also have to contact your telco for further T1 line troubleshooting. Ensure that the T1 lines are clean and stable before proceeding further.

  2. Reseat the CE1/T1/T3 trunk cards (feature boards) on the Dial-Shelf. Use the procedure described in the document Replacing or Installing Dial Shelf Cards.

    warning Warning: This causes all calls through that card to be dropped.

  3. Try moving the T1/E1 line in question to a different port on the same card. This can help isolate problems associated with a specific port on the T1/E1 trunk card.

  4. Follow the instructions below to configure the AS5800 to set the primary clock source as a different T1/E1 line. This can help to overcome clocking and slip problems, commonly seen when using T1s from multiple vendors into the same unit. Instructions

    For more information on clocking, refer to the document Clock Synchronization for AS5xxx Network Access Servers.

    1. Issue the show dial-shelf clocks command to determine the current primary and secondary clock sources. This output shows a good Primary and Secondary (Backup) Clock Source:

      AS5800# show dial-shelf clocks
      Primary Clock:
      --------------
      Slot 12:
      System primary is 1/3/1 of priority 3 
      
      !--- The primary clock is T1 1 in slot 3
      
      TDM Bus Master Clock Generator State = NORMAL
      
      
      !--- Clock state is NORMAL. This is the desired state
      
      
       
      Backup clocks:
      Source  Slot    Port    Priority      Status      State
      -------------------------------------------------------
      Trunk   1       2       10             Good        Configured 
      
      !--- The Backup clock source is T1 2 in slot 1
      
       
      Status of trunk clocks:
      -----------------------
      Slot    Type    11 10  9  8  7  6  5  4  3  2  1  0
      1       T1       B  B  B  B  B  B  B  B  B  G  B  B  
      3       T1       B  B  B  B  B  B  B  B  B  B  G  B  
      AS5800#

      This output shows a bad Primary Clock Source. A Secondary (Backup) clock source is not configured:

      AS5800# show dial-shelf clocks
      Slot 12:
      System primary is 6/76/0 of priority 76
      TDM Bus Master Clock Generator State = HOLDOVER
      
      !--- Clock state is HOLDOVER. This is NOT Desired.
      
       
      Backup clocks:
      Source  Slot    Port    Priority      Status      State
      -------------------------------------------------------
      
      !--- No Backup clocks are configured
      
       
      Slot    Type    11 10  9  8  7  6  5  4  3  2  1  0
      0       E1       B  B  B  B  B  B  B  B  B  B  B  B
      
      
    2. Issue the dial-tdm-clock command to configure a T1 as the primary clock source. Refer to the section Configuring Clocking in the document Configuring Your Cisco AS5800 Universal Access Server.

%DSIPPF-5-DS_KEEPALIVE_LOSS: DSIP Keepalive Loss From Shelf <x> slot <y>

This indicates that the router-shelf has not received a scheduled keepalive message from slot y on the dial-shelf x. The default value for x is 1 if there is only one DSC. Perform these steps to troubleshoot:

  1. Issue the show dial-shelf command to verify that the feature board (card) in slot y is UP. If the Board in slot y is up, then this error could be caused by a faulty DSIC cable. Replace the cable (Part number CAB-DSIC-5= or CAB-DSIC-20=) and check if the error message disappears. Make sure that the new cable is properly secured.

    For more information on attaching the DSI Cable refer to the section Attaching the Dial Shelf Interconnect Cable in the document Cisco AS5800 Universal Access Server Dial Shelf Interconnect Port Adapter Installation and Replacement.

  2. Perform an OIR procedure on the affected feature board in slot y. This effectively resets the affected feature board.

    1. Execute the show dial-shelf slot slot_number command from the enable prompt on the RS and note down the numeric value associated with the type of the FB (in slot y) you are trying to reset. In this sample output, the Board Type for slot 3 is 259:

      AS5800#show dial-shelf slot 3 
      Slot: 3, Type: Channelised T3 (259)
      
      !--- Output suppressed
      
      
    2. Open a virtual connection to the DSC using the command dsip console slave dsc_slot_number. The value for dsc_slot_number should be either 12 or 13 depending on the slot number of the DSC that the affected feature board belongs to. For example:

      AS5800#dsip console slave 12
           Trying Dial shelf slot 12 ...
           Entering CONSOLE for slot 12
           Type "^C^C^C" to end this session
    3. Issue the command oir testport enable to enable OIR simulation.

      AS5800-Slot12#oir testport enable
      
    4. Issue the command oir slot slot_number remove to remove the card using the software. The value for slot_number should be the slot number of the feature board you want to OIR (slot y in the error message above).

      AS5800-Slot12#oir slot 3 remove
           AS5800-Slot12#
           08:45:53: %OIR-6-REMCARD: Card removed from slot 3, interfaces disabled
    5. To re-insert the card back, issue the command oir slot slot_number insert <board type from step (a)> .

      AS5800-Slot12#oir slot 3 insert 259
           AS5800-Slot12#
           08:46:00: %OIR-6-INSCARD: Card inserted in slot 3, interfaces administratively shut down
    6. To check the OIR events, issue the oir log command:

      AS5800-Slot12#oir log
        Slot 3: CT3 card inserted
      
      
  3. Physically remove and reseat the feature board that is experiencing errors. Refer to the document Replacing or Installing Dial Shelf Cards for more information.

  4. Remove and insert the feature board in another empty slot. Keep in mind that T3/T1/E3/E1 trunk cards can only be installed in slots 0 though 5 (slots 0 and 1 are recommended). Modem feature boards can only be installed in slots 0 through 11.

%ENVM-4-ENVWARN chassis outlet 3 measured at 50C/122F

This error message indicates that the router has detected an abnormally high chassis temperature. Verify that all three fans are operational and that ventilation in the facility is adequate. Refer to Troubleshooting the Cooling Subsystems for more information on troubleshooting the fans. You should consider powering down the AS5800 to prevent hardware damage.

%DSC_ENV-1-CRITICAL: Slot 24, 48V supply outside critical range Value=79 V; Power supply may shut itself off

This error message indicates a problem with the power supply subsystem. Verify that the input cable from the AC to DC Supply Shelf is wired correctly. Consult a qualified electrician, if necessary.

You should also check whether the bus voltage at the connector is -48Volts. If the input voltage is not 48Volts, then there is a power supply issue. You can measure the voltage using a Digital Volt Meter at the PEM input connector.

Tip: You can use the show environment all command to verify that there are no critical grounding, heating, or power problems.

This example shows a normal operating environment:

AS5800>show environment all
Power Supplies:
             Power supply 1 is empty.
             Power supply 2 is Zytek AC Power Supply. Unit is on.

Temperature readings:
             chassis inlet    measured at 25C/77F
             chassis outlet 1 measured at 27C/80F
             chassis outlet 2 measured at 33C/91F
             chassis outlet 3 measured at 41C/105F

Voltage readings:
             +3.45 V measured at +3.49 V
             +5.15 V measured at +5.21 V
             +12.15  measured at +12.34 V 
             -11.95  measured at -11.81 V 

Envm stats saved 1 time(s) since reload
AS5800>

Refer to the Troubleshooting the Power Subsystems in the document Hardware Troubleshooting for more information.

%DSC_ENV-0-CRITICAL_BLOWER: Slot 28, Fans in the blower unit have failed. Operation of unit may lead to cards overheating and being shutdown

The above error message can be generated due to one of these reasons:

  • The fan actually has failed inside the fan blower unit, which means that the fan tray has to be replaced. Refer to the section Replacing the Blower Assembly in the document AS5800 Maintenance for more information on the replacement procedure.

  • The Mbus module that resides in the EMI filter module is defective and needs to be replaced.

As far as cooling goes, if you start to see warning messages from each card stating that the card is going overtemp, then the cards automatically power down in minutes if proper cooling is not restored.

Try this procedure:

  1. Swap the fan tray, and see if the problem follows the tray on another system.

  2. If the error message does not move with the fan tray, then the Filter module must be pulled and replaced.

Summary

Once you are able to identify the piece of hardware that is causing the issue, open a service request online with the Cisco Technical Support and request a Return Materials Authorization (RMA) for the part that is causing the issue if the router is covered by a valid warranty or service contract. Make sure you follow the procedure described in the document Cisco IOS Software Commands for Cisco AS5800 Hardware Inspection before calling the Cisco Technical Support.

If you were not successful in identifying which piece of hardware is causing the issue, open a service request with the Cisco Technical Support, attach all relevant information such as troubleshooting logs and troubleshooting steps taken, and an engineer can assist you in further troubleshooting the issue.

Related Information

Updated: Jul 14, 2005
Document ID: 18899