Guest

Cisco AS5200 Series Universal Access Servers

Hardware Troubleshooting for AS5200 and AS5300 Series Routers

Document ID: 17881

Updated: Mar 09, 2009

   Print

Introduction

Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps you to troubleshoot common hardware issues on Cisco AS5200 and Cisco AS5300 series routers, and provides pointers for identifying whether or not there is a hardware fault. This document does not cover any software-related failures except for those that are often mistaken for hardware issues. For more information on identifying the modules and controller cards installed on the AS5200 and AS5300 Series, see Identifying the Controllers and Modem Hardware On AS5xxx Platforms.

Prerequisites

Requirements

We recommend that you read Troubleshooting Router Crashes before proceeding with this document.

Components Used

The information in this document covers all AS5200 and AS5300 series routers running any version of Cisco IOS® software. This document does not cover AS5350 series routers.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Hardware-Software Compatibility and Memory Requirements

Whenever you install a new card, module, or Cisco IOS software image, it is important to verify that the access server has enough memory, and that the hardware and software are compatible with the features you wish to use.

Perform the following recommended steps to check for hardware-software compatibility and memory requirements:

  1. Use the Software Advisor tool (registered customers only) to choose software for your network device.

    Tip:

    • The "Software Support for Hardware" section (available for registered customers only) helps you to verify whether the modules and cards installed are supported by the desired Cisco IOS software version.

    • The "Software Support for Features" section (available for registered customers only) helps you to determine the Cisco IOS software image needed by choosing the types of features you wish to implement.

  2. Use the Download Software Area (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. To determine the amount of memory (RAM and Flash) installed on your router, see Memory Requirements.

    Tips:

    • If you want to keep the same features as the version that is currently running on your router, but don't know which feature set you are using, enter the show version command on your router and paste it on the Output Interpreter (registered customers only) tool to find out. It is important to check for feature support, especially if you plan to use recent software features.

    • If you need to upgrade the Cisco IOS software image to a new version or feature set, see How to Choose a Cisco IOS Software Release for more information.

  3. If you determine that a Cisco IOS software upgrade is required, follow the Software Installation and Upgrade Procedure for the Cisco AS5200 or Software Installation and Upgrade Procedure for the AS5300.

Error Messages

The system software sends the error messages to the console (and, optionally, to a logging server on another system) during operation. Not all system error messages indicate problems with your system. Some are purely informational, and others may help diagnose problems with communications lines, internal hardware, or the system software. Also, some error messages appear when the system crashes.

Refer the document Cisco IOS System Error Messages for more information in Cisco IOS System Error Messages.

The Error Message Decoder (registered customers only) tool allows you to check the meaning of an error message, provides a recommended action (if needed) and, if available, a link to a document that provides extensive troubleshooting information about that error message.

Error messages appear on the console of Cisco products, usually in the following form:

%XXX-n-YYYY : [text]

Here is an example error message:

Router# %SYS-2-MALLOCFAIL: Memory allocation of [dec] bytes failed from [hex], 
pool [chars], alignment [dec]  

Identifying the Issue

Since the problem encountered may be caused by many factors such as hardware, software, cables, telco, configuration, and so on, it is important that you isolate and verify each option. This section describes some commonly seen symptoms and the appropriate resolution procedure.

Capturing Information

In order to determine what is causing the issue, the first step is to capture as much information about the problem as possible. The following information is essential in determining the cause of the problem:

If you have the output of a show command from your Cisco device (including show tech-support ), you can use to display potential issues and fixes. To use , you must be a registered customer, be logged in, and have JavaScript enabled.

Misleading Symptoms

There are a few issues that can be misinterpreted as hardware problems, when in fact they are not. Some of the more common issues are when the router stops responding or "hangs", or a failure occurs with a new hardware installation. Consult the following for proper explanation and troubleshooting steps for these commonly misinterpreted issues.

Symptom Recommended Action
No LEDs are on after powering on the router. Check if the power supply is plugged in firmly. If that does not resolve the issue, replace the power supply. If the problem persists, replace the router.
LEDs are on after powering on the router, but there is nothing on the console. Follow these steps to connect the PC or terminal to the router:
  1. Be sure that your PC or terminal emulation software is configured to communicate with the switch using hardware flow control. Refer to Applying Correct Terminal Emulator Settings for Console Connections for more information.
  2. Configure the baud rate and character format of the PC or terminal to match these console port default characteristics: 9600 baud, 8 data bits, 1 stop bit, No parity.
  3. Using the supplied rollover cable, insert the RJ-45 connector into the console port.
  4. Attach the supplied RJ-45-to-DB-9 female DTE adapter to a PC, or attach an appropriate adapter to the terminal.
  5. Insert the other end of the supplied rollover cable in the attached adapter. Refer to the Cabling Guide for Console and AUX Ports for more information on Cables and Connectors.
  6. Run the terminal emulation program if you are using a PC or a terminal.
If the above procedure doesn't help, verify that the equipment used for connecting to the console is operating properly. You can do this by connecting to a known good router to check your console equipment. If the equipment is successfully tested, but the problem remains, replace the router.
Router boots in ROMmon; no error messages on the console. Set the configuration register to 0x2102 and reload the router:
rommon 1 > confreg 0x2102
rommon 2 > reset 
If the router remains in ROMmon, follow the procedure described in the following documents:
Router displays the System Bootstrap version and hangs at that point or falls into a booting loop:
ROM: 
System Bootstrap, 
Version 12.0(2)XD1, 
EARLY DEPLOYMENT 
RELEASE SOFTWARE (fc1) 
Copyright (c) 1994-1996 
by cisco Systems, Inc. 
The memory could be mis-seated. First try to reseat (remove and then reinsert) the single inline memory module (SIMM) memory using proper electrostatic protection. If the router still does not boot, replace the router. Here is information on the location of the Dynamic RAM (DRAM) SIMMs for the AS5200 and the AS5300.
Router is running normally and then unexpectedly hangs or stops responding. A hang is when the router no longer accepts any commands or keystrokes. In other words, the console screen hangs after a certain point. Hangs are not necessarily hardware issues and most of the time, they are software issue. If your router is experiencing a router hang, refer to Troubleshooting Router Hangs.
Router boots in boot mode; no error messages on the console. Set the configuration register to 0x2102 and reload the router. There is no need to save the running configuration before reloading:
router(boot)#configure terminal
Enter configuration commands, one per line.  
End with CNTL/Z. 
router(boot)(config)#config-register 0x2102 
router
(boot)(config)#end 
router(boot)#reload 
System configuration has been modified. 
Save? [yes/no]: no 
Proceed with reload? [confirm] 

Note: The config-register command is the only Cisco IOS software configuration command that doesn't get saved in NVRAM. It changes the configuration register immediately, but this only takes effect during the next boot.

Router boots in boot mode, with the following messages on the console:
device does 
not contain a valid 
magic number 
boot: 
cannot open "flash:" 
boot: 
cannot determine 
first file name on 
device "flash:" 
The Flash is empty or the filesystem is corrupted. Copy a valid image on the Flash, and while copying, you will be prompted to erase the old image on the Flash (if one exists). Then reload the router. See the Software Installation and Upgrade Procedures for the AS5200 and AS5300 for instructions on how to copy a valid image onto the flash.
New network module is not recognized. Use the Software Advisor tool (registered customers only) to verify that your IOS software version supports the new card or module.

Boot errors can be a result of hardware not being supported by the Cisco IOS software version that is installed on the router. Use the Software Advisor (registered customers only) tool to make sure that your router is running the minimum required Cisco IOS software version that supports your hardware.

Packet Loss

Packet losses caused by hardware problems are fairly easy to identify. The following section uses the output of the show interfaces command to identify packet losses.

Cyclic Redundancy Check (CRC) and Frame Errors

If cyclic redundancy check errors or frame errors are constantly increasing on the interface, this usually indicates a hardware problem.

router#show interface ethernet 0 
     Ethernet0/0 is up, line protocol is up 
     ... 
          121 input errors, 102 CRC, 19 frame, 0 overrun, 0 ignored 

An exception to this is when CRC and frame errors are found on channelized interfaces, since they can indicate clocking problems as well. The fault that is causing the errors can be anywhere between two connected interfaces, such as on cables, intermediate devices, or on interfaces themselves. Troubleshooting techniques differ slightly for different interface types:

Serial Interfaces

Consult the Troubleshooting Serial Line Input Errors section of the Troubleshooting Serial Line Problems document.

Refer to the document T1 Troubleshooting Flowchart for troubleshooting T1 lines.

Ethernet Interfaces

For Ethernet interfaces, troubleshooting differs between a shared environment (devices connected through a hub) and a switched environment (devices connected to a switch).

In a switched environment, there are only five components that could cause the error:

  • cable

  • local interface (port)

  • remote interface (port)

  • speed

  • duplex mismatch

Consequently, the troubleshooting steps are simple. For example, if a router is connected to a switch, the troubleshooting steps are as follows:

  1. Replace the cable.

  2. If this does not solve the problem, try another port on the switch.

  3. If the problem still persists, replace the specific hardware.

In a shared environment, the source of the problem is harder to find. Every piece of hardware that makes up the shared segment can be the cause. You therefore have to test all components (cables, connectors, and so on) one by one.

Ignored Packets

router#show interface ethernet 0 
      Ethernet0/0 is up, line protocol is up 
      ... 
       21 input errors, 0 CRC, 0 frame, 0 overrun, 21 ignored 

Packets are ignored if there are no free buffers to accept the new packet. This can happen if the router is overloaded with traffic, but can also happen if the interface is faulty. If "ignores" are present on all interfaces, then the router is probably overloaded with traffic, or doesn't have sufficient free buffers in the pool that matches the maximum transmission unit (MTU) on interfaces. In the latter case, an increment of the ignored counter is followed by an increment of the no buffer counter:

router#show interfaces serial 0 
     ... 
      1567 packets input, 0 bytes, 22 no buffer 
      22 input errors, 0 CRC, 0 frame, 0 overrun, 22 ignored, 0 abort 

You may also see an increase in the buffer failures counter (using the command show buffers ) in the pool that matches the MTU size:

router#show buffers 
     ... 
     Big buffers, 1524 bytes (total 50, permanent 50): 
          50 in free list (5 min, 150 max allowed) 
          3066 hits, 189 misses, 0 trims, 24 created 
          12 failures (0 no memory)

The number of preconfigured permanent, free, and maximum allowed buffers may not be completely compatible for every environment. You can read more about this and how to avoid it in the Buffer Tuning document.

If "ignores" are only increasing on one interface, are not followed by an increment of the no buffer counter, and the interface is not heavily loaded, then this interface could be faulty. In that case, capture the output of the show tech-support command and contact the Technical Assitance Center (TAC). The load on the interface can be viewed in the output of the show interfaces command:

router#show interfaces serial 0 
     ... 
          reliability 255/255, txload 100/255, rxload 122/255 

Input and Output Queue Drops

Input queue drops are never caused by hardware problems. Output queue drops may be caused by a hardware problem only if the output queue is constantly full and no packets are being sent out of the interface. You can read more about these kinds of drops in Troubleshooting Input Queue Drops and Output Queue Drops.

Troubleshooting Crashes

Routers may reboot or reload for various reasons, some of which may be due to hardware failure. Refer to the examples below for some of the most common symptoms resulting from a potential hardware issue on the AS5200 and AS5300 series routers, and click on the hyperlinks for each symptom to find the solution.

First, check if the router is rebooting or is in a continuous loop. If the router reloads and returns to normal operation, it is rebooting. There is no defined time frame as to when or whether the router will reboot. The router can experience these reboots within periods as short as two to three minutes of being operational (meaning the router is passing traffic and you can log into it or gain access) or longer periods such as one or two weeks. If your router experiences a reboot, please see the Router Reboot/Reload section to troubleshoot the issue.

If the router is in a continuous or boot loop, you cannot gain access to the router. A router experiences a continuous loop when it repeatedly cycles through the boot process and is unrecoverable. If your router is experiencing a continuous or boot loop, please see the Continuous Loop section to troubleshoot the issue.

Router Reboot/Reload

A router can reboot or reload for various reasons. When a router reboots, it returns to a normal state, but could possibly reboot again. A normal state means that the router passes traffic, or is functional, and that you can gain access to the router. See the following examples of a router reboot and some reasons why it might occur. If you are experiencing one of these issues, click on the link to access a troubleshooting guide for that particular issue. To check why the router rebooted, issue the show version command and consult the output (see the examples below).

Router# show version 

	 Router uptime is 2 weeks, 19 hours, 22 minutes 
  System returned to ROM by power-on

Continuous Loop

A hardware issue may also cause the router to experience a continuous loop. During a continuous loop, you are unable to gain access to the router (by logging in to enable mode, for example) and the router continues to give scrolling error messages until it is powered off. Refer to the examples and troubleshooting steps listed below to determine which piece of hardware is causing the continuous loop.

  • Bus Error Exception

    Example:

    *** System received a Bus Error exception ***
    Access address = 0x3c210040
    signal= 0xa, code= 0x1c, context= 0x60e632f0
    PC = 0x6037668c, Cause = 0xc20, Status Reg = 0x34008002
    
    
    ** TLB (Load/Fetch) Exception ***
    Access address = 0x4
    PC = 0xbfc165f8, Cause = 0x8, Status Reg = 0x30408403
    monitor: command "boot" aborted due to exception
  • Write Bus Error Interrupt

    Example:

    *** System received a Write Bus Error Interrupt ***
    Signal = 0x15, code= 0x0, context= 0x6036f580
    PC = 0x600f45d8, Cause = 0x20, Status Reg = 0x34008002http

Troubleshooting Steps for Bus Error Exception and Continuous Loops

The following flowchart will help you in troubleshooting the bus error exception and continuous loops.

hwts_2600_16138.gif

** If the router does not experience the continuous loop after trying out with all the Network Modules, it could have been caused by a mis-seated network module. It is recommended that you monitor the router for 24 hours to be sure that the router continues to function without experiencing the issue again.

LEDs

Refer to the following documents for more information on the LEDs and what they indicate :

Pay attention to the chassis LEDs as well as the LEDs on the modules.

Cabling

Cabling Specifications lists the various cables (and pinouts) that can be used on the AS5200 or AS5300.

Verify that the cable adheres to standards and has the correct pinouts. It is always advisable to swap cables during troubleshooting so as to eliminate it as a cause.

Memory Replacement and Upgrade

A common cause for hardware failure is memory issues. Often the router fails to boot after a memory upgrade. This could be caused by a misseated memory module or if you have not met certain conditions for the size of the memory. For example, on the AS5300, there are 2 slots for main memory DRAM SIMMs, however the capacity of the SIMMs in both sockets must match. The following diagram illustrates the location of the memory on the AS5300.

as5300_17881.gif

It is recommended that you try reseating or even replacing the memory to see if that alleviates the problem. You should also verify that the memory is installed in the correct slot. For example, ensure that Boot Flash memory is not installed in the System Flash slot and vice versa. Refer to the diagram above for more information on the memory slots.

For more information on memory upgrades, refer to the following documents:

Troubleshooting the T1/E1 Modules

If you encounter circuit issues on the Access Server, the issue could be due to a telco problem, cabling or a hardware issue.

First, verify that the Board OK (OK) LED is on, which indicates that the card has passed initial power up diagnostics tests and is operating normally. If it is not "OK", then swap out the module and insert a functioning module from another slot or router. This can help determine if the problem follows the slot or the module. If the problem follows the module, replace it. However, if the problem follows the slot, then the slot is most likely defective and the chassis should be replaced (or the slot should be left empty).

To conclusively determine whether a specific T1/E1 port on the card is defective, perform a Hard Plug Loopback test. Refer to the following documents for more information on performing the loopback test:

If the port successfully passes a hard plug loopback test, then you can safely eliminate the T1/E1 Card/Port as the cause of the problem. However, if an individual port on the module fails the loopback test, then you must use another port or replace the entire module.

Optional Testing: You can also perform a Bit Error Rate Test (BERT) with the assistance of the telco. This provides a comprehensive test of the integrity of the circuit.

Troubleshooting the Modem Modules

Refer to the document Troubleshooting MICA Modem Hardware Issues on the AS5200 and AS5300 to troubleshoot MICA modems.

Summary

Once you are able to identify the piece of hardware that is causing the issue, open a service request with the Cisco Technical Assistance Center (TAC) via the web (registered customers only) and request a Return Materials Authorization (RMA) for the part that is causing the issue if the router is covered by a valid warranty or service contract.

When hardware replacement is indicated after troubleshooting, use one of the following options:

  • If you have a hardware support contract directly with Cisco for this part, use the Service Order Submit Tool (registered customers only) to request a replacement part directly.

  • For warranty service, contact the Cisco Technical Assistance Center (TAC) online using the TAC Service Request Tool (registered customers only) .

  • If your product is not covered by contract or warranty, contact your Cisco partner or reseller to request a replacement part for the hardware component that is causing the issue.

If you were not successful in identifying which piece of hardware is causing the issue, open a TAC service request (registered customers only) , attach all relevant information such as troubleshooting logs and troubleshooting steps taken, and an engineer can assist you in further troubleshooting the issue.

Information to Collect if You Open a TAC Service Request

If you have identified a component that needs to be replaced, contact your Cisco partner or reseller to request a replacement for the hardware component that is causing the issue. If you have a support contract directly with Cisco, use the TAC Service Request Tool (registered customers only) to open a TAC service request and request a hardware replacement. Make sure you attach the following information when you open a service request:
  • Console captures showing the error messages
  • Console captures showing the troubleshooting steps taken and the boot sequence during each step
  • The hardware component that failed and the serial number for the chassis
  • Troubleshooting logs
  • Output from the show tech-support command

Related Information

Updated: Mar 09, 2009
Document ID: 17881