Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps you to troubleshoot common hardware issues on Cisco AS5200 and Cisco AS5300 series routers, and provides pointers for identifying whether or not there is a hardware fault. This document does not cover any software-related failures except for those that are often mistaken for hardware issues. For more information on identifying the modules and controller cards installed on the AS5200 and AS5300 Series, see Identifying the Controllers and Modem Hardware On AS5xxx Platforms.
Whenever you install a new card, module, or Cisco IOS software image, it is important to verify that the access server has enough memory, and that the hardware and software are compatible with the features you wish to use.
Perform the following recommended steps to check for hardware-software compatibility and memory requirements:
The "Software Support for Hardware" section (available for registered customers only) helps you to verify whether the modules and cards installed are supported by the desired Cisco IOS software version.
The "Software Support for Features" section (available for registered customers only) helps you to determine the Cisco IOS software image needed by choosing the types of features you wish to implement.
Use the Download Software Area (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. To determine the amount of memory (RAM and Flash) installed on your router, see Memory Requirements.
If you want to keep the same features as the version that is currently running on your router, but don't know which feature set you are using, enter the show versioncommand on your router and paste it on the Output Interpreter (registered customers only) tool to find out. It is important to check for feature support, especially if you plan to use recent software features.
The system software sends the error messages to the console (and, optionally, to a logging server on another system) during operation. Not all system error messages indicate problems with your system. Some are purely informational, and others may help diagnose problems with communications lines, internal hardware, or the system software. Also, some error messages appear when the system crashes.
The Error Message Decoder (registered customers only) tool allows you to check the meaning of an error message, provides a recommended action (if needed) and, if available, a link to a document that provides extensive troubleshooting information about that error message.
Error messages appear on the console of Cisco products, usually in the following form:
%XXX-n-YYYY : [text]
Here is an example error message:
Router# %SYS-2-MALLOCFAIL: Memory allocation of [dec] bytes failed from [hex],
pool [chars], alignment [dec]
Since the problem encountered may be caused by many factors such as hardware, software, cables, telco, configuration, and so on, it is important that you isolate and verify each option. This section describes some commonly seen symptoms and the appropriate resolution procedure.
In order to determine what is causing the issue, the first step is to capture as much information about the problem as possible. The following information is essential in determining the cause of the problem:
show tech-support command output - The show tech-support command is a compilation of many different commands including show version, show running-config, and show stacks. TAC engineers usually ask for this information to troubleshoot hardware issues. It is important to collect the show tech-support information before doing a reload or power-cycle as these actions can cause all information about the problem to be lost.
Complete bootup sequence if the router experiences boot errors.
There are a few issues that can be misinterpreted as hardware problems, when in fact they are not. Some of the more common issues are when the router stops responding or "hangs", or a failure occurs with a new hardware installation. Consult the following for proper explanation and troubleshooting steps for these commonly misinterpreted issues.
No LEDs are on after powering on the router.
Check if the power supply is plugged in firmly. If that does not resolve the issue, replace the power supply. If the problem persists, replace the router.
LEDs are on after powering on the router, but there is nothing on the console.
Follow these steps to connect the PC or terminal to the router:
Run the terminal emulation program if you are using a PC or a terminal.
If the above procedure doesn't help, verify that the equipment used for connecting to the console is operating properly. You can do this by connecting to a known good router to check your console equipment. If the equipment is successfully tested, but the problem remains, replace the router.
Router boots in ROMmon; no error messages on the console.
Set the configuration register to 0x2102 and reload the router:
rommon 1 > confreg 0x2102
rommon 2 > reset
If the router remains in ROMmon, follow the procedure described in the following documents:
Router displays the System Bootstrap version and hangs at that point or falls into a booting loop:
RELEASE SOFTWARE (fc1)
Copyright (c) 1994-1996
by cisco Systems, Inc.
The memory could be mis-seated. First try to reseat (remove and then reinsert) the single inline memory module (SIMM) memory using proper electrostatic protection. If the router still does not boot, replace the router. Here is information on the location of the Dynamic RAM (DRAM) SIMMs for the AS5200 and the AS5300.
Router is running normally and then unexpectedly hangs or stops responding.
A hang is when the router no longer accepts any commands or keystrokes. In other words, the console screen hangs after a certain point. Hangs are not necessarily hardware issues and most of the time, they are software issue. If your router is experiencing a router hang, refer to Troubleshooting Router Hangs.
Router boots in boot mode; no error messages on the console.
Set the configuration register to 0x2102 and reload the router. There is no need to save the running configuration before reloading:
Enter configuration commands, one per line.
End with CNTL/Z.
System configuration has been modified.
Save? [yes/no]: no
Proceed with reload? [confirm]
Note: The config-register command is the only Cisco IOS software configuration command that doesn't get saved in NVRAM. It changes the configuration register immediately, but this only takes effect during the next boot.
Router boots in boot mode, with the following messages on the console:
not contain a valid
cannot open "flash:"
first file name on
The Flash is empty or the filesystem is corrupted. Copy a valid image on the Flash, and while copying, you will be prompted to erase the old image on the Flash (if one exists). Then reload the router. See the Software Installation and Upgrade Procedures for the AS5200 and AS5300 for instructions on how to copy a valid image onto the flash.
Boot errors can be a result of hardware not being supported by the Cisco IOS software version that is installed on the router. Use the Software Advisor (registered customers only) tool to make sure that your router is running the minimum required Cisco IOS software version that supports your hardware.
If cyclic redundancy check errors or frame errors are constantly increasing on the interface, this usually indicates a hardware problem.
router#show interface ethernet 0
Ethernet0/0 is up, line protocol is up
121 input errors, 102 CRC, 19 frame, 0 overrun, 0 ignored
An exception to this is when CRC and frame errors are found on channelized interfaces, since they can indicate clocking problems as well. The fault that is causing the errors can be anywhere between two connected interfaces, such as on cables, intermediate devices, or on interfaces themselves. Troubleshooting techniques differ slightly for different interface types:
For Ethernet interfaces, troubleshooting differs between a shared environment (devices connected through a hub) and a switched environment (devices connected to a switch).
In a switched environment, there are only five components that could cause the error:
local interface (port)
remote interface (port)
Consequently, the troubleshooting steps are simple. For example, if a router is connected to a switch, the troubleshooting steps are as follows:
Replace the cable.
If this does not solve the problem, try another port on the switch.
If the problem still persists, replace the specific hardware.
In a shared environment, the source of the problem is harder to find. Every piece of hardware that makes up the shared segment can be the cause. You therefore have to test all components (cables, connectors, and so on) one by one.
router#show interface ethernet 0
Ethernet0/0 is up, line protocol is up
21 input errors, 0 CRC, 0 frame, 0 overrun, 21 ignored
Packets are ignored if there are no free buffers to accept the new packet. This can happen if the router is overloaded with traffic, but can also happen if the interface is faulty. If "ignores" are present on all interfaces, then the router is probably overloaded with traffic, or doesn't have sufficient free buffers in the pool that matches the maximum transmission unit (MTU) on interfaces. In the latter case, an increment of the ignored counter is followed by an increment of the no buffer counter:
You may also see an increase in the buffer failures counter (using the command show buffers) in the pool that matches the MTU size:
Big buffers, 1524 bytes (total 50, permanent 50):
50 in free list (5 min, 150 max allowed)
3066 hits, 189 misses, 0 trims, 24 created
12 failures (0 no memory)
The number of preconfigured permanent, free, and maximum allowed buffers may not be completely compatible for every environment. You can read more about this and how to avoid it in the Buffer Tuning document.
If "ignores" are only increasing on one interface, are not followed by an increment of the no buffer counter, and the interface is not heavily loaded, then this interface could be faulty. In that case, capture the output of theshow tech-support command and contact the Technical Assitance Center (TAC). The load on the interface can be viewed in the output of the show interfaces command:
Input queue drops are never caused by hardware problems. Output queue drops may be caused by a hardware problem only if the output queue is constantly full and no packets are being sent out of the interface. You can read more about these kinds of drops in Troubleshooting Input Queue Drops and Output Queue Drops.
Routers may reboot or reload for various reasons, some of which may be due to hardware failure. Refer to the examples below for some of the most common symptoms resulting from a potential hardware issue on the AS5200 and AS5300 series routers, and click on the hyperlinks for each symptom to find the solution.
First, check if the router is rebooting or is in a continuous loop. If the router reloads and returns to normal operation, it is rebooting. There is no defined time frame as to when or whether the router will reboot. The router can experience these reboots within periods as short as two to three minutes of being operational (meaning the router is passing traffic and you can log into it or gain access) or longer periods such as one or two weeks. If your router experiences a reboot, please see the Router Reboot/Reload section to troubleshoot the issue.
If the router is in a continuous or boot loop, you cannot gain access to the router. A router experiences a continuous loop when it repeatedly cycles through the boot process and is unrecoverable. If your router is experiencing a continuous or boot loop, please see the Continuous Loop section to troubleshoot the issue.
A router can reboot or reload for various reasons. When a router reboots, it returns to a normal state, but could possibly reboot again. A normal state means that the router passes traffic, or is functional, and that you can gain access to the router. See the following examples of a router reboot and some reasons why it might occur. If you are experiencing one of these issues, click on the link to access a troubleshooting guide for that particular issue. To check why the router rebooted, issue the show version command and consult the output (see the examples below).
Router# show version
Router uptime is 2 weeks, 19 hours, 22 minutes
System returned to ROM by power-on
A hardware issue may also cause the router to experience a continuous loop. During a continuous loop, you are unable to gain access to the router (by logging in to enable mode, for example) and the router continues to give scrolling error messages until it is powered off. Refer to the examples and troubleshooting steps listed below to determine which piece of hardware is causing the continuous loop.
*** System received a Bus Error exception ***
Access address = 0x3c210040
signal= 0xa, code= 0x1c, context= 0x60e632f0
PC = 0x6037668c, Cause = 0xc20, Status Reg = 0x34008002
** TLB (Load/Fetch) Exception ***
Access address = 0x4
PC = 0xbfc165f8, Cause = 0x8, Status Reg = 0x30408403
monitor: command "boot" aborted due to exception
The following flowchart will help you in troubleshooting the bus error exception and continuous loops.
** If the router does not experience the continuous loop after trying out with all the Network Modules, it could have been caused by a mis-seated network module. It is recommended that you monitor the router for 24 hours to be sure that the router continues to function without experiencing the issue again.
A common cause for hardware failure is memory issues. Often the router fails to boot after a memory upgrade. This could be caused by a misseated memory module or if you have not met certain conditions for the size of the memory. For example, on the AS5300, there are 2 slots for main memory DRAM SIMMs, however the capacity of the SIMMs in both sockets must match. The following diagram illustrates the location of the memory on the AS5300.
It is recommended that you try reseating or even replacing the memory to see if that alleviates the problem. You should also verify that the memory is installed in the correct slot. For example, ensure that Boot Flash memory is not installed in the System Flash slot and vice versa. Refer to the diagram above for more information on the memory slots.
For more information on memory upgrades, refer to the following documents:
If you encounter circuit issues on the Access Server, the issue could be due to a telco problem, cabling or a hardware issue.
First, verify that the Board OK (OK) LED is on, which indicates that the card has passed initial power up diagnostics tests and is operating normally. If it is not "OK", then swap out the module and insert a functioning module from another slot or router. This can help determine if the problem follows the slot or the module. If the problem follows the module, replace it. However, if the problem follows the slot, then the slot is most likely defective and the chassis should be replaced (or the slot should be left empty).
To conclusively determine whether a specific T1/E1 port on the card is defective, perform a Hard Plug Loopback test. Refer to the following documents for more information on performing the loopback test:
If the port successfully passes a hard plug loopback test, then you can safely eliminate the T1/E1 Card/Port as the cause of the problem. However, if an individual port on the module fails the loopback test, then you must use another port or replace the entire module.
Optional Testing: You can also perform a Bit Error Rate Test (BERT) with the assistance of the telco. This provides a comprehensive test of the integrity of the circuit.
If your product is not covered by contract or warranty, contact your Cisco partner or reseller to request a replacement part for the hardware component that is causing the issue.
If you were not successful in identifying which piece of hardware is causing the issue, open a TAC service request (registered customers only) , attach all relevant information such as troubleshooting logs and troubleshooting steps taken, and an engineer can assist you in further troubleshooting the issue.
Information to Collect if You Open a TAC Service Request
If you have identified a component that needs to be replaced, contact your Cisco partner or reseller to request a replacement for the hardware component that is causing the issue. If you have a support contract directly with Cisco, use the TAC Service Request Tool (registered customers only) to open a TAC service request and request a hardware replacement. Make sure you attach the following information when you open a service request:
Console captures showing the error messages
Console captures showing the troubleshooting steps taken and the boot sequence during each step
The hardware component that failed and the serial number for the chassis