Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps you to troubleshoot common hardware issues on Cisco AS5200 and Cisco AS5300 series routers, and provides pointers for identifying whether or not there is a hardware fault. This document does not cover any software-related failures except for those that are often mistaken for hardware issues. For more information on identifying the modules and controller cards installed on the AS5200 and AS5300 Series, see Identifying the Controllers and Modem Hardware On AS5xxx Platforms.
We recommend that you read Troubleshooting Router Crashes before proceeding with this document.
The information in this document covers all AS5200 and AS5300 series routers running any version of Cisco IOS® software. This document does not cover AS5350 series routers.
For more information on document conventions, see the Cisco Technical Tips Conventions.
Whenever you install a new card, module, or Cisco IOS software image, it is important to verify that the access server has enough memory, and that the hardware and software are compatible with the features you wish to use.
Perform the following recommended steps to check for hardware-software compatibility and memory requirements:
The "Software Support for Hardware" section (available for registered customers only) helps you to verify whether the modules and cards installed are supported by the desired Cisco IOS software version.
The "Software Support for Features" section (available for registered customers only) helps you to determine the Cisco IOS software image needed by choosing the types of features you wish to implement.
Use the Download Software Area (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. To determine the amount of memory (RAM and Flash) installed on your router, see Memory Requirements.
If you want to keep the same features as the version that is currently running on your router, but don't know which feature set you are using, enter the show version command on your router and paste it on the Output Interpreter (registered customers only) tool to find out. It is important to check for feature support, especially if you plan to use recent software features.
If you need to upgrade the Cisco IOS software image to a new version or feature set, see How to Choose a Cisco IOS Software Release for more information.
If you determine that a Cisco IOS software upgrade is required, follow the Software Installation and Upgrade Procedure for the Cisco AS5200 or Software Installation and Upgrade Procedure for the AS5300.
The system software sends the error messages to the console (and, optionally, to a logging server on another system) during operation. Not all system error messages indicate problems with your system. Some are purely informational, and others may help diagnose problems with communications lines, internal hardware, or the system software. Also, some error messages appear when the system crashes.
Refer the document Cisco IOS System Error Messages for more information in Cisco IOS System Error Messages.
The Error Message Decoder (registered customers only) tool allows you to check the meaning of an error message, provides a recommended action (if needed) and, if available, a link to a document that provides extensive troubleshooting information about that error message.
Error messages appear on the console of Cisco products, usually in the following form:
%XXX-n-YYYY : [text]
Here is an example error message:
Router# %SYS-2-MALLOCFAIL: Memory allocation of [dec] bytes failed from [hex], pool [chars], alignment [dec]
Since the problem encountered may be caused by many factors such as hardware, software, cables, telco, configuration, and so on, it is important that you isolate and verify each option. This section describes some commonly seen symptoms and the appropriate resolution procedure.
In order to determine what is causing the issue, the first step is to capture as much information about the problem as possible. The following information is essential in determining the cause of the problem:
Console logs (for more information, see Applying Correct Terminal Emulator Settings for Console Connections)
Syslog information - If the router is set up to send logs to a syslog server, you may be able to obtain information on what happened.
For details, refer to the How to Configure Cisco Devices for Syslog section of the document Resource Manager Essentials and Syslog Analysis: How-To.
show tech-support command output - The show tech-support command is a compilation of many different commands including show version , show running-config , and show stacks . TAC engineers usually ask for this information to troubleshoot hardware issues. It is important to collect the show tech-support information before doing a reload or power-cycle as these actions can cause all information about the problem to be lost.
Complete bootup sequence if the router experiences boot errors.
There are a few issues that can be misinterpreted as hardware problems, when in fact they are not. Some of the more common issues are when the router stops responding or "hangs", or a failure occurs with a new hardware installation. Consult the following for proper explanation and troubleshooting steps for these commonly misinterpreted issues.
|No LEDs are on after powering on the router.||Check if the power supply is plugged in firmly. If that does not resolve the issue, replace the power supply. If the problem persists, replace the router.|
|LEDs are on after powering on the router, but there is nothing on the console.||
Follow these steps to connect the PC or terminal to the router:
|Router boots in ROMmon; no error messages on the console.||
Set the configuration register to 0x2102 and reload the router:
rommon 1 > confreg 0x2102 rommon 2 > resetIf the router remains in ROMmon, follow the procedure described in the following documents:
Router displays the System Bootstrap version and hangs at that
point or falls into a booting loop:
ROM: System Bootstrap, Version 12.0(2)XD1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) Copyright (c) 1994-1996 by cisco Systems, Inc.
|The memory could be mis-seated. First try to reseat (remove and then reinsert) the single inline memory module (SIMM) memory using proper electrostatic protection. If the router still does not boot, replace the router. Here is information on the location of the Dynamic RAM (DRAM) SIMMs for the AS5200 and the AS5300.|
|Router is running normally and then unexpectedly hangs or stops responding.||A hang is when the router no longer accepts any commands or keystrokes. In other words, the console screen hangs after a certain point. Hangs are not necessarily hardware issues and most of the time, they are software issue. If your router is experiencing a router hang, refer to Troubleshooting Router Hangs.|
|Router boots in boot mode; no error messages on the console.||
Set the configuration register to 0x2102 and reload the router.
There is no need to save the running configuration before reloading:
router(boot)#configure terminal Enter configuration commands, one per line. End with CNTL/Z. router(boot)(config)#config-register 0x2102 router (boot)(config)#end router(boot)#reload System configuration has been modified. Save? [yes/no]: no Proceed with reload? [confirm]
Note: The config-register command is the only Cisco IOS software configuration command that doesn't get saved in NVRAM. It changes the configuration register immediately, but this only takes effect during the next boot.
Router boots in boot mode, with the following messages on the
device does not contain a valid magic number boot: cannot open "flash:" boot: cannot determine first file name on device "flash:"
|The Flash is empty or the filesystem is corrupted. Copy a valid image on the Flash, and while copying, you will be prompted to erase the old image on the Flash (if one exists). Then reload the router. See the Software Installation and Upgrade Procedures for the AS5200 and AS5300 for instructions on how to copy a valid image onto the flash.|
|New network module is not recognized.||Use the Software Advisor tool (registered customers only) to verify that your IOS software version supports the new card or module.|
Boot errors can be a result of hardware not being supported by the Cisco IOS software version that is installed on the router. Use the Software Advisor (registered customers only) tool to make sure that your router is running the minimum required Cisco IOS software version that supports your hardware.
Packet losses caused by hardware problems are fairly easy to identify. The following section uses the output of the show interfaces command to identify packet losses.
If cyclic redundancy check errors or frame errors are constantly increasing on the interface, this usually indicates a hardware problem.
router#show interface ethernet 0 Ethernet0/0 is up, line protocol is up ... 121 input errors, 102 CRC, 19 frame, 0 overrun, 0 ignored
An exception to this is when CRC and frame errors are found on channelized interfaces, since they can indicate clocking problems as well. The fault that is causing the errors can be anywhere between two connected interfaces, such as on cables, intermediate devices, or on interfaces themselves. Troubleshooting techniques differ slightly for different interface types:
Consult the Troubleshooting Serial Line Input Errors section of the Troubleshooting Serial Line Problems document.
Refer to the document T1 Troubleshooting Flowchart for troubleshooting T1 lines.
For Ethernet interfaces, troubleshooting differs between a shared environment (devices connected through a hub) and a switched environment (devices connected to a switch).
In a switched environment, there are only five components that could cause the error:
local interface (port)
remote interface (port)
Consequently, the troubleshooting steps are simple. For example, if a router is connected to a switch, the troubleshooting steps are as follows:
Replace the cable.
If this does not solve the problem, try another port on the switch.
If the problem still persists, replace the specific hardware.
In a shared environment, the source of the problem is harder to find. Every piece of hardware that makes up the shared segment can be the cause. You therefore have to test all components (cables, connectors, and so on) one by one.
router#show interface ethernet 0 Ethernet0/0 is up, line protocol is up ... 21 input errors, 0 CRC, 0 frame, 0 overrun, 21 ignored
Packets are ignored if there are no free buffers to accept the new packet. This can happen if the router is overloaded with traffic, but can also happen if the interface is faulty. If "ignores" are present on all interfaces, then the router is probably overloaded with traffic, or doesn't have sufficient free buffers in the pool that matches the maximum transmission unit (MTU) on interfaces. In the latter case, an increment of the ignored counter is followed by an increment of the no buffer counter:
router#show interfaces serial 0 ... 1567 packets input, 0 bytes, 22 no buffer 22 input errors, 0 CRC, 0 frame, 0 overrun, 22 ignored, 0 abort
You may also see an increase in the buffer failures counter (using the command show buffers ) in the pool that matches the MTU size:
router#show buffers ... Big buffers, 1524 bytes (total 50, permanent 50): 50 in free list (5 min, 150 max allowed) 3066 hits, 189 misses, 0 trims, 24 created 12 failures (0 no memory)
The number of preconfigured permanent, free, and maximum allowed buffers may not be completely compatible for every environment. You can read more about this and how to avoid it in the Buffer Tuning document.
If "ignores" are only increasing on one interface, are not followed by an increment of the no buffer counter, and the interface is not heavily loaded, then this interface could be faulty. In that case, capture the output of the show tech-support command and contact the Technical Assitance Center (TAC). The load on the interface can be viewed in the output of the show interfaces command:
router#show interfaces serial 0 ... reliability 255/255, txload 100/255, rxload 122/255
Input queue drops are never caused by hardware problems. Output queue drops may be caused by a hardware problem only if the output queue is constantly full and no packets are being sent out of the interface. You can read more about these kinds of drops in Troubleshooting Input Queue Drops and Output Queue Drops.
Routers may reboot or reload for various reasons, some of which may be due to hardware failure. Refer to the examples below for some of the most common symptoms resulting from a potential hardware issue on the AS5200 and AS5300 series routers, and click on the hyperlinks for each symptom to find the solution.
First, check if the router is rebooting or is in a continuous loop. If the router reloads and returns to normal operation, it is rebooting. There is no defined time frame as to when or whether the router will reboot. The router can experience these reboots within periods as short as two to three minutes of being operational (meaning the router is passing traffic and you can log into it or gain access) or longer periods such as one or two weeks. If your router experiences a reboot, please see the Router Reboot/Reload section to troubleshoot the issue.
If the router is in a continuous or boot loop, you cannot gain access to the router. A router experiences a continuous loop when it repeatedly cycles through the boot process and is unrecoverable. If your router is experiencing a continuous or boot loop, please see the Continuous Loop section to troubleshoot the issue.
A router can reboot or reload for various reasons. When a router reboots, it returns to a normal state, but could possibly reboot again. A normal state means that the router passes traffic, or is functional, and that you can gain access to the router. See the following examples of a router reboot and some reasons why it might occur. If you are experiencing one of these issues, click on the link to access a troubleshooting guide for that particular issue. To check why the router rebooted, issue the show version command and consult the output (see the examples below).
Router# show version Router uptime is 2 weeks, 19 hours, 22 minutes System returned to ROM by power-on
System was restarted by processor memory parity error at PC 0x6014F7C0, address 0x0
*** Cache Error Exception *** Cache Err Reg = 0xa401a65a data reference, primary cache
*** Shared Memory Parity Error *** shared memory control register= 0xffdf error(s) reported for: NIM1 on byte(s): 0/1 2/3
System restarted by error - a bus error at PC 0x30EE546, address 0xBB4C4
System restarted by error - a Write Bus Error Interrupt, PC 0x6000FA64
A hardware issue may also cause the router to experience a continuous loop. During a continuous loop, you are unable to gain access to the router (by logging in to enable mode, for example) and the router continues to give scrolling error messages until it is powered off. Refer to the examples and troubleshooting steps listed below to determine which piece of hardware is causing the continuous loop.
*** System received a Bus Error exception *** Access address = 0x3c210040 signal= 0xa, code= 0x1c, context= 0x60e632f0 PC = 0x6037668c, Cause = 0xc20, Status Reg = 0x34008002 ** TLB (Load/Fetch) Exception *** Access address = 0x4 PC = 0xbfc165f8, Cause = 0x8, Status Reg = 0x30408403 monitor: command "boot" aborted due to exception
*** System received a Write Bus Error Interrupt *** Signal = 0x15, code= 0x0, context= 0x6036f580 PC = 0x600f45d8, Cause = 0x20, Status Reg = 0x34008002http
The following flowchart will help you in troubleshooting the bus error exception and continuous loops.
** If the router does not experience the continuous loop after trying out with all the Network Modules, it could have been caused by a mis-seated network module. It is recommended that you monitor the router for 24 hours to be sure that the router continues to function without experiencing the issue again.
Refer to the following documents for more information on the LEDs and what they indicate :
Pay attention to the chassis LEDs as well as the LEDs on the modules.
Cabling Specifications lists the various cables (and pinouts) that can be used on the AS5200 or AS5300.
Verify that the cable adheres to standards and has the correct pinouts. It is always advisable to swap cables during troubleshooting so as to eliminate it as a cause.
A common cause for hardware failure is memory issues. Often the router fails to boot after a memory upgrade. This could be caused by a misseated memory module or if you have not met certain conditions for the size of the memory. For example, on the AS5300, there are 2 slots for main memory DRAM SIMMs, however the capacity of the SIMMs in both sockets must match. The following diagram illustrates the location of the memory on the AS5300.
It is recommended that you try reseating or even replacing the memory to see if that alleviates the problem. You should also verify that the memory is installed in the correct slot. For example, ensure that Boot Flash memory is not installed in the System Flash slot and vice versa. Refer to the diagram above for more information on the memory slots.
For more information on memory upgrades, refer to the following documents:
If you encounter circuit issues on the Access Server, the issue could be due to a telco problem, cabling or a hardware issue.
First, verify that the Board OK (OK) LED is on, which indicates that the card has passed initial power up diagnostics tests and is operating normally. If it is not "OK", then swap out the module and insert a functioning module from another slot or router. This can help determine if the problem follows the slot or the module. If the problem follows the module, replace it. However, if the problem follows the slot, then the slot is most likely defective and the chassis should be replaced (or the slot should be left empty).
To conclusively determine whether a specific T1/E1 port on the card is defective, perform a Hard Plug Loopback test. Refer to the following documents for more information on performing the loopback test:
If the port successfully passes a hard plug loopback test, then you can safely eliminate the T1/E1 Card/Port as the cause of the problem. However, if an individual port on the module fails the loopback test, then you must use another port or replace the entire module.
Optional Testing: You can also perform a Bit Error Rate Test (BERT) with the assistance of the telco. This provides a comprehensive test of the integrity of the circuit.
Refer to the document Troubleshooting MICA Modem Hardware Issues on the AS5200 and AS5300 to troubleshoot MICA modems.
Once you are able to identify the piece of hardware that is causing the issue, open a service request with the Cisco Technical Assistance Center (TAC) via the web (registered customers only) and request a Return Materials Authorization (RMA) for the part that is causing the issue if the router is covered by a valid warranty or service contract.
When hardware replacement is indicated after troubleshooting, use one of the following options:
If your product is not covered by contract or warranty, contact your Cisco partner or reseller to request a replacement part for the hardware component that is causing the issue.
If you were not successful in identifying which piece of hardware is causing the issue, open a TAC service request (registered customers only) , attach all relevant information such as troubleshooting logs and troubleshooting steps taken, and an engineer can assist you in further troubleshooting the issue.
Information to Collect if You Open a TAC Service Request
|If you have identified a component that needs to be replaced, contact your Cisco partner or reseller to request a replacement for the hardware component that is causing the issue. If you have a support contract directly with Cisco, use the TAC Service Request Tool (registered customers only) to open a TAC service request and request a hardware replacement. Make sure you attach the following information when you open a service request:|
- Understanding Software-forced Crashes
- Troubleshooting Router Crashes
- System Message Guides
- Troubleshooting Router Hangs
- Hardware Installation for the AS5300
- Cisco AS5200 Documentation
- AS5200 Series Universal Access Servers Hardware Support Page
- AS5300 Access Server Hardware Support Page
- Dial Technology Support Page
- Technical Support & Documentation - Cisco Systems
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.