Document ID: 18676
Interactive: This document offers customized analysis of your Cisco device.
Contents
Introduction
Prequisites
Requirements
Components Used
Hardware-Software Compatibility and Memory Requirements
Background
Conventions
Identifying the Issue
Capturing Information
Misleading Symptoms
Step-by-Step Troubleshooting
Information to Collect if You Open a TAC Service Request
Related Information
Introduction
Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps troubleshoot potential hardware issues with the Cisco 7400 Series Router, and provides pointers for identifying a hardware failure.
Note: This document does not cover any software-related failures except for those that are often mistaken as hardware issues.
Prequisites
Requirements
We recommend that you review the following documents before proceeding with this document:
Components Used
The information in this document is not specific to any Cisco IOS® software release, but applies to all Cisco IOS software versions that run on the 7400 Series Router. This document covers troubleshooting on the 7400 Series Router for both the Back Bone (BB) and Customer Premise (CP) models.
Hardware-Software Compatibility and Memory Requirements
Whenever you install a new line card, module, or Cisco IOS software image, it is important to verify that the router has enough memory, and that the hardware and software are compatible with the features you wish to use.
Perform the following recommended steps to check for hardware-software compatibility and memory requirements:
-
Use the Software Advisor (registered customers only) tool to choose software for your network device.
Tips:
-
The Software Support for Hardware (registered customers only) section helps you verify whether the modules and cards installed on the router are supported by the desired Cisco IOS software version.
-
-
Use the IOS Upgrade Planner (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. To determine the amount of memory (RAM and Flash) installed on your router, refer to the Memory Requirements section of How to Choose a Cisco IOS Software Release.
Tips:
-
If you want to keep the same features as the version that is currently running on your router, but don't know which feature set you are using, enter the show version command on your router and paste it into the Output Interpreter
(registered customers only) to find out. It is important to check for feature support, especially if you plan to use recent software features. -
If you need to upgrade the Cisco IOS software image to a new version or feature set, refer to How to Choose a Cisco IOS Software Release for more information.
-
-
If you determine that a Cisco IOS software upgrade is required, follow the Software Installation and Upgrade Procedures for the Cisco 7400 Series Router.
Tip: For information on how to recover a Cisco 7400 series router stuck in ROMmon (rommon # > prompt), refer to the ROMmon Recovery Procedure.
Background
The Cisco 7400 Application Specific Router (ASR) combines the Network Services Engine (NSE) and Input/Output (I/O) controller card into a single unit. With a single Port Adapter (PA) slot and two fixed 10/100/1000 Ethernet ports, the 7400 takes up only one Rack Unit (RU) of space.
Conventions
For more information on document conventions, refer to the Cisco Technical Tips Conventions.
Identifying the Issue
The Cisco 7400 Series Router can reboot or reload for various reasons, several of which are potential hardware issues. Below are the most common symptoms resulting from a potential hardware issue seen on this platform. The troubleshooting steps for each symptom are listed in the Step-by-Step Troubleshooting section.
Capturing Information
In order to determine what is causing the issue, the first step is to capture as much information about the problem as possible. The following information is essential for determining the cause of the problem:
-
Console logs - For more information, refer to Applying Correct Terminal Emulator Settings for Console Connections.
-
Syslog information - If the router is set up to send logs to a syslog server, you may be able to obtain information on what happened. For details, refer to the How to Configure Cisco Devices for Syslog section of Resource Manager Essentials and Syslog Analysis: How-To.
-
show technical-support command output - The show technical-support command is a compilation of many different commands including show version, show running-config, and show stacks. TAC engineers usually ask for this information to troubleshoot hardware issues. It is important to collect the show technical-support information before doing a reload or power-cycle as these actions can cause all information about the problem to be lost.
-
The complete bootup sequence if the router experiences boot errors.
-
Crashinfo File (if available) - Information on obtaining the crashinfo file can be found at Retrieving Information from the Crashinfo File.
If you have the output of a show command from your Cisco device (including show technical-support), you can use the Output Interpreter
(registered customers only) to display potential issues and fixes. To use the Output Interpreter
(registered customers only) , you must be logged in and have JavaScript enabled.
Misleading Symptoms
There are a few issues that can be misinterpreted as hardware problems when, in fact, they are not. For instance, a failure following a new hardware installation is not always a hardware issue. Another example is when the router stops responding or "hangs". The table below provides symptoms, explanations, and troubleshooting steps for these commonly misinterpreted issues:
|
Symptom |
Explanation |
|---|---|
|
The Cisco 7400 hangs during normal operation |
This is usually caused by software problems. Refer to Troubleshooting Router Hangs. |
|
The Cisco 7400 boots into boot-mode |
The 7400 ASR router, upon initial boot up and install, may come up in boot mode. The default configuration does not presume there is a disk in the Compact Flash Disk slot. Add the boot system disk0: command to the configuration. The next time the system is reloaded, it boots the Cisco IOS software from the Compact Flash Disk. |
|
Router stuck in ROMmon (rommon # > prompt) |
For information on how to recover a Cisco 7400 Series Router stuck in ROMmon (rommon # > prompt), refer to ROMmon Recovery Procedure. |
|
The Port Adapter (PA) is not recognized and comes up with the console message, "%PA-2-UNDEFPA: Undefined Port Adapter type 106 in bay 1" or something similar to this |
This is generally an issue of hardware-software compatibility. Refer to the Software Advisor tool (registered customers only) to ensure the PA is supported in the Cisco IOS software currently installed on the router. |
|
CPU (Central Processing Unit) utilization is running very high |
While there are hardware problems that can cause this, it is much more likely that the router is either misconfigured or something on the network is causing the problem. The Troubleshooting High CPU Utilization on Cisco Routers page should help troubleshoot this. |
|
Memory allocation errors - SYS-2-MALLOCFAIL |
Memory allocation errors are almost never caused by hardware problems. Troubleshooting tips for memory allocation errors are found in Troubleshooting Memory Problems. |
|
Router crashes |
Not all crashes are caused by bad hardware. Troubleshooting Router Crashes can help you determine whether or not the crash was caused by software. |
|
Reload due to a bus error crash |
The system encounters a bus error when the processor tries to access a memory location that either does not exist (a software error) or does not respond properly (a hardware problem). A bus error can be identified by looking at the output of the show version command provided by the router (if not power-cycled or manually reloaded). Here are two examples of bus error crashes: Router uptime is 2 days, 21 hours, 30 minutes System restarted by bus error at PC 0x30EE546, address 0xBB4C4 System image file is "flash:igs-j-l.111-24.bin", booted via flash ......... At the console prompt, the following error message might also be seen during a bus error: *** System received a Bus Error exception *** signal= 0xa, code= 0x8, context= 0x608c3a50 PC = 0x60368518, Cause = 0x20, Status Reg = 0x34008002 For more information regarding this issue, refer to Troubleshooting Bus Error Crashes. |
|
At the first occurrence, simply monitor the router. At the second occurrence, replace the corresponding hardware as described in Processor Memory Parity Errors. |
|
|
Check the memory map of a show region versus the address of the bus error. If the address is valid, this is most likely a hardware issue. Try the Output Interpreter |
|
|
This is almost always a software problem. Upgrade to the latest Cisco IOS software release in your release train. |
|
|
SegV errors are always software-related problems. Upgrade to the latest Cisco IOS software release in your release train, or use the Output Interpreter |
|
|
Most of the time, these messages indicate a hardware issue. Replacing the CPU board usually solves the problem, unless other elements (for example, if a new module has been inserted and it starts reloading) point to a specific piece of hardware. Refer to Troubleshooting Watchdog Timeouts for more information on how to troubleshoot this problem. |
|
|
What Causes a Router To Be Restarted By the abort or trace trap Commands? |
If you don't power-cycle or manually reload the router, the show version output displays the following: Router uptime is 1 minute System restarted by abort at PC 0x802737BC System image file is "flash:c2600-i-mz.120-4.T" or Router uptime is 2 minutes System restarted by trace trap at PC 0x3171310 System image file is "flash:c2500-jos56i-l.120-9.bin" |
|
CPUHOG messages during the boot sequence are fairly common. The error message itself means that the boot process has held the CPU just a little longer than the system wanted it to, and then has sent a message to the console output to let you know. |
|
|
In most cases, this is the result of an improperly set configuration register. The configuration register is usually changed during password recovery to bypass the startup configuration upon reboot. Many times, the configuration register is not returned back to a normal setting. |
For more information, refer to Less Common Types of System Crashes.
Step-by-Step Troubleshooting
-
Parity errors - Parity errors can occur on the Cisco 7400 Series Router, including:
-
PCI Bus Parity Errors: These can be caused by a mis-seated or faulty port adapter (PA) or a faulty main board. The error messages should look something like this:
%ERR-1-GT64120A (PCI0):Fatal error, Parity error on master read GT=B4000000, cause=0x0110E083, mask=0x0ED01F00, real_cause=0x00100000 Bus_err_high=0x00000000, bus_err_low=0x00000000, addr_decode_err=0x00000470 %ERR-1-SERR: PCI bus system/parity error %ERR-1-FATAL: Fatal error interrupt, No reloading Err_stat=0x81, err_enable=0xFF, mgmt_event=0x40
Do not be concerned if your error message contains "GT64010" or "GT64120A". These are the same components, but with different revisions. Because of the similarities between the Cisco 7200 Series Routers and the Cisco 7400 Series Routers, it is recommended that you review the Cisco 7200 Parity Error Fault Tree page (section GT64120 System Parity Error Master Read) to determine the potential component causing this issue.
-
Processor Memory Parity Errors: Follow the applicable instructions on the Cisco 7200 Parity Error Fault Tree page to troubleshoot this error message.
-
-
System Restarted By Bus Error Exception: This can be either hardware- or software-related. Below is an example of such an error message:
*** System received a Bus Error exception *** signal= 0xa, code= 0x18, context= 0x6206b820 PC = 0x606e356c, Cause = 0x6020, Status Reg = 0x34008002
For potential hardware-related issues, follow these steps:
-
Power down the router and remove the Port Adapter (PA) from the unit. Power the system back up and see if the problem continues.
-
If the system reloads correctly, place the PA back into the router, power the system back up and see if the problem continues. The purpose of this step is to remove the possibility of the PA being mis-seated.
-
If the system does NOT reload correctly, and continues to reboot or display the Bus Error Exception message, further investigation is necessary to determine the root cause of the errors. The issue might be within the chassis or it could be a software error. You can open a TAC case for further troubleshooting. Ensure that you include all relevant information (for example, troubleshooting logs, troubleshooting steps taken, and so on).
-
-
Continuously Rebooting: If the Cisco 7400 Series Router is continuously rebooting, even after a power-cycle of the router, follow these steps:
-
Remove the PA, then power-cycle the router.
-
If it still fails, check if there is a valid image on the Compact Flash Disk. In order to do this, you must be directly connected to the console port of the router. Send the break key within the first 60 seconds of bootup to go into ROMmon. From there, you can follow the procedures in the ROMmon Recovery Procedure to try to recover.
-
If the router still does not boot, and you are sure that there is a valid image on it, then further investigation is necessary to determine the root cause of the errors. The issue might be within the chassis or it could be a software error. You can open a TAC case for further troubleshooting. Ensure that you include all relevant information (for example, troubleshooting logs, troubleshooting steps taken, and so on).
-
Information to Collect if You Open a TAC Service Request
|
If you still need assistance after following the troubleshooting steps above and want to create a service request (registered customers only) with the Cisco TAC, be sure to include the following information: |
|---|
You can attach information to your service request by uploading it using the TAC Service Request Tool (registered customers only) . If you cannot access the TAC Service Request Tool, you can send the information in an email attachment to attach@cisco.com with your service request number in the subject line of your message to attach the relevant information to your service request. Note: Please do not manually reload or power-cycle the router before collecting the above information unless required to troubleshoot memory problems as this can cause important information to be lost that is needed for determining the root cause of the problem. |
Related Information
- Standard Break Key Sequence Combinations During Password Recovery
- Retrieving Information from the Crashinfo File
- ROMmon Recovery Procedure
- Troubleshooting Router Hangs
- Troubleshooting High CPU Utilization on Cisco Routers
- Troubleshooting Memory Problems
- Troubleshooting Router Crashes
- Cisco 7200 Parity Error Fault Tree
- Technical Support - Cisco Systems
| Updated: Jul 07, 2005 | Document ID: 18676 |
