Interactive: This document offers customized analysis of your Cisco device.
Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps troubleshoot common hardware issues with the Cisco 7500 series routers and, more specifically, its Versatile Interface Processor (VIP) cards. This document also provides pointers for identifying faulty hardware.
Note: This document does not cover any software-related failures except for those that are often mistaken as hardware issues.
Readers of this document should be knowledgeable of the following:
The information in this document is based on the software and hardware versions below.
All Versatile Interface Processors (VIPs) for the Cisco 7500 Series Routers, including the following:
VIP Type Description VIP2-10 Contains a (MIPS) R4600 CPU that runs at 100 MHz internally, 50 MHz external bus speed and contains 8 MB memory VIP2-15 Contains a (MIPS) R4600 CPU that runs at 100 MHz internally, 50 MHz external bus speed and contains 8 MB memory VIP2-40 Contains a (MIPS) R4600 CPU that runs at 100 MHz internally, 50 MHz external bus speed and contains 16 MB memory VIP2-50 Contains a (MIPS) R5000 CPU that runs at 250 MHz internally, and supports memory options from 32MB to 128MB VIP4-50 Contains a (MIPS) R5000 CPU and supports memory options from 64MB to 256MB. Memory is ECC-protected VIP4-80 Contains a (MIPS) R7000 CPU and supports memory options from 64MB to 256MB. Memory is ECC-protected. VIP6-80 Contains a (MIPS) R7000 CPU that runs at 400 MHz internally, and supports memory options from 64MB to 256MB. Memory is ECC-protected.
All Cisco IOS® software versions
The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.
Whenever you install a new RSP, VIP, port adapter, or Cisco IOS software image, it is important to verify that the router has enough memory, and that the hardware and software are compatible.
Perform the following recommended steps to check for hardware-software compatibility and memory requirements:
Use the Cisco Download Software Area (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. IN order to determine the amount of memory (RAM and Flash) installed, refer to Memory Requirements.
In the Cisco IOS Upgrade Planner, you need to select the platform and the recommended Cisco IOS software release from step 1 in order to view the memory requirements.
If you need to upgrade the Cisco IOS software image to a new version, see How to Choose a Cisco IOS Software Release for more information.
If you determine that a Cisco IOS software upgrade is required, follow the Software Installation and Upgrade Procedures for the Cisco 7500 Series Router.
%XXX-n-YYYY : [text]
Here is an example of an error message:
Router# %SYS-2-MALLOCFAIL: Memory allocation of [dec] bytes failed from [hex], pool [chars], alignment [dec]
Some error messages are informational only, while others indicate hardware or software failures and require action. The Error Message Decoder (registered customers only) tool provides an explanation of the message, a recommended action (if needed), and if available, a link to a document that provides extensive troubleshooting information about that error message.
For more information on document conventions, see the Cisco Technical Tips Conventions.
This section covers the Versatile Interface Processor 2 (VIP2) and the Versatile Interface Processor 4 (VIP4) for Cisco 7500 Series Routers.
Interface Processors (IPs), in general, contain the network interfaces for the Cisco 7500 Series Router. There are two kinds of IPs:
Legacy Interface Processors: These cards are considered to be non-intelligent compared to the VIPs as they cannot make any packet switching decisions. Also, the entire card needs to be swapped out if there is a need for a different interface.
Legacy interface processors do not contain dissimilar interfaces, meaning that you cannot have a Token Ring interface with an Ethernet interface on the same legacy interface processor.
Versatile Interface Processors (VIPs): VIPs are versatile, meaning that they can contain two different interfaces. Thus, unlike the Legacy Interface Processor, the VIP can contain a Token Ring and Ethernet interface on the same Interface Processor. These interfaces are built into the Port Adapter (PA) which needs to be inserted on the VIP.
Changing an interface is as simple as swapping a PA. PAs come in two sizes: single-wide and dual-wide. A single-wide PA takes up only one slot, whereas dual-wide PAs are twice the size of single-wide PAs and fill two slots. A VIP can hold only one dual-wide PA.
If distributed switching is enabled on the router, the VIP can make packet switching decisions. This decreases the load from the Route Switch Processor (RSP) and shifts it to the VIP.
In a distributed switching configuration environment, the RSP sends information such as routing table updates to the VIP that is used to make switching decisions. VIPs are essentially routers on a blade that allow the 7500 to be a distributed system in which switching decisions can be made on the VIP instead of by the route processor.
Distributed switching is not available on VIP2-10s and VIP2-15s.
There may be restrictions and limitations on which PAs can be inserted into a specific VIP. Consult the product documentation listed below to verify that the combination is supported before you attempt to implement it.
For the VIP2 Series, see the VIP2 and Port Adapter Compatibility section of Second-Generation Versatile Interface Processor (VIP2) Installation and Configuration.
For the VIP4 Series, see the VIP4 and Port Adapter Compatibility section of the Fourth Generation Versatile Interface Processor (VIP4) Installation and Configuration Guide.
VIPs can reboot or reload for various reasons. Several of these are due to potential hardware issues. Below is information on how to capture outputs helpful for troubleshooting and identifying misleading symptoms caused by bad hardware. The troubleshooting steps for the symptoms are listed in the Troubleshooting section below.
In order to determine what is causing the issue, the first step is to capture as much information about the problem as possible. The following information is essential to determining the cause of the problem:
VIP crashinfo file(s) - When a VIP crashes, a file is saved into the bootflash of the primary RSP. The crashinfo contains logs, memory dumps, and other vital information to facilitate the troubleshooting. Detailed information on crashinfo can be found in Retrieving Information from the Crashinfo File.
RSP console logs and/or Syslog information - These are crucial in determining the originating issue if multiple symptoms are occurring (this is usually the case when a VIP crashes or has other problems). Effective troubleshooting can be done if the console log/syslog is made available. If the router is set up to send logs to a syslog server, check the server for the log. For RSP console logs, make sure you are directly connected to the console port of the router and Apply Correct Terminal Emulator Settings for Console Connections. Ensure that logging is enabled.
Output from the show diagbus command - When a VIP crashes, the reason for the latest crash may be displayed in the show diagbus command. This information can be helpful in troubleshooting the problem. This command is also part of the show technical-support command which may be requested by the Cisco Technical Assistance Center (TAC).
There are a few issues that can be misinterpreted as hardware problems when, in fact, they are not. For instance, a failure following a new hardware installation is not always a hardware issue. The table below lists symptoms, explanations, and troubleshooting steps for these commonly misinterpreted issues:
|The VIP is not recognized when installed, or it crashes on bootup or Online Insertion and Removal (OIR) of a new VIP.||Use the Software Advisor (registered customers only) tool to see if the VIP is supported in your current Cisco IOS software version. Also, make sure your RxBoot image supports the VIP. The background section of What Causes "Bad CPU ID" Messages contains a good explanation of the differences between the main Cisco IOS software image and the RxBoot image.|
|Output Stuck/Output Frozen/Not Transmitting error messages||These error messages are usually caused by software problems and are discussed in detail in What Causes %RSP-3-RESTART: interface [xxx], output stuck/frozen/not transmitting Messages?.|
|The error message "RSP-3-RESTART: cbus complex"||This error message may be due to configuration changes, OIR of an interface processor or other software, or bad hardware issues. This error message is discussed in detail in What Causes a "%RSP-3-RESTART: cbus complex"?.|
|A VIP running at very high CPU utilization||This is very rarely caused by a hardware issue. More information on one of the most common reasons for high VIP CPU utilization is discussed in Understanding VIP CPU Running at 99% and Rx-Side Buffering.|
|VIP crashes||Not all VIP crashes are caused by bad hardware. Troubleshooting Versatile Interface Processor (VIP) Crashes can help you determine whether or not the crash was caused by software.|
|Memory Size Unknown error message||
This message may be seen in the output of the show
diagbus command. This message simply means that the VIP did not
complete the bootup process. There are several reasons why a VIP does not
completely boot up, such as:
|VIP4 fails to boot up||The VIP4 uses the same type of memory - Synchronous Dynamic RAM (SDRAM) - for both processor memory and packet memory. For this reason, it is not uncommon to mistakenly install the packet memory in the processor memory slot or the processor memory in the packet memory slot. If this occurs, it is possible that the VIP will not have enough memory to boot its microcode. It is very important to make sure that the proper memory is installed in the different memory slots.|
Parity Errors - Parity errors on a 7500 are usually triggered due to bad hardware. To troubleshoot parity errors, capture (as stated above in the Capturing Information section) the output at the time of the crash.
When you have collected this information, see Troubleshooting Versatile Interface Processor Crashes - Parity Errors for troubleshooting steps. VIP Crash Fault Tree Analysis can also help you narrow down the cause of the VIP parity error crash.
Negative acknowledgement message (NACK) present on CyBus - While this is usually a software problem, it can also be a hardware problem. Be sure to capture the RSP console log at the time of the problem and then refer to Troubleshooting Versatile Interface Processor Crashes - NACK present on CyBus for further troubleshooting information.
Note: If you see the message "Parity Error from CyBus" or "NACK Present on CyBus access" anywhere in these error messages, the parity error comes from some other component, or there is a mis-inserted card within the 7500.
|If you still need assistance after following the troubleshooting steps above and want to create a service request with the Cisco TAC, use the TAC Service Request Tool (registered customers only) and be sure to include the following information:|
- Hardware Troubleshooting for the Cisco 7500 Series Router
- Hardware Troubleshooting for the Cisco Route Switch Processor (RSP)
- Port Adapters Documentation
- VIP Crash Fault Tree Analysis
- Retrieving Information from the Crashinfo File
- What Causes "Bad CPU ID" Messages
- Cisco 7500 Series: Frequently Asked Questions
- What Causes %RSP-3-RESTART: interface [xxx], output stuck/frozen/not transmitting Messages?
- What Causes a "%RSP-3-RESTART: cbus complex"?
- Understanding VIP CPU Running at 99% and Rx-Side Buffering
- Troubleshooting Versatile Interface Processor Crashes
- Cisco Versatile Interface Processors Product Support
- Technical Support & Documentation - Cisco Systems
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.