Hardware Troubleshooting for the Cisco Versatile Interface Processor (VIP)

Available Languages

Download Options

  • PDF     (16.2 KB)
    View with Adobe Reader on a variety of devices
Updated:March 9, 2009
Document ID:16123

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps troubleshoot common hardware issues with the Cisco 7500 series routers and, more specifically, its Versatile Interface Processor (VIP) cards. This document also provides pointers for identifying faulty hardware.

Note: This document does not cover any software-related failures except for those that are often mistaken as hardware issues.

Prerequisites

Requirements

Readers of this document should be knowledgeable of the following:

Components Used

The information in this document is based on the software and hardware versions below.

  • All Versatile Interface Processors (VIPs) for the Cisco 7500 Series Routers, including the following:

    VIP Type Description
    VIP2-10 Contains a (MIPS) R4600 CPU that runs at 100 MHz internally, 50 MHz external bus speed and contains 8 MB memory
    VIP2-15 Contains a (MIPS) R4600 CPU that runs at 100 MHz internally, 50 MHz external bus speed and contains 8 MB memory
    VIP2-40 Contains a (MIPS) R4600 CPU that runs at 100 MHz internally, 50 MHz external bus speed and contains 16 MB memory
    VIP2-50 Contains a (MIPS) R5000 CPU that runs at 250 MHz internally, and supports memory options from 32MB to 128MB
    VIP4-50 Contains a (MIPS) R5000 CPU and supports memory options from 64MB to 256MB. Memory is ECC-protected
    VIP4-80 Contains a (MIPS) R7000 CPU and supports memory options from 64MB to 256MB. Memory is ECC-protected.
    VIP6-80 Contains a (MIPS) R7000 CPU that runs at 400 MHz internally, and supports memory options from 64MB to 256MB. Memory is ECC-protected.

  • All Cisco IOS® software versions

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Hardware-Software Compatibility and Memory Requirements

Whenever you install a new RSP, VIP, port adapter, or Cisco IOS software image, it is important to verify that the router has enough memory, and that the hardware and software are compatible.

Perform the following recommended steps to check for hardware-software compatibility and memory requirements:

  1. Use the Software Advisor (registered customers only) Tool to verify whether the modules and cards are supported by the desired Cisco IOS software version.

  2. Use the Cisco Download Software Area (registered customers only) to check the minimum amount of memory (RAM and Flash) required by the Cisco IOS software, and/or download the Cisco IOS software image. IN order to determine the amount of memory (RAM and Flash) installed, refer to Memory Requirements.

    Tips:

    • In the Cisco IOS Upgrade Planner, you need to select the platform and the recommended Cisco IOS software release from step 1 in order to view the memory requirements.

    • If you need to upgrade the Cisco IOS software image to a new version, see How to Choose a Cisco IOS Software Release for more information.

If you determine that a Cisco IOS software upgrade is required, follow the Software Installation and Upgrade Procedures for the Cisco 7500 Series Router.

Error Messages

The Error Message Decoder (registered customers only) tool allows you to check the meaning of an error message. Error messages appear on the console of Cisco products, usually in the following form: 

%XXX-n-YYYY : [text]

Here is an example of an error message:

Router# %SYS-2-MALLOCFAIL: Memory allocation of [dec] bytes failed from [hex],
   pool [chars], alignment [dec]

Some error messages are informational only, while others indicate hardware or software failures and require action. The Error Message Decoder (registered customers only) tool provides an explanation of the message, a recommended action (if needed), and if available, a link to a document that provides extensive troubleshooting information about that error message.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Platform Description

This section covers the Versatile Interface Processor 2 (VIP2) and the Versatile Interface Processor 4 (VIP4) for Cisco 7500 Series Routers.

Interface Processors (IPs), in general, contain the network interfaces for the Cisco 7500 Series Router. There are two kinds of IPs:

  • Legacy Interface Processors: These cards are considered to be non-intelligent compared to the VIPs as they cannot make any packet switching decisions. Also, the entire card needs to be swapped out if there is a need for a different interface.

    Legacy interface processors do not contain dissimilar interfaces, meaning that you cannot have a Token Ring interface with an Ethernet interface on the same legacy interface processor.

  • Versatile Interface Processors (VIPs): VIPs are versatile, meaning that they can contain two different interfaces. Thus, unlike the Legacy Interface Processor, the VIP can contain a Token Ring and Ethernet interface on the same Interface Processor. These interfaces are built into the Port Adapter (PA) which needs to be inserted on the VIP.

    Changing an interface is as simple as swapping a PA. PAs come in two sizes: single-wide and dual-wide. A single-wide PA takes up only one slot, whereas dual-wide PAs are twice the size of single-wide PAs and fill two slots. A VIP can hold only one dual-wide PA.

    If distributed switching is enabled on the router, the VIP can make packet switching decisions. This decreases the load from the Route Switch Processor (RSP) and shifts it to the VIP.

    In a distributed switching configuration environment, the RSP sends information such as routing table updates to the VIP that is used to make switching decisions. VIPs are essentially routers on a blade that allow the 7500 to be a distributed system in which switching decisions can be made on the VIP instead of by the route processor.

Notes:

Identifying the Issue

VIPs can reboot or reload for various reasons. Several of these are due to potential hardware issues. Below is information on how to capture outputs helpful for troubleshooting and identifying misleading symptoms caused by bad hardware. The troubleshooting steps for the symptoms are listed in the Troubleshooting section below.

Capturing Information

In order to determine what is causing the issue, the first step is to capture as much information about the problem as possible. The following information is essential to determining the cause of the problem:

  • VIP crashinfo file(s) - When a VIP crashes, a file is saved into the bootflash of the primary RSP. The crashinfo contains logs, memory dumps, and other vital information to facilitate the troubleshooting. Detailed information on crashinfo can be found in Retrieving Information from the Crashinfo File.

  • RSP console logs and/or Syslog information - These are crucial in determining the originating issue if multiple symptoms are occurring (this is usually the case when a VIP crashes or has other problems). Effective troubleshooting can be done if the console log/syslog is made available. If the router is set up to send logs to a syslog server, check the server for the log. For RSP console logs, make sure you are directly connected to the console port of the router and Apply Correct Terminal Emulator Settings for Console Connections. Ensure that logging is enabled.

  • Output from the show diagbus command - When a VIP crashes, the reason for the latest crash may be displayed in the show diagbus command. This information can be helpful in troubleshooting the problem. This command is also part of the show technical-support command which may be requested by the Cisco Technical Assistance Center (TAC).

If you have the output of a show command from your Cisco device (including show technical-support), you can use to display potential issues and fixes. To use , you must be a registered customer, be logged in, and have JavaScript enabled.

Misleading Symptoms

There are a few issues that can be misinterpreted as hardware problems when, in fact, they are not. For instance, a failure following a new hardware installation is not always a hardware issue. The table below lists symptoms, explanations, and troubleshooting steps for these commonly misinterpreted issues:

Symptom Explanation
The VIP is not recognized when installed, or it crashes on bootup or Online Insertion and Removal (OIR) of a new VIP. Use the Software Advisor (registered customers only) tool to see if the VIP is supported in your current Cisco IOS software version. Also, make sure your RxBoot image supports the VIP. The background section of What Causes "Bad CPU ID" Messages contains a good explanation of the differences between the main Cisco IOS software image and the RxBoot image.
Output Stuck/Output Frozen/Not Transmitting error messages These error messages are usually caused by software problems and are discussed in detail in What Causes %RSP-3-RESTART: interface [xxx], output stuck/frozen/not transmitting Messages?.
The error message "RSP-3-RESTART: cbus complex" This error message may be due to configuration changes, OIR of an interface processor or other software, or bad hardware issues. This error message is discussed in detail in What Causes a "%RSP-3-RESTART: cbus complex"?.
A VIP running at very high CPU utilization This is very rarely caused by a hardware issue. More information on one of the most common reasons for high VIP CPU utilization is discussed in Understanding VIP CPU Running at 99% and Rx-Side Buffering.
VIP crashes Not all VIP crashes are caused by bad hardware. Troubleshooting Versatile Interface Processor (VIP) Crashes can help you determine whether or not the crash was caused by software.
Memory Size Unknown error message This message may be seen in the output of the show diagbus command. This message simply means that the VIP did not complete the bootup process. There are several reasons why a VIP does not completely boot up, such as:
  • There is no PA in the VIP - This is not a supported configuration. A VIP that is installed in a 7500 Series Router MUST have at least one supported PA installed.
  • The VIP or the PA installed is not supported by the Cisco IOS software that is running.
  • There is no memory installed in the VIP.
  • There is not enough memory in the VIP to boot its microcode.
More information is available at Cisco 7500 Series: Frequently Asked Questions.
VIP4 fails to boot up The VIP4 uses the same type of memory - Synchronous Dynamic RAM (SDRAM) - for both processor memory and packet memory. For this reason, it is not uncommon to mistakenly install the packet memory in the processor memory slot or the processor memory in the packet memory slot. If this occurs, it is possible that the VIP will not have enough memory to boot its microcode. It is very important to make sure that the proper memory is installed in the different memory slots.

Troubleshooting

  • Parity Errors - Parity errors on a 7500 are usually triggered due to bad hardware. To troubleshoot parity errors, capture (as stated above in the Capturing Information section) the output at the time of the crash.

    When you have collected this information, see Troubleshooting Versatile Interface Processor Crashes - Parity Errors for troubleshooting steps. VIP Crash Fault Tree Analysis can also help you narrow down the cause of the VIP parity error crash.

  • Negative acknowledgement message (NACK) present on CyBus - While this is usually a software problem, it can also be a hardware problem. Be sure to capture the RSP console log at the time of the problem and then refer to Troubleshooting Versatile Interface Processor Crashes - NACK present on CyBus for further troubleshooting information.

    Note: If you see the message "Parity Error from CyBus" or "NACK Present on CyBus access" anywhere in these error messages, the parity error comes from some other component, or there is a mis-inserted card within the 7500.

Information to Collect if You Open a TAC Case

If you still need assistance after following the troubleshooting steps above and want to create a service request with the Cisco TAC, use the TAC Service Request Tool (registered customers only) and be sure to include the following information:
  • Console captures showing the error messages
  • Console captures showing the troubleshooting steps taken and the boot sequence during each step
  • The hardware component that failed and the serial number for the chassis
  • Troubleshooting logs
  • Output from the show technical-support command

Related Information

Revision History

Revision Publish Date Comments
1.0
09-Mar-2009
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco