Troubleshooting Versatile Interface Processor (VIP) Crashes

Available Languages

Download Options

  • PDF     (116.9 KB)
    View with Adobe Reader on a variety of devices
Updated:October 2, 2006
Document ID:12811

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document provides information to troubleshoot a Versatile Interface Processor (VIP) crash.

Prerequisites

Requirements

Cisco recommends that you have knowledge of the 7500 Series Router Field Notices.

Components Used

The information in this document is based on these software and hardware versions:

  • VIP1

  • VIP2-10

  • VIP2-15

  • VIP2-20

  • VIP2-40

  • VIP2-50

  • FEIP2

  • GEIP

  • GEIP+

  • VIP4-50

  • VIP4-80

  • VIP6-80

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

VIP Architecture

In order to interpret a VIP crash, it is important to first understand the basic architecture of the VIP. The figure in this section shows the functional block diagram of the VIP2, which includes these components:

  • Orion reduced instruction set computing (RISC) CPU and associated circuitry, which includes Dynamic RAM (DRAM), L2 cache, RENO application-specific integrated circuit (ASIC), and Boot ROM.

  • CyBus ASIC—The component that controls and transfers packets between the VIP2 Static RAM (SRAM) and the system packet memory (MEMD) across the CyBus or CxBus.

  • Packet memory ASIC—Responsible for moving packets between port adapters and SRAM.

  • Peripheral Component Interconnect (PCI) buses—Data paths between the port adapters and VIP2 SRAM.

  • Bridges—Responsible for isolating the individual PCI buses of the port adapters.

vip_crash1.gif

The VIP2 microcode (firmware) is an image that provides card-specific software instructions. A programmable read-only memory (PROM) device on the VIP2 contains a default microcode boot image that assists the system in finding and loading the microcode image from the Cisco IOS® software bundle or from Flash memory. The microcode boot image in the PROM initializes the VIP2, and then assists downloading the VIP2 microcode image. All interfaces of the same type load the same microcode image, either from the Cisco IOS software bundle or from Flash memory. Although Flash memory can store multiple microcode versions for a specific interface type, only one image can load at startup.

The show controllers cbus command displays the currently loaded and running microcode version for each interface processor and the VIP2. The show startup-config command shows the current system instructions for loading microcode at startup.

When you troubleshoot, you can use the figure in this section as a guide to read information from a VIP crashinfo file or the syslog. As an example, look at this syslog output that shows bad parity is found when read from the VIP SRAM: 

Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 Nevada Error Interrupt Register = 0x2
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 PMA error register = 0046000000001000
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6    Packet Memory Read Parity error 

!--- Bad parity is found when read from the VIP SRAM.

Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6   PCI master address = 0460000
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 PA Bay 0 Upstream PCI-PCI Bridge, Handle=0
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 DEC21050 bridge chip, config=0x0
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x00): cfid   = 0x00011011
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x04): cfcs   = 0x02800147
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x08): cfccid = 0x06040002
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x0C): cfpmlt = 0x00010000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x18): cfsmlt = 0x00010100 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x1C): cfsis  = 0x02807020 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x20): cfmla  = 0x01F00000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x24): cfpmla = 0x0000FE00 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x3C): cfbc   = 0x00030000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x40): cfseed = 0x00000000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x44): cfstwt = 0x00000000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x48): cfswac = 0x00FFFFFF 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x4C): cfpwac = 0x00FFFFFF 
Apr 29 23:19:26: %VIP2 R5K-1-MSG: slot6 System reloaded by a fatal hardware error

As you see from the recommendations in the next section, the VIP in this output should be monitored, and the SRAM or VIP should be replaced if similar crashes reoccur.

How to Identify Your VIP

You can check the contents of address 0x21 in EEPROM in the show diag command output in order to verify the model of a VIP. The values that correspond to each VIP type are shown in this table:

Value VIP Controller Type
0x14 VIP1 VIP controller
0x15 VIP2 (VIP2-10, VIP2-15, VIP2-20, VIP2-40) VIP2 controller
0x1E VIP2-50 VIP2 R5K controller
0x20 FEIP2 FEIP2 controller
0x21 GEIP GEIP controller
0x40 GEIP +* GEIP+ controller
0x22 VIP4-80 VIP4-80 RM7000 controller
0x31 VIP4-50 VIP4-50 RM5271 controller
0x4E VIP6-80 VIP6-80 RM7000B controller

Note: The GEIP+ is based on the VIP4-80. All further information in this document about the VIP4-80 also applies to the GEIP+.

Here is an example:

Router#show diag 10 
     Slot 10: 
             Physical slot 10, ~physical slot 0x5, logical slot 10, CBus 0 
             Microcode Status 0x4 
             Master Enable, LED, WCS Loaded 
             Board is analyzed 
             Pending I/O Status: None 
             EEPROM format version 1 
             VIP2 R5K controller, HW rev 2.02, board revision D0 
             Serial number: 17090200  Part number: 73-2167-05 
             Test history: 0x00        RMA number: 00-00-00 
             Flags: cisco 7000 board; 7500 compatible 

             EEPROM contents (hex): 
               0x20: 01 1E 02 02 01 04 C6 98 49 08 77 05 00 00 00 00 
               0x30: 68 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 

             Slot database information: 
             Flags: 0x4      Insertion time: 0x18C0 (00:29:13 ago) 

             Controller Memory Size: 32 MBytes DRAM, 4096 KBytes SRAM

From this output, you can see that this VIP is a VIP2-50.

The difference between a VIP2-10, VIP2-15, VIP2-20, and a VIP2-40 is the amount of DRAM and SRAM on each. The various VIP2s (if they have not been upgraded) can be distinguished in the show diag command output by the memory configurations shown in this table:

Memory VIP
8 MB DRAM/512 KB SRAM VIP2-10
8 MB DRAM/1 MB SRAM VIP2-15
16 MB DRAM/1 MB SRAM VIP2-20
32 MB DRAM/2 MB SRAM VIP2-40

Obtain a VIP Crashinfo File

The information contained in the crashinfo file can prove to be invaluable when you try to resolve software issues or attempt to diagnose the underlying cause of system crashes. Not only does the crashinfo file contain logging information and a stack trace for the VIP, it also contains extensive memory and context information. Each time a VIP crashes, the VIP attempts to write a crashinfo file to the bootflash of the RSP. Crashinfo files are stored in this format:

crashinfo_vip_<slot#>_<data>_<time>

You can issue the dir command in order to locate VIP crashinfo files as shown here: 

7500a#dir bootflash: 
     Directory of bootflash:/ 

       1  -rw-     3951876   Jan 01 2000 00:01:22  rsp-boot-mz.111-22.CA 
       2  -rw-      162641   Jun 21 2000 12:53:40  crashinfo_vip_0_20000621-125340 
       3  -rw-      162778   Jun 21 2000 13:00:10  crashinfo_vip_0_20000621-130010 

     7602176 bytes total (3324492 bytes free) 
     7500a#

This bootflash of the router contains two VIP crashinfo files. Issue the show file or more commands in order to view and capture the contents of these files in accordance to this procedure:

  1. Start logging with your terminal program.

  2. Issue the term length 0 command.

  3. Issue the more bootflash:<crashinfo filename> command.

  4. Save the output to a file.

Refer to Retrieving Information from the Crashinfo File for more information on how to work with crashinfo files.

If you have the output of a show technical-support (from enable mode) command from your Cisco device, you can use in order to display potential issues and fixes. You must be a registered customer, be logged in, and have JavaScript enabled in order to use .

Crash Types

VIP crashes are classified into several categories based on the cause of the crash. Anytime a non-recoverable error is found, the VIP crashes. These errors can be the result of parity errors, software or hardware that cause a negative acknowledgement message (NACK) to be present on the CyBus, or software problems. This section provides information on each of these error types.

If you have the output of a show technical-support (from enable mode) command from your Cisco device, you can use to display potential issues and fixes. You must be a registered customer, be logged in, and have JavaScript enabled in order to use .

Parity Errors

VIP2 Parity Error Detection

Parity errors occur on a VIP when the hardware attempts to check the validity of data by comparing computed parity values to previous parity values for the same data. A single bit flip in the data can result in a parity error. When you diagnose parity errors on a VIP, it is important to understand each location at which parity is checked and at which parity errors can potentially occur. This diagram outlines this information. In addition, refer to Cisco 7500 VIP Fault Tree Analysis for more information about parity errors.

vip_crash2.gif

As shown in this diagram, there are seven different types of parity errors that can occur on a VIP. Note that errors can be received from another source and might not have originated within the VIP itself. The source of the parity error can be from the Route/Switch Processor (RSP), another VIP, or from poorly seated or faulty port adapters. In order to properly understand a VIP crash, it is important to diagnose the source of the crash.

It is also important to understand that data with bad parity can be reported by several of the parity checking devices on the VIP and Cisco 7500 Series Router for any single read or write operation. For example, if the VIP reads a packet on a transmit queue on the RSP into its own SRAM, and there is a parity error in the SRAM of the RSP, then you see error messages from the MD ASIC on the RSP, the CYA ASIC on the VIP, and also the PCI/packet memory ASIC on the VIP.

This diagram shows fault-tree analysis for VIP crashes:

vip_crash3.gif

VIP4 and VIP6 Parity Errors and ECC Detection

The VIP4-50, VIP4-80, and VIP6-80 use Single Bit Error Correction and Double Bit Error Detection Error Code Correction (ECC) for CPU Memory and Packet Memory. Both are Synchronous Dynamic RAM (SDRAM). A single bit error in SDRAM is corrected and the system continues to operate normally.

Multibit parity errors at numbers 2 or 3 in this table are a fatal event which cause ECC multibit errors to occur. The CPU internal cache and buses in the system use single bit parity detection. As shown here, the architecture of the VIP4 and VIP6 are different from the VIP2. Therefore, some error messages are not seen and other error messages are reported differently than they are on the VIP2. In this parity error section, differences between the VIP2, the VIP4, and the VIP6 are denoted and explained.

vip_crash4.gif

Cache Parity Error Exception

Cache parity error exceptions occur when bad parity is discovered in the CPU or in the primary data cache. The parity error might have occurred in the VIP DRAM, the DRAM controller, the primary cache, or in the CPU itself. Parity errors discovered in this location are also referred to as processor memory parity errors (PMPEs). These errors result in an immediate crash of the VIP and the output looks similar on both VIPs and RSPs. A sig value of twenty (sig=20) indicates that a cache parity error exception has occurred. The sig value is displayed in the system log messages for the crash.

Recent code also provides a meaningful verbose line as shown here: 

Oct 21 00:11:14.913: %VIP2-1-MSG: slot0 System reloaded by a Cache Parity Exception 
Oct 21 00:11:14.913: %VIP2-1-MSG: slot0 System Reload called from 0x60125C8C, 
context=0x60220930 
Oct 21 00:11:14.913: %VIP2-1-MSG: slot0 System exception: sig=20, code=0x20025B69, 
context=0x60220930

Information contained in the VIP crashinfo file also points to the same parity error location in the primary data cache:

Error: primary data cache, fields: data, 
         virtual addr 0x6058A000, physical addr(21:3) 0x18A000, vAddr(14:12) 0x2000 
         virtual address corresponds to main:data, cache word 0 

                      Low Data   High Data  Par    Low Data   High Data  Par 
         L1 Data :  0:0xFEFFFEFE 0x65776179 0x13 1:0x20536572 0x76657220 0x89 
                    2:0x646F6573 0x206E6F74 0x9C 3:0x20737570 0x706F7274 0xF8 

                       Low Data   High Data  Par    Low Data   High Data  Par 
          Mem Data:  0:0xFEFFFEFE 0x65776179 0x13 1:0x20536572 0x76657220 0x89 
                     2:0x646F6573 0x206E6F74 0x9C 3:0x20737570 0x706F7274 0xF8

A primary cache or PMPE can be a transient error. If this is the first instance of a PMPE, you can usually safely ignore it. However, if the same VIP experiences a second or subsequent PMPEs, you should replace the VIP. Sometimes the replacement of the DRAM itself can also resolve the issue.

VIP4 and VIP6 Note—Parity errors which occur in the CPU internal cache and in the CyAsic are detected as cache parity error exceptions. Single bit parity errors in the CPU memory are corrected and no action needs to be taken. Multi-bit parity errors in the CPU memory are detected as a procmem ecc multi-bit parity error. The CPU memory in the VIP should be replaced if a procmem ecc multi-bit parity error is reported. 

Oct 25 09:30:54.708: %VIP4-50 RM5271-1-MSG: slot4 PMA error register1 00000000 
00002000
Oct 25 09:30:54.716: %VIP4-50 RM5271-1-MSG: slot4 Procmem ECC multi-bit error
Oct 25 09:30:54.724: %VIP4-50 RM5271-1-MSG: slot4 PCI1 master address 00000000
Oct 25 09:30:54.732: %VIP4-50 RM5271-1-MSG: slot4 PCI1 slave address 00000000
Oct 25 09:30:54.740: %VIP4-50 RM5271-1-MSG: slot4 Latched Addresses
Oct 25 09:30:54.748: %VIP4-50 RM5271-1-MSG: slot4 Procmem ECC multi-bit exception 
addr 22220000 025F0860
Oct 25 09:30:54.756: %VIP4-50 RM5271-1-MSG: slot4 Procmem ECC multi-bit exception 
data 00000000 00000000
Oct 25 09:30:54.764: %VIP4-50 RM5271-1-MSG: slot4 MPU addr exception/WPE address 
00000000 00000000
Oct 25 09:30:54.772: %VIP4-50 RM5271-1-MSG: slot4 MPU WPE addr/WPE data 00000000 
00000000
Oct 25 09:30:54.780: %VIP4-50 RM5271-1-MSG: slot4 ProcMem addr exception 0 0000000
Oct 25 09:30:54.788: %VIP4-50 RM5271-1-MSG: slot4 Pakmem addr exception 00000000
Oct 25 09:31:15.824: %VIP4-50 RM5271-1-MSG: slot4 System reloaded by a fatal 
hardware error
Oct 25 09:31:15.836: %VIP4-50 RM5271-1-MSG: slot4 caller=0x600BCE18
Oct 25 09:31:15.844: %VIP4-50 RM5271-1-MSG: slot4 System exception: sig22, 
code 0x0, context=0x60615F28

Parity Error from CyBus

When a VIP downloads from the MEMD in the RSP and these errors are seen, this usually indicates that another VIP has written bad parity to the MEMD, or the MEMD has been corrupted. If the source is from the MEMD and it continues, you need to replace the RSP. Conversely, if the source of the bad parity is another VIP, you should reseat and, if necessary, replace the VIP that writes the bad parity.

%VIP2-1-MSG: slot1 Nevada Error Interrupt Register 0x3 
%VIP2-1-MSG: slot1 CYASIC Error Interrupt register 0x2020000C 
%VIP2-1-MSG: slot1 Parity Error internal to CYA 
%VIP2-1-MSG: slot1 Parity Error in data from CyBus 

!--- Bad parity is received by the VIP from the CyBus.

%VIP2-1-MSG: slot1 CYASIC Other Interrupt register 0x200100 
%VIP2-1-MSG: slot1 QE HIGH Priority Interrupt 
%VIP2-1-MSG: slot1 CYBUS Error register 0xD001A02, PKT Bus Error register 0x0 
%VIP2-1-MSG: slot1 PMA error register = 0070000440000000 
%VIP2-1-MSG: slot1 Packet Bus Write Parity error 

!--- The bad parity that was received from the CyBus is written to SRAM.

%VIP2-1-MSG: slot1 PCI master address = 0700004 
%VIP2-1-MSG: slot1 PA Bay 0 Upstream PCI-PCI Bridge, Handle=0 
%VIP2-1-MSG: slot1 DEC21050 bridge chip, config=0x0 
%VIP2-1-MSG: slot1 (0x00): cfid = 0x00011011 
%VIP2-1-MSG: slot1 (0x04): cfcs= 0x02800147 
%VIP2-1-MSG: slot1 (0x08):cfccid = 0x06040002 
%VIP2-1-MSG: slot1 (0x0C):cfpmlt = 0x00010000 
%VIP2-1-MSG: slot1 (0x18): cfsmlt = 0x00010100 
%VIP2-1-MSG: slot1 (0x1C): cfsis = 0x22807020 
%VIP2-1-MSG: slot1 Received Master Abort on secondary bus 
%VIP2-1-MSG: slot1 (0x20): cfmla = 0x01F00000

Note: The VIP4 and VIP6 show the same error messages about the CyBus parity error, but the packet bus write parity error message is not displayed.

VIP I/O Controller and Reno Read Parity Error

Both DRAM controller parity errors and Input/Output (I/O) controller parity errors are detected by the RENO ASIC. A parity error that originates in DRAM or in the DRAM controller is reported as a cache parity exception. A parity error detected by the I/O controller is reported, as shown in this output. Often, parity errors reported by the I/O controller have originated elsewhere and are reported by the I/O controller in addition to messages from other locations.

Feb 17 23:03:04 cst: %VIP2 R5K-1-MSG: slot0 Reno read parity error - bytes 0 & 1 
Feb 17 23:03:04 cst: %VIP2 R5K-1-MSG: slot0 PMA error register = 0080004000001000 
Feb 17 23:03:04 cst: %VIP2 R5K-1-MSG: slot0 Packet Memory Read Parity error

Note: The VIP4 and VIP6 do not show this error message.

VIP Packet Memory Read Parity Error

The PMA ASIC reports a packet memory read parity error any time a parity error is read out of packet memory (SRAM) on the VIP. This error is reported in the system log as shown here:

Oct 30 05:18:06.120: %VIP2-1-MSG: slot9 Nevada Error Interrupt Register = 0x22 
Oct 30 05:18:06.120: %VIP2-1-MSG: slot9 PCI bus 0 parity error 
Oct 30 05:18:07.120: %VIP2-1-MSG: slot9 PMA error register = 4080103C00004000 
Oct 30 05:18:07.120: %VIP2-1-MSG: slot9 PCI Transmit Parity error 
Oct 30 05:18:08.120: %VIP2-1-MSG: slot9 Packet Memory Read Parity error

You can also see evidence of a packet memory read parity error in the VIP crashinfo file:

Nevada Error Interrupt Register = 0x2 
PMA error register = 0046000000001000 
Packet Memory Read Parity error
PCI master address = 0460000

SRAM parity errors can also be transient, so treat the first occurrence the same way as DRAM parity errors. If the errors persist, replace the SRAM or the VIP.

VIP4 and VIP6 Note—Single bit parity errors in the packet memory are corrected. Multi-bit parity errors in the packet memory are detected as a pakmem ecc multi-bit parity error. The VIP packet memory should be replaced if a pakmem ecc multi-bit parity error is reported. 

%VIP4-80 RM7000-1-MSG: slot1 PMA error register0 = 0000000000002000
%VIP4-80 RM7000-1-MSG: slot1 Pakmem ECC multi-bit error
%VIP4-80 RM7000-1-MSG: slot1 PCI0 master address = 00000000
%VIP4-80 RM7000-1-MSG: slot1 PCI0 slave address = 00000000
%VIP4-80 RM7000-1-MSG: slot1 PMA error register1 = 0000000000000000
%VIP4-80 RM7000-1-MSG: slot1 PCI1 master address = 00000000
%VIP4-80 RM7000-1-MSG: slot1 PCI1 slave address = 00000000
%VIP4-80 RM7000-1-MSG: slot1 Latched Addresses
%VIP4-80 RM7000-1-MSG: slot1 Pakmem ECC multi-bit exception addr = 00012358 000000CA
%VIP4-80 RM7000-1-MSG: slot1 Pakmem ECC multi-bit exception data = 00000000 00040800
%VIP4-80 RM7000-1-MSG: slot1 MPU addr exception/WPE address = 00000000 00000000
%VIP4-80 RM7000-1-MSG: slot1 MPU WPE addr/WPE data = 00000000 00000000
%VIP4-80 RM7000-1-MSG: slot1 ProcMem addr exception = 00000000
%VIP4-80 RM7000-1-MSG: slot1 Pakmem addr exception = 00000000

VIP Packet Bus Write Parity Error

The PMA ASIC reports a packet bus write parity error any time a parity error is being written to packet memory. In this example, the VIP is only the messenger and the problem does not exist with the memory of this VIP.

May 10 09:22:14.520: %VIP2-1-MSG: slot11 PMA error register = 2080002800800200 
May 10 09:22:15.520: %VIP2-1-MSG: slot11 Packet Bus Write Parity error

Note: The VIP4 and VIP6 do not show this error message.

VIP PCI Bus Parity Error

Parity errors can be detected in PCI buses 1 and 2, both of which directly interface with the port adapters. These buses are bridged together by a third PCI bus, bus 0, on which parity errors can also be detected. Parity errors that originate from any of the PCI buses are most commonly caused by poorly seated or faulty port adapters. Any time you see these messages in the syslog output of a VIP crash, you need to reseat the port adapter in order to resolve the issue.

PCI bus <num> parity error 
PCI bus <num> system error
Detected Parity Error on secondary bus

If reseating the port adapter does not solve the issue, the problem lies with either the port adapter or the VIP. Move the port adapter to another bay and insert a second port adapter into the original bay in order to troubleshoot. This usually points to the offending hardware. An example is shown here:

Mar 16 19:34:54: %GEIP-1-MSG: slot9 Nevada Error Interrupt Register = 0x6 
Mar 16 19:34:54: %GEIP-1-MSG: slot9 PCI bus 0 system error
Mar 16 19:34:54: %GEIP-1-MSG: slot9 PMA error register = 0080043800100000 
Mar 16 19:34:54: %GEIP-1-MSG: slot9 PCI IRDY time-out 
Mar 16 19:34:54: %GEIP-1-MSG: slot9 PCI master address = 0800438 
Mar 16 19:34:54: %GEIP-1-MSG: slot9 PA Bay 0 Upstream PCI-PCI Bridge, Handle=0

Note: The same errors occur with the VIP4 and the VIP6, but the error message is different. It is detected as a PCI master parity error and a PCI slave parity error. Perform the same steps as outlined for VIP PCI Bus Parity Errors to troubleshoot this problem.

00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PMA error register0 = 0000000001800000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PCI Master Parity error
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PCI Slave Parity error
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PCI0 master address = 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PCI0 slave address = 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PMA error register1 = 0000000000000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PCI1 master address = 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 PCI1 slave address = 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 Latched Addresses
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 MPU addr exception/WPE address = 
00000000 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 MPU WPE addr/WPE data = 
00000000 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 ProcMem addr exception = 00000000
00:00:44: %VIP4-50 RM5271-1-MSG: slot1 Pakmem addr exception = 00000000

VIP CyAsic Parity Error

Parity errors can also be detected by the VIP in the data or address of a read or write operation on the CyBus. If this occurs, you see syslog output similar to this:

CYASIC Error Interrupt register 0x2020000C 
Parity Error internal to CYA 
Parity Error in data from CyBus

Use this information in conjunction with the information present in the system logs to determine the true source of the error.

Note: The VIP4 and the VIP6 show the same error messages for CyBus parity errors.

NACK Present on the CyBus

When the VIP tries to write to an invalid address in MEMD, the RSP places a NACK on the CyBus for that slot. This is usually a software problem, but can also be a hardware issue. For example, in this output, the VIP writes 4 bytes to an invalid address, so the RSP places a NACK on the CyBus for that slot.

%RSP-3-ERROR: CyBus0 error 10 
%RSP-3-ERROR: command/address mismatch 
%RSP-3-ERROR: bus command write 4bytes (0xE) 
%RSP-3-ERROR: address offset (bits 3:1) 0 
%RSP-3-ERROR: virtual address (bits 23:17) 000000 
%VIP2-1-MSG: slot5 Nevada Error Interrupt Register = 0x1 
%VIP2-1-MSG: slot5 CYASIC Error Interrupt register 0x20000003 
%VIP2-1-MSG: slot5 Missing ACK on CyBus access 
%VIP2-1-MSG: slot5 NACK present on CyBus access
%VIP2-1-MSG: slot5 CYASIC Other Interrupt register 0x0 
%VIP2-1-MSG: slot5 CYBUS Error register 0x8001C48, PKT Bus Error register 0x0 
%VIP2-1-MSG: slot5 System reloaded by a fatal hardware error 
%VIP2-1-MSG: slot5 caller=0x60126C44 
%VIP2-1-MSG: slot5 System exception: sig=22, code=0x0, context=0x60265C68

However, as shown in this output, the RSP also places a NACK on a VIP for trying to write bad parity to the MEMD.

CYASIC Error Interrupt register 0x1B 
Parity Error in data from Packet Bus 
Parity Error internal to CYA 
Missing ACK on CyBus access 
NACK present on CyBus access

When a NACK appears on all slots, as shown in this output, it is a hardware error. The arbiter is faulty and the card cage must be replaced.

Jan  1 23:55:21: %FEIP2-1-MSG: slot0 Nevada Error Interrupt Register =0x1 
Jan  1 23:55:21: %FEIP2-1-MSG: slot0 CYASIC Error Interrupt register0x20000001 
Jan  1 23:55:21: %FEIP2-1-MSG: slot0 NACK present on CyBus access 
Jan  1 23:55:21: %FEIP2-1-MSG: slot0 CYASIC Other Interrupt register 0x0 
Jan  1 23:55:21: %FEIP2-1-MSG: slot0 CYBUS Error register 0x8001A00, 
PKTBus Error register 0x0 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 Nevada Error Interrupt Register = 0x1 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 CYASIC Error Interrupt register 0x20000001 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 NACK present on CyBus access
Jan  1 23:55:21: %VIP2-1-MSG: slot2 CYASIC Other Interrupt register 0x0 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 CYBUS Error register 0x800006A, 
PKT Bus Error register 0x0 
Jan  1 23:55:21: %VIP2-1-MSG: slot5 Nevada Error Interrupt Register = 0x1 
Jan  1 23:55:21: %VIP2-1-MSG: slot5 CYASIC Error Interrupt register 0x20200001 
Jan  1 23:55:21: %VIP2-1-MSG: slot5 NACK present on CyBus access 
Jan  1 23:55:21: %VIP2-1-MSG: slot5 CYASIC Other Interrupt register 0x200000 
Jan  1 23:55:21: %VIP2-1-MSG: slot5 CYBUS Error register 0x800006C, 
PKT Bus Error register 0x0 
Jan  1 23:55:21: %FEIP2-1-MSG: slot8 Nevada Error Interrupt Register = 0x1 
Jan  1 23:55:21: %FEIP2-1-MSG: slot8 CYASIC Error Interrupt register 0x20000001 
Jan  1 23:55:21: %FEIP2-1-MSG: slot8 NACK present on CyBus access
Jan  1 23:55:21: %FEIP2-1-MSG: slot8 CYASIC Other Interrupt register 0x0 
Jan  1 23:55:21: %FEIP2-1-MSG: slot8 CYBUS Error register 0x8001B80, 
PKT Bus Error register 0x0 
Jan  1 23:55:21: %FEIP2-1-MSG: slot10 Nevada Error Interrupt Register = 0x1 
Jan  1 23:55:21: %FEIP2-1-MSG: slot10 CYASIC Error Interrupt register 0x20000001 
Jan  1 23:55:21: %FEIP2-1-MSG: slot10 NACK present on CyBus access
Jan  1 23:55:21: %FEIP2-1-MSG: slot10 CYASIC Other Interrupt register 0x0 
Jan  1 23:55:21: %FEIP2-1-MSG: slot10 CYBUS Error register 0x8001C08,
PKT Bus Error register 0x0 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 System reloaded by a fatal hardware error 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 caller=0x6012640C 
Jan  1 23:55:21: %VIP2-1-MSG: slot2 System exception: sig=22, code=0x0, 
context=0x60265028

VIP Crashes Due to Software

VIP crashes not caused by any of the reasons in this document are most commonly due to other software issues. These crashes can be manifested in a variety of different ways. These are general suggestions to reduce the risk of VIP crashes due to software problems and to cope with them if they occur:

  • Always make sure that the Cisco IOS software image supports the VIP.

  • Always keep the RSP-BOOT image and the main Cisco IOS software image at the same version.

  • Ensure that the VIP configuration and the port adapter are supported by the current version of Cisco IOS software.

  • Check the release notes for the correct Cisco IOS software level and memory requirements.

This is an example of a system log output of a VIP crash due to a software problem:

Apr 18 17:13:33.884: %VIP2 R5K-1-MSG: slot0 System reloaded by a 
Bus Error exception 
Apr 18 17:13:33.892: %VIP2 R5K-1-MSG: slot0 caller=0x600BC974 
Apr 18 17:13:33.900: %VIP2 R5K-1-MSG: slot0 System exception: 
sig=10, code=0x408, context=0x605B51E0 
Apr 18 17:13:33.912: %VIP2 R5K-1-MSG: slot0 $0 : 00000000, 
AT : 605B0000, v0 : 00000001, v1 : FFFFFFFC, 
Apr 18 17:13:33.924: %VIP2 R5K-1-MSG: slot0 a0 : 00000002, 
a1 : 6042CEE0, a2 : 00000000, a3 : 6112FEC4, 
Apr 18 17:13:33.936: %VIP2 R5K-1-MSG: slot0 t0 : 00000053, 
t1 : 3400FF01, t2 : 00000000, t3 : FFFFFFFF, 
Apr 18 17:13:33.948: %VIP2 R5K-1-MSG: slot0 t4 : 600BC9B0, 
t5 : 000000F8, t6 : 00000000, t7 : 00000002, 
Apr 18 17:13:33.956: %VIP2 R5K-1-MSG: slot0 s0 : 0C58BA24, 
s1 : 00000064, s2 : 6112C7AC, s3 : 60560000, 
Apr 18 17:13:33.964: %VIP2 R5K-1-MSG: slot0 s4 : 60560000, 
s5 : 00000001, s6 : 6041433C, s7 : 60414310, 
Apr 18 17:13:33.972: %VIP2 R5K-1-MSG: slot0 t8 : 00008945, 
t9 : 00000000, k0 : 607F6CA0, k1 : 00000200, 
Apr 18 17:13:33.980: %VIP2 R5K-1-MSG: slot0 gp : 6056AFC0, 
sp : 6112FEC0, s8 : 60414460, ra : 6026EC4C, 
Apr 18 17:13:33.988: %VIP2 R5K-1-MSG: slot0 EPC : 6026EAA0, 
ErrorEPC : 800086B8, SREG : 3400FF03 
Apr 18 17:13:33.996: %VIP2 R5K-1-MSG: slot0 Cause 00000408 (Code 0x2) 
Apr 18 17:13:34.004: %VIP2 R5K-1-MSG: slot0 Traceback= 6026EAA0 
6026E2E8 6009BAF4 6009BAE0 
Apr 18 17:13:35.012: %DBUS-3-DBUSINTERRSWSET: Slot 0, Internal 
Error due to VIP crash

Possibly the most important piece of information to obtain in the event of a software problem is the crashinfo file for the VIP. See the Obtain a VIP Crashinfo File section for instructions to capture this information.

Bus Error Exception

The VIP crashes many times and when you review the crash info file, you might see this message:

00:00:11: %LINK-3-UPDOWN: Interface POS1/0, changed state to up

IOBUS Error Interrupt Status register 0x0


Unexpected exception, CPU signal 10, PC = 0x602A7660


-Traceback= 602A7660 602AB238

The CPU signal 10 error message means bus exception error. Bus errors can be either software or hardware issues. The workaround for this problem is to reseat the module and monitor the router. If the module keeps crashing after you reseat the module, contact TAC Case Open tool (registered customers only) with the crash info file.

Report VIP Crashes to Cisco Technical Support

Before You Create a TAC Case

It is a good idea to create a VIP crash summary file with this information before you open a case. Include this information in the Problem Description field of the TAC Case Open tool (registered customers only) .

  • Problem description

  • Output of the show version command

  • Output of the show diag slot [x] command

  • Crashinfo file excerpt

  • Syslog excerpt

Then, collect the Information to Collect if You Open a TAC Case.

This is an example of a crash summary file: 

******problem description.. 
VIP crashed with parity errors. 
The parity errors are being read by the SRAM, suspect the PA! 

******show version 
Cisco Internetwork Operating System Software 
IOS (tm) GS Software (RSP-PV-M), Version 11.1(29)CC1, EARLY DEPLOYMENT 
RELEASE SOFTWARE (fc1) 
V111_29_CC_THROTTLE_BRANCH Synced to mainline version: 11.1(29)CA 
Copyright (c) 1986-1999 by cisco Systems, Inc. 
Compiled Wed 13-Oct-99 02:21 by sharpd 
Image text-base: 0x60010910, data-base: 0x60832000 

ROM: System Bootstrap, Version 11.1(8)CA1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) 
ROM: GS Software (RSP-BOOT-M), Version 11.1(29)CC1, EARLY DEPLOYMENT 
RELEASE SOFTWARE (fc1) 

attga711c7 uptime is 27 weeks, 11 minutes 
System restarted by reload at 00:49:05 UTC Sun Oct 24 1999 
System image file is "slot0:rsp-pv-mz.111-29.CC1", booted via slot0 
Host configuration file is "cbb/al/ar-2", booted via tftp from 199.37.184.170 

cisco RSP4 (R5000) processor with 262144K/2072K bytes of memory. 
R5000 processor, Implementation 35, Revision 2.1 (512KB Level 2 Cache) 
Last reset from power-on 
G.703/E1 software, Version 1.0. 
G.703/JT2 software, Version 1.0. 
X.25 software, Version 2.0, NET2, BFE and GOSIP compliant. 
Chassis Interface. 
3 VIP2 R5K controllers (3 ATM). 
3 ATM network interfaces. 
123K bytes of non-volatile configuration memory. 

20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K). 
8192K bytes of Flash internal SIMM (Sector size 256K). 


******show diag slot 6 
Slot 6: 
 Physical slot 6, ~physical slot 0x9, logical slot 0, CBus 0 
 Microcode Status 0x4 
 WCS Loaded 
 Board is disabled analyzed wedged 
 Pending I/O Status: None 
 EEPROM format version 1 
 VIP2 R5K controller, HW rev 2.02, board revision C0 
 Serial number: 12639078  Part number: 73-2167-05 
 Test history: 0x00        RMA number: 00-00-00 
 Flags: cisco 7000 board; 7500 compatible 

 EEPROM contents (hex): 
   0x20: 01 1E 02 02 00 C0 DB 66 49 08 77 05 00 00 00 00 
   0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 

 Slot database information: 
 Flags: 0x295 Insertion time: 0x3AA4 (27w0d ago) 

 Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM 

 PA Bay 0 Information: 
  ENHANCED ATM OC3 PA (MM), 1 port 
  EEPROM format version 1 
  HW rev 2.00, Board revision A0 
  Serial number: 12366362  Part number: 73-2430-04 

 1 crash since restart. 
 Last crash context (Apr 29 2000 23:19:26): 
 Nevada Error Interrupt Register = 0x2 
 PMA error interrupt 
 PMA Error Register = 0046000000001000 
     Packet Memory Read Parity error 
     PCI master address = 0460000 
 $0 : 00000000, AT : 60179244, v0 : 601D337C, v1 : 0000AAAA 
 a0 : 604CF3E0, a1 : 604C8180, a2 : 00001182, a3 : 00000050 
 t0 : 00000800, t1 : 4E90424C, t2 : 00000001, t3 : 6014A620 
 t4 : 6016E220, t5 : 000000F8, t6 : 00000000, t7 : 00000000 
 s0 : 321735CC, s1 : 6052B508, s2 : 604C8180, s3 : 604CF3E0 
 s4 : 3226C120, s5 : 604D1440, s6 : 00000002, s7 : 00000CED 
 t8 : 34000000, t9 : 603C9930, k0 : 00000000, k1 : 00000002 
 gp : 60337700, sp : 603C0350, s8 : 00000001, ra : 601476E8 
 EPC : 601D337C, ErrorEPC : 800086B8, SREG : 3400E103 
 Cause 00000000 (Code 0x0): Interrupt exception 
 Traceback= 0x601D337C 0x601476E8 0x6014A674 

******excerpt from crashinfo 
Nevada Error Interrupt Register = 0x2 
PMA error register = 0046000000001000 
    Packet Memory Read Parity error 
    PCI master address = 0460000 

******excerpt from syslog 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 Nevada Error Interrupt Register = 0x2 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 PMA error register = 0046000000001000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6     Packet Memory Read Parity error 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6     PCI master address = 0460000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 PA Bay 0 Upstream PCI-PCI Bridge, Handle=0 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6 DEC21050 bridge chip, config=0x0 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x00): cfid   = 0x00011011 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x04): cfcs   = 0x02800147 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x08): cfccid = 0x06040002 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x0C): cfpmlt = 0x00010000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x18): cfsmlt = 0x00010100 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x1C): cfsis  = 0x02807020 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x20): cfmla  = 0x01F00000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x24): cfpmla = 0x0000FE00 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x3C): cfbc   = 0x00030000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x40): cfseed = 0x00000000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x44): cfstwt = 0x00000000 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x48): cfswac = 0x00FFFFFF 
Apr 29 23:19:13: %VIP2 R5K-1-MSG: slot6  (0x4C): cfpwac = 0x00FFFFFF 
Apr 29 23:19:26: %VIP2 R5K-1-MSG: slot6 System reloaded by a fatal hardware error

Information to Collect if You Open a TAC Case

If you still need assistance after you perform the troubleshooting steps in this document and want to open a case with Cisco Technical Support, make sure to include this information:
  • The troubleshooting you performed before you open the case
  • Output from the show technical-support command (in enable mode, if possible)
  • Output from the show log command or console captures, if possible
  • VIP crashinfo file
Attach the collected data to your case in non-zipped, plain text format (.txt). You can attach information by uploading it using the Case Query tool (registered customers only) . If you cannot access the Case Query tool, you can attach the relevant information to your case by sending it to attach@cisco.com with your case number in the subject line of your message.

Note: If possible, do not manually reload or power-cycle the router before you collect this information as this can cause important information to be lost that is needed to determine the root cause of the problem.

Related Information

Revision History

Revision Publish Date Comments
1.0
02-Oct-2006
Initial Release

Contributed by

  • michewil
    tokatsu
  • jrode
    gnadela

Was this Document Helpful?

FeedbackFeedback

Contact Cisco