Guest

Cisco 10000 Series Routers

10000 ESR PRE1 Parity Error Fault Tree

Cisco - 10000 ESR PRE1 Parity Error Fault Tree

Document ID: 25701

Updated: Jul 07, 2005

   Print

Introduction

This document explains the steps to troubleshoot and isolate which part or component of a Cisco 10000 Series Edge Services Router (ESR) and the Performance Routing Engine (PRE1) are failing when you identify a variety of parity error messages.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Components Used

The information in this document is based on the software and hardware versions below.

  • Cisco 10000 Series Edge Services Routers (ESRs) using the Performance Routing Engine (PRE1)

  • All versions of Cisco IOS® software

Note: This document does not apply to Cisco 10720 Series Internet Routers.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

PRE1 Route Processor Parity Error Fault Tree Analysis

The Cisco 10000 Series ESR PRE1 consists of two circuit cards: the Route Processor (RP) and the Forwarding Processor (FP) card. The flowchart below helps you determine which component of a Cisco 10000 Series ESR PRE1 is responsible for parity/Error Code Correction (ECC) error messages on the route processor.

esr10000a.gif

Note: Capture and record the show tech-support output and console logs, and collect all crashinfo and pxf_crashinfo files during parity/ECC error events.

PRE1 Parity/ECC Detection

The following diagram describes the portion of the PRE1 RP architecture that can experience parity/ECC errors.

esr10000b.gif

The PRE1 RP uses Single Bit Error Correction and Multi Bit Error Detection ECC (Error Code Correction) to shared memory (SDRAM). A single bit error in SDRAM is corrected automatically, and the system continues to operate as normal.

A multi-bit error in SDRAM is a fatal event, which causes a cache error exception or bus error to occur. The rest of the memory and buses in the system use single bit parity detection. Single bit errors at 1 and 3 in the diagram above are fatal and cause the router to reset.

Parity/ECC Errors in the Cisco 10000 Series ESR Route Processor

Data with bad parity can be reported by several of the parity-checking devices for any read or write operation on the Cisco 10000 Series ESR using the PRE1.

The following is a description of the various RP error messages reported on a Cisco 10000 Series ESR with a PRE1 installed:

  • GT64120B SDRAM Error

    The following error message is reported when a GT64120B system controller detects a multi-bit ECC error when reading SDRAM:

    %ERR-1-GT64120 (PCI-0): Fatal error, Memory parity error (external)
     GT=0xB4000000, cause=0x0100E283, mask=0x0ED01F00, real_cause=0x00000200
     bus_err_high=0x00000000, bus_err_low=0x00000000, addr_decode_err=0x00000470
     %ERR-1-FATAL: Fatal error interrupt, reloading
     RP FPGA status 0x00000004
     EPC 0x6084116C Error EPC 0xBFC00C54 BadVA 0xD6E8B233
     Status 0x3400FF03
    

    Replace the SDRAM after a second failure. If the failure persists, replace the PRE1.

  • GT64120B System Parity Error Master Read

    Accessing either of the PCI bridges triggers a parity error in Master Read. The following is an example of a parity error message:

    %ERR-1-GT64120 (PCI0):Fatal error, Parity error on master read
    GT=B4000000, cause=0x0110E083, mask=0x0ED01F00, real_cause=0x00100000
    Bus_err_high=0x00000000, bus_err_low=0x00000000,
    addr_decode_err=0x00000470
     
    %ERR-1-SERR: PCI bus system/parity error
    %ERR-1-FATAL: Fatal error interrupt, No reloading
    Err_stat=0x81, err_enable=0xFF, mgmt_event=0x40
    

    Replace the PRE1 upon detection of these errors.

  • CPU Parity Error

    A CPU parity error message is reported if the CPU detects a parity error when accessing the processor's external (L3 on the PRE1) cache through its SysAD bus, or either of the CPU internal cache memories (L1 or L2). Table 1 shows examples of the messages that would be printed out for each type of cache parity error:

    Table 1: CPU Parity Error Location

    Location of Parity Error Error Message
    L1 Instruction Cache Error: Primary, instr cache, fields: data
    L1 Data Cache Error: Primary, data cache, fields: data
    L2 Instruction Cache Error: SysAD, instr cache, fields: data
    L2 Data Cache Error: SysAD, data cache, fields: data
    L3 Instruction Cache Error: SysAD, instr cache, fields: 1st dword
    L3 Data Cache Error: SysAD, data cache, fields: 1st dword

    Use Table 1 to identify the location of the parity error reported to the console of the Cisco 10000 Series ESR.

Example 1:

The first line of the error message indicates the location of the parity error, and can be any location listed in Table 1. In this example, the location is L3 Data Cache.

Error: SysAD, data cache, fields: data, 1st dword
Physical addr(21:3) 0x195BE88,
Virtual address is imprecise.
    Imprecise Data Parity Error
    Imprecise Data Parity Error

Replace the PRE1 after a second failure.

Example 2:

The first line of the error message indicates the location of the parity error, and can be any location listed in Table 1. In this example, the location is L3 Instruction Cache.

Error: SysAD, instr cache, fields: data, 1st dword
Physical addr(21:3) 0x000000,
virtual addr 0x6040BF60, vAddr(14:12) 0x3000
virtual address corresponds to main:text, cache word 0
 
          Low Data     High Data  Par  Low Data     High Data  Par
L1 Data:  0:0xAE620068 0x8C830000 0x00 1:0x50400001 0xAC600004 0x01
          2:0xAC800000 0x00000000 0x02 3:0x1600000B 0x00000000 0x01
           Low Data     High Data  Par  Low Data     High Data  Par
DRAM Data: 0:0xAE620068 0x8C830000 0x00 1:0x50400001 0xAC600004 0x01
           2:0xAC800000 0x00000000 0x02 3:0x1600000B 0x00000000 0x01

As in Example 1, replace the PRE1 after a second failure.

  • Crash due to parity error:

    Mar 14 10:32:01.029 UTC: %C10K_TOASTER-3-ERROR: TCAM0 has parity error
    Mar 14 10:32:01.033 UTC: %C10KEVENTMGR-1-MINOR_FAULT: PXF DMA ToasterFault
    Mar 14 10:32:01.033 UTC: %C10KEVENTMGR-1-MINOR_REOCCURRING: PXF DMAToaster Fault 

    TCAM parity errors are to be expected based on the known MTBF calculations. The error message is simply a transient hardware problem.

PRE1 Fast Packet ECC Fault Tree Analysis

The Forwarding Processor (FP) circuit card is the top board of the PRE1 assembly. The FP board contains three application-specific integrated circuits (ASICs), a single Backplane Interface ASIC, and two Parallel Express Forwarding (PXF) Network Processing ASICs. Each ASIC has access to external memory systems.

The following diagram helps you determine which component of a Cisco 10000 Series ESR PRE1 FP is responsible for ECC error messages.

esr10000c.gif

Backplane Interface ASIC SDRAM ECC Errors

  • Backplane Interface ASIC SDRAM Single Bit Error-Correcting Code (ECC) Errors

    The Backplane Interface ASIC has access to ECC-protected SDRAM. Single bit errors are detected and the corrected data presented. Single bit errors are reported as follows:

    %C10KEVENTMGR-1-MINOR_FAULT: PXF DMA Single Bit SDRAM Error
    %C10KEVENTMGR-1-SBE_DEBUG: Address: 0x01003C00, 
    Who: 0x02, Data With ECC: 0x6E453363 2843ADAC D4
    
    10769E 9773870C, Data w/o ECC: 0x6E453363 2843ADAC D410769E 9773870C

    Single bit errors are counted and may be displayed using the show hardware pxf dma counters Cisco IOS software command. Action is usually not required for single bit errors; however, repeated or frequent instances of single bit errors are cause for replacement of the PRE1.

  • Backplane Interface ASIC SDRAM Multi-bit ECC Errors

    When detected, Backplane Interface ASIC SDRAM multi-bit errors cause the PXF Network Processing microcode to reload, and also create a pxf_crashinfo file in bootflash. The PXF Network Processing microcode reload causes the Backplane Interface ASIC to be re-initialized, effectively scrubbing the multi-bit error from the SDRAM. The following is an example of the message printed to the console in response to a Multi-bit ECC error in Backplane Interface ASIC SDRAM:

    %C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Multi-bit SDRAM Error, Restarting PXF
     Downloading Microcode: file=system:pxf/c10k-1-ucode.3.1.0,
     version=3.1.0, description=Release Software created Tue 11-Sep-01 19:25
    

    Replace the PRE1 after a second failure.

PXF Network Processing ASIC Column Memory ECC Errors

The two PXF Network Processing ASICs have access to ECC protected SDRAM column memory, or eXternal Column Memory (XCM).

  • PXF Network Processing ASIC XCM Single bit ECC errors

    Single bit errors are detected and the corrected data is presented. Single bit errors are counted, and the PXF Network Processing ASIC XCM single bit error count can be displayed using the show hardware pxf xcm command.

    When the single bit error counter wraps, single bit errors are reported and the RP scrubs the address of the first single bit error that was detected by the PXF Network Processing ASIC. The following is an example of a message reported when a single bit error is reported:

    %TOASTER-2-FAULT: T0 XCM1 SDRAM-A: ECC Single bit error counter has wrapped

    The Toaster (PXF Network Processing ASIC) number and the SDRAM interface in the above message reflect the XCM interface where the single bit error from the wrapped counter was detected. Action is usually not required for single bit error counter wraps; however, repeated or frequent instances of these errors are cause for replacement of the PRE1.

  • PXF Network Processing ASIC XCM multi-bit ECC errors

    XCM multi-bit ECC errors cannot be corrected. In systems with redundant PRE1s, XCM multi-bit errors cause a crash and a PRE failover. In systems with a single PRE1, detection of XCM multi-bit errors forces a PXF Network Processing ASIC microcode reload. The microcode reload reinitializes all PXF Network Processing ASIC XCM memories, effectively scrubbing the ECC multi-bit failure from memory. The following messages appear in the log and the pxf_crashinfo/crashinfo file:

    %TOASTER-2-FAULT: T0 XCM1 SDRAM-A: Multi-bit ECC error
    %C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Toaster Fault, Restarting PXF
     Downloading Microcode: file=system:pxf/c10k-1-ucode.102.3.0.0, version=102.3.0.0,
    

    When this happens, the error message specifies the Toaster (T0 or T1), the appropriate XCM number (1 through 4), and the SDRAM Interface (A or B) that encountered the multi-bit ECC error.

    Replace the PRE1 after a second failure.

Related Information

Updated: Jul 07, 2005
Document ID: 25701