Guest

Cisco 10000 Series Routers

10000 ESR PRE2 Parity Error Fault Tree

Cisco - 10000 ESR PRE2 Parity Error Fault Tree

Document ID: 44228

Updated: Dec 18, 2007

   Print

Introduction

This document explains the steps to troubleshoot and isolate which component of a Cisco 10000 Series Edge Services Router (ESR) with the Performance Routing Engine (PRE2) is failing when you identify a variety of parity error messages.

Prerequisites

Requirements

Readers of this document should be knowledgeable of the following:

Components Used

The information in this document is based on the following software and hardware versions:

  • Cisco 10000 Series ESRs using the PRE2

  • All versions of Cisco IOS?? Software

Note: This document does not apply to Cisco 10720 Series Internet Routers.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

PRE2 Route Processor Parity Error Fault Tree Analysis

The Cisco 10000 Series ESR PRE2 consists of two circuit cards: the Route Processor (RP) and the Forwarding Processor (FP). The flowchart below can help you determine which component of an ESR PRE2 is responsible for parity or Error-Correcting Code (ECC) error messages on the route processor.

esr10000a.gif

Note: Capture and record the show tech-support command output and console logs, and collect all crashinfo and pxf_crashinfo files during parity or ECC error events.

PRE2 Parity and ECC Detection

The following diagram describes the portion of the PRE2 RP architecture that can experience parity or ECC errors.

esr10000b.gif

The PRE2 RP uses single-bit error (SBE) correction and multi-bit error (MBE) detection ECC to shared memory (SDRAM). An SBE in SDRAM is corrected automatically, and the system continues to operate as normal.

An MBE in SDRAM is a fatal event, which causes a cache error exception or bus error to occur. The rest of the memory and buses in the system use single-bit parity detection. SBEs at 1 and 3 in the diagram above are fatal and cause the router to reset.

Parity and ECC Errors in the Cisco 10000 Series ESR Route Processor

Data with bad parity can be reported by several of the parity-checking devices for any read or write operation on the Cisco ESR PRE2.

The following is a description of the various RP error messages reported on an ESR with a PRE2 installed:

  • GT64120B SDRAM Error

    The following error message is reported when a GT64120B system controller detects a multi-bit ECC error when reading SDRAM:

    %ERR-1-GT64120 (PCI-0): Fatal error, Memory parity error (external)
     GT=0xB4000000, cause=0x0100E283, mask=0x0ED01F00, real_cause=0x00000200
     bus_err_high=0x00000000, bus_err_low=0x00000000, addr_decode_err=0x00000470
     %ERR-1-FATAL: Fatal error interrupt, reloading
     RP FPGA status 0x00000004
     EPC 0x6084116C Error EPC 0xBFC00C54 BadVA 0xD6E8B233
     Status 0x3400FF03
    

    Replace the PRE2 after a second failure.

  • GT64120B System Parity Error Master Read

    Accessing either of the PCI buses triggers a parity error in master read. The following is an example of a parity error message:

    %ERR-1-GT64120 (PCI0):Fatal error, Parity error on master read
    GT=B4000000, cause=0x0110E083, mask=0x0ED01F00, real_cause=0x00100000
    Bus_err_high=0x00000000, bus_err_low=0x00000000,
    addr_decode_err=0x00000470
     
    %ERR-1-SERR: PCI bus system/parity error
    %ERR-1-FATAL: Fatal error interrupt, No reloading
    Err_stat=0x81, err_enable=0xFF, mgmt_event=0x40
    

    Replace the PRE2 upon detection of these errors.

  • CPU Parity Error

    A CPU parity error message is reported if the CPU detects a parity error when accessing the processor's external cache (Layer 3 [L3] on the PRE2) through its SysAD bus, or when accessing either of the CPU internal cache memories (Layer 1 [L1] or Layer 2 [L2]). The table below shows examples of the messages printed for each type of cache parity error.

    Location of Parity Error Error Message
    L1 Instruction Cache Error: Primary, instr cache, fields: data
    L1 Data Cache Error: Primary, data cache, fields: data
    L2 Instruction Cache Error: SysAD, instr cache, fields: data
    L2 Data Cache Error: SysAD, data cache, fields: data
    L3 Instruction Cache Error: SysAD, instr cache, fields: 1st dword
    L3 Data Cache Error: SysAD, data cache, fields: 1st dword

    Use the table above to identify the location of the parity error reported to the console of the Cisco 10000 Series ESR.

Example 1:

The first line of the error message indicates the location of the parity error, and can be any location listed in the table above. In this example, the location is the L3 Data Cache.

Error: SysAD, data cache, fields: data, 1st dword
Physical addr(21:3) 0x195BE88,
Virtual address is imprecise.
    Imprecise Data Parity Error
    Imprecise Data Parity Error

Replace the PRE2 after a second failure.

Example 2:

The first line of the error message indicates the location of the parity error, and can be any location listed in the table above. In this example, the location is the L3 Instruction Cache.

Error: SysAD, instr cache, fields: data, 1st dword
Physical addr(21:3) 0x000000,
virtual addr 0x6040BF60, vAddr(14:12) 0x3000
virtual address corresponds to main:text, cache word 0
 
          Low Data     High Data  Par  Low Data     High Data  Par
L1 Data:  0:0xAE620068 0x8C830000 0x00 1:0x50400001 0xAC600004 0x01
          2:0xAC800000 0x00000000 0x02 3:0x1600000B 0x00000000 0x01
           Low Data     High Data  Par  Low Data     High Data  Par
DRAM Data: 0:0xAE620068 0x8C830000 0x00 1:0x50400001 0xAC600004 0x01
           2:0xAC800000 0x00000000 0x02 3:0x1600000B 0x00000000 0x01

As in Example 1, replace the PRE2 after a second failure.

PRE2 Fast Packet ECC Fault Tree Analysis

The FP circuit card is the top board of the PRE2 assembly. The FP board contains five application-specific integrated circuits (ASICs), a single Backplane Interface ASIC, and four Parallel Express Forwarding (PXF) Network Processing ASICs. Each ASIC has access to external memory systems.

The following diagram helps you determine which component of a Cisco 10000 Series ESR PRE2 FP is responsible for ECC error messages:

esr10000c.gif

Backplane Interface ASIC DDR FCRAM ECC Errors

The Backplane Interface ASIC has access to two different ECC-protected Double Data Rate (DDR) Fast Cycle RAM (FCRAM) memories, External Packet Memory (EPM) and External Control Memory (ECM).

  • Backplane Interface ASIC DDR FCRAM Single-bit ECC Errors

    SBEs are detected and the corrected data presented.

    Single-bit EPM errors are reported as follows:

      %C10KEVENTMGR-1-MINOR_FAULT: PXF DMA Single Bit PMC (EPM) Error 
      %C10KEVENTMGR-1-PMC_SBE_DEBUG: Address: 0x0FFE4608, Who: 0x02
      Error taken in: Check bits, bit number: 0, Check byte value = 0x58
      Errant Data:    0x00008F00 80350000
      Corrected Data: 0x00008F00 80350000
    

    Single-bit ECM errors are reported as follows:

      %C10KEVENTMGR-1-MINOR_FAULT: PXF DMA Error - Correctable ECM Error 
      %C10KEVENTMGR-1-ECM_SBE_DEBUG: Address: 0x013FD0A8, Who: 0x01
      Error taken in:  Data bits, bit number: 32, Check byte value = 0x67
      Errant Data:    0x67CFFE58 00000000
      Corrected Data: 0x00CFFE59 00000000
    

    SBEs are counted and may be displayed by issuing the show pxf dma counters command. Action is usually not required for SBEs; however, repeated or frequent instances of these errors are cause for replacement of the PRE2.

  • Backplane Interface ASIC DDR FCRAM Multi-bit ECC Errors

    When detected, Backplane Interface ASIC DDR FCRAM MBEs cause the PXF Network Processing microcode to reload, and also create a pxf_crashinfo file in bootflash. The PXF Network Processing microcode reload causes the Backplane Interface ASIC to be reinitialized, effectively scrubbing the MBE from the DDR FCRAM.

    The following is an example of the message printed to the console in response to an EPM multi-bit ECC error in Backplane Interface ASIC DDR FCRAM:

      %C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Multi-bit PMC (EPM) Error 
    Downloading Microcode: file=system:pxf/c10k2-11-ucode.106.1.0.0, version=106.1.0.0,
     description=Release Software created Tue 03-Jun-03 00:57
    

    Replace the PRE2 after a second failure.

    The following is an example of the message printed to the console in response to an ECM multi-bit ECC error in Backplane Interface ASIC DDR FCRAM:

      %C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Error - Uncorrectable ECM Error
    Downloading Microcode: file=system:pxf/c10k2-11-ucode.106.1.0.0, version=106.1.0.0,
     description=Release Software created Tue 03-Jun-03 00:57

    Replace the PRE2 after a second failure.

PXF Network Processing ASIC Column Memory ECC Errors

The four PXF Network Processing ASICs have access to ECC-protected DDR FCRAM column memory, or External Column Memory (XCM).

  • PXF Network Processing ASIC XCM Single-bit ECC Errors

    SBEs are detected and the corrected data is presented. SBEs are counted, and the PXF Network Processing ASIC XCM SBE count can be displayed by issuing the show pxf xcm command.

    When the SBE counter wraps, SBEs are reported and the RP scrubs the address of the first SBE that was detected by the PXF Network Processing ASIC. The following is an example of a message reported when an SBE is reported:

    %C10KEVENTMGR-1-MINOR_FAULT: T0 XCM1 FCRAM-A: Too many Toaster XCM ECC single bit errors

    The Toaster (PXF Network Processing ASIC) number and the DDR FCRAM interface in the preceding message reflect the XCM interface where the SBE from the wrapped counter was detected. The error message above indicates that the error occurred on Toaster 0, XCM 1, Interface A. Action is usually not required for SBE counter wraps; however, repeated or frequent instances of these errors are cause for replacement of the PRE2.

  • PXF Network Processing ASIC XCM Multi-bit ECC Errors

    XCM multi-bit ECC errors cannot be corrected. In systems with redundant PRE2s, XCM MBEs cause a crash and a PRE failover. In systems with a single PRE2, detection of XCM MBEs forces a PXF Network Processing ASIC microcode reload. The microcode reload reinitializes all PXF Network Processing ASIC XCM memories, effectively scrubbing the ECC MBE from memory. The following messages appear in the log and the crashinfo or pxf_crashinfo file:

      %PXF-2-FAULT: T3 XCM1 FCRAM-D: Multi-bit ECC error on bits [0:31]
    %C10KEVENTMGR-4-PXF_CRASHINFO: Writing PXF debug information to 
     bootflash:pxf_crashinfo_20030729-153845.
      %C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Toaster Fault, Restarting PXF
    00:08:01: Downloading Microcode: file=system:pxf/c10k2-11-ucode.6.1.0.0,
     version=6.1.0.0, description=Release Software created Mon 21-Jul-03 12:17
    

    When this happens, the error message specifies the Toaster (T0, T1, T2, or T3), the appropriate XCM number (0 or 1), and the DDR FCRAM interface (A, B, C, or D) that encountered the multi-bit ECC error. The error message above indicates that the failure was on Toaster 2, XCM 1, Interface B.

    Replace the PRE2 after a second failure.

Related Information

Updated: Dec 18, 2007
Document ID: 44228