This document explains the steps to troubleshoot and isolate which part or component of a Cisco 10000 Series Edge Services Router (ESR) and the Performance Routing Engine (PRE1) are failing when you identify a variety of parity error messages.
Cisco recommends that you have knowledge of these topics:
The information in this document is based on the software and hardware versions below.
Cisco 10000 Series Edge Services Routers (ESRs) using the Performance Routing Engine (PRE1)
All versions of Cisco IOS® software
Note: This document does not apply to Cisco 10720 Series Internet Routers.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Refer to Cisco Technical Tips Conventions for more information on document conventions.
The Cisco 10000 Series ESR PRE1 consists of two circuit cards: the Route Processor (RP) and the Forwarding Processor (FP) card. The flowchart below helps you determine which component of a Cisco 10000 Series ESR PRE1 is responsible for parity/Error Code Correction (ECC) error messages on the route processor.
Note: Capture and record the show tech-support output and console logs, and collect all crashinfo and pxf_crashinfo files during parity/ECC error events.
The following diagram describes the portion of the PRE1 RP architecture that can experience parity/ECC errors.
The PRE1 RP uses Single Bit Error Correction and Multi Bit Error Detection ECC (Error Code Correction) to shared memory (SDRAM). A single bit error in SDRAM is corrected automatically, and the system continues to operate as normal.
A multi-bit error in SDRAM is a fatal event, which causes a cache error exception or bus error to occur. The rest of the memory and buses in the system use single bit parity detection. Single bit errors at 1 and 3 in the diagram above are fatal and cause the router to reset.
Data with bad parity can be reported by several of the parity-checking devices for any read or write operation on the Cisco 10000 Series ESR using the PRE1.
The following is a description of the various RP error messages reported on a Cisco 10000 Series ESR with a PRE1 installed:
GT64120B SDRAM Error
The following error message is reported when a GT64120B system controller detects a multi-bit ECC error when reading SDRAM:
%ERR-1-GT64120 (PCI-0): Fatal error, Memory parity error (external) GT=0xB4000000, cause=0x0100E283, mask=0x0ED01F00, real_cause=0x00000200 bus_err_high=0x00000000, bus_err_low=0x00000000, addr_decode_err=0x00000470 %ERR-1-FATAL: Fatal error interrupt, reloading RP FPGA status 0x00000004 EPC 0x6084116C Error EPC 0xBFC00C54 BadVA 0xD6E8B233 Status 0x3400FF03
Replace the SDRAM after a second failure. If the failure persists, replace the PRE1.
GT64120B System Parity Error Master Read
Accessing either of the PCI bridges triggers a parity error in Master Read. The following is an example of a parity error message:
%ERR-1-GT64120 (PCI0):Fatal error, Parity error on master read GT=B4000000, cause=0x0110E083, mask=0x0ED01F00, real_cause=0x00100000 Bus_err_high=0x00000000, bus_err_low=0x00000000, addr_decode_err=0x00000470 %ERR-1-SERR: PCI bus system/parity error %ERR-1-FATAL: Fatal error interrupt, No reloading Err_stat=0x81, err_enable=0xFF, mgmt_event=0x40
Replace the PRE1 upon detection of these errors.
CPU Parity Error
A CPU parity error message is reported if the CPU detects a parity error when accessing the processor's external (L3 on the PRE1) cache through its SysAD bus, or either of the CPU internal cache memories (L1 or L2). Table 1 shows examples of the messages that would be printed out for each type of cache parity error:
Table 1: CPU Parity Error Location
Location of Parity Error Error Message L1 Instruction Cache Error: Primary, instr cache, fields: data L1 Data Cache Error: Primary, data cache, fields: data L2 Instruction Cache Error: SysAD, instr cache, fields: data L2 Data Cache Error: SysAD, data cache, fields: data L3 Instruction Cache Error: SysAD, instr cache, fields: 1st dword L3 Data Cache Error: SysAD, data cache, fields: 1st dword
Use Table 1 to identify the location of the parity error reported to the console of the Cisco 10000 Series ESR.
The first line of the error message indicates the location of the parity error, and can be any location listed in Table 1. In this example, the location is L3 Data Cache.
Error: SysAD, data cache, fields: data, 1st dword Physical addr(21:3) 0x195BE88, Virtual address is imprecise. Imprecise Data Parity Error Imprecise Data Parity Error
Replace the PRE1 after a second failure.
The first line of the error message indicates the location of the parity error, and can be any location listed in Table 1. In this example, the location is L3 Instruction Cache.
Error: SysAD, instr cache, fields: data, 1st dword Physical addr(21:3) 0x000000, virtual addr 0x6040BF60, vAddr(14:12) 0x3000 virtual address corresponds to main:text, cache word 0 Low Data High Data Par Low Data High Data Par L1 Data: 0:0xAE620068 0x8C830000 0x00 1:0x50400001 0xAC600004 0x01 2:0xAC800000 0x00000000 0x02 3:0x1600000B 0x00000000 0x01 Low Data High Data Par Low Data High Data Par DRAM Data: 0:0xAE620068 0x8C830000 0x00 1:0x50400001 0xAC600004 0x01 2:0xAC800000 0x00000000 0x02 3:0x1600000B 0x00000000 0x01
As in Example 1, replace the PRE1 after a second failure.
Crash due to parity error:
Mar 14 10:32:01.029 UTC: %C10K_TOASTER-3-ERROR: TCAM0 has parity error Mar 14 10:32:01.033 UTC: %C10KEVENTMGR-1-MINOR_FAULT: PXF DMA ToasterFault Mar 14 10:32:01.033 UTC: %C10KEVENTMGR-1-MINOR_REOCCURRING: PXF DMAToaster Fault
TCAM parity errors are to be expected based on the known MTBF calculations. The error message is simply a transient hardware problem.
The Forwarding Processor (FP) circuit card is the top board of the PRE1 assembly. The FP board contains three application-specific integrated circuits (ASICs), a single Backplane Interface ASIC, and two Parallel Express Forwarding (PXF) Network Processing ASICs. Each ASIC has access to external memory systems.
The following diagram helps you determine which component of a Cisco 10000 Series ESR PRE1 FP is responsible for ECC error messages.
Backplane Interface ASIC SDRAM Single Bit Error-Correcting Code (ECC) Errors
The Backplane Interface ASIC has access to ECC-protected SDRAM. Single bit errors are detected and the corrected data presented. Single bit errors are reported as follows:
%C10KEVENTMGR-1-MINOR_FAULT: PXF DMA Single Bit SDRAM Error %C10KEVENTMGR-1-SBE_DEBUG: Address: 0x01003C00, Who: 0x02, Data With ECC: 0x6E453363 2843ADAC D4 10769E 9773870C, Data w/o ECC: 0x6E453363 2843ADAC D410769E 9773870C
Single bit errors are counted and may be displayed using the show hardware pxf dma counters Cisco IOS software command. Action is usually not required for single bit errors; however, repeated or frequent instances of single bit errors are cause for replacement of the PRE1.
Backplane Interface ASIC SDRAM Multi-bit ECC Errors
When detected, Backplane Interface ASIC SDRAM multi-bit errors cause the PXF Network Processing microcode to reload, and also create a pxf_crashinfo file in bootflash. The PXF Network Processing microcode reload causes the Backplane Interface ASIC to be re-initialized, effectively scrubbing the multi-bit error from the SDRAM. The following is an example of the message printed to the console in response to a Multi-bit ECC error in Backplane Interface ASIC SDRAM:
%C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Multi-bit SDRAM Error, Restarting PXF Downloading Microcode: file=system:pxf/c10k-1-ucode.3.1.0, version=3.1.0, description=Release Software created Tue 11-Sep-01 19:25
Replace the PRE1 after a second failure.
The two PXF Network Processing ASICs have access to ECC protected SDRAM column memory, or eXternal Column Memory (XCM).
PXF Network Processing ASIC XCM Single bit ECC errors
Single bit errors are detected and the corrected data is presented. Single bit errors are counted, and the PXF Network Processing ASIC XCM single bit error count can be displayed using the show hardware pxf xcm command.
When the single bit error counter wraps, single bit errors are reported and the RP scrubs the address of the first single bit error that was detected by the PXF Network Processing ASIC. The following is an example of a message reported when a single bit error is reported:
%TOASTER-2-FAULT: T0 XCM1 SDRAM-A: ECC Single bit error counter has wrapped
The Toaster (PXF Network Processing ASIC) number and the SDRAM interface in the above message reflect the XCM interface where the single bit error from the wrapped counter was detected. Action is usually not required for single bit error counter wraps; however, repeated or frequent instances of these errors are cause for replacement of the PRE1.
PXF Network Processing ASIC XCM multi-bit ECC errors
XCM multi-bit ECC errors cannot be corrected. In systems with redundant PRE1s, XCM multi-bit errors cause a crash and a PRE failover. In systems with a single PRE1, detection of XCM multi-bit errors forces a PXF Network Processing ASIC microcode reload. The microcode reload reinitializes all PXF Network Processing ASIC XCM memories, effectively scrubbing the ECC multi-bit failure from memory. The following messages appear in the log and the pxf_crashinfo/crashinfo file:
%TOASTER-2-FAULT: T0 XCM1 SDRAM-A: Multi-bit ECC error %C10KEVENTMGR-1-MAJOR_FAULT: PXF DMA Toaster Fault, Restarting PXF Downloading Microcode: file=system:pxf/c10k-1-ucode.220.127.116.11, version=18.104.22.168,
When this happens, the error message specifies the Toaster (T0 or T1), the appropriate XCM number (1 through 4), and the SDRAM Interface (A or B) that encountered the multi-bit ECC error.
Replace the PRE1 after a second failure.
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.