Guest

Cisco 12000 Series Routers

Field Notice: Fabric Parity Error on GSR CSC16 Card


Updated February 20, 2002
January 24, 2002



Products Affected

Product Comments
GSR16/80-CSC=   Cisco 12016 80 Gbps GSR Scheduler/Fabric card with TAN of 800-05597-02 or lower  

Problem Description

A small number of Clock Scheduler Cards (CSCs) for the Cisco 12016 may experience hardware failures that result in grant parity errors. The failure rate in the field for this issue is less than one percent of the installed base.

On routers running Cisco IOS® Software without the fix to software bug CSCdw10748 (see DDTS section below), grant parity errors may result in a system level failure. With the fix for CSCdw10748, a router with redundant CSCs will not experience a system level disruption if this hardware failure occurs. See the Workaround/Solution section for list of Cisco IOS Software releases which have or will have the fix for CSCdw10748.

Background

  1. Routers running Cisco IOS Software releases earlier than 12.0(16)S7 do not switch over to the backup CSC after grant parity errors have occurred. This has been recorded as bug CSCdw10748.

  2. The oscillator (P/N 19-0984-01) that provides timing for the 12016 Clock Scheduler Card (800-05597-02) has been found to fail at a rate outside of the predicted mean time between failures (MTBF). This failure can result in a grant parity error. A grant parity error occurs when a Route Processor or Line Card receives an invalid response to a request for access to the switch fabric.

Problem Symptoms

The failure symptoms are grant parity or request parity errors reported by multiple line cards. The grant parity or request parity errors will be seen in the outputs of the show log or show controller fia commands as shown.

An example of a show log output follows:

Sep 22 03:00:56.674 JST: %FABRIC-3-PARITYERR: To Fabric parity error was detected. Grant parity error Data = 0x2. 
SLOT 1:Sep 22 03:00:56.674 JST: %FABRIC-3-PARITYERR: To Fabric parity error was detected. Grant parity error Data = 0x1 

An example of the show controller fia output follows.

12016_Router#show controller fia
Fabric configuration: Full bandwidth redundant
Master Scheduler: Slot 17       Backup Scheduler: Slot 16

From Fabric FIA Errors
-----------------------
redund fifo parity 0   redund overflow 0    cell drops 116
crc32 lkup parity  0   cell parity     0    crc32      0
Switch cards present    0x001F    Slots 16 17 18 19 20
Switch cards monitored  0x001F    Slots 16 17 18 19 20
Slot:     16         17         18        19         20
Name:    csc0       csc1       sfc0      sfc1       sfc2
        ------     ------     ------    ------     ------
los     0          1          0         0          0
state   Off        Off        Off       Off        Off
crc16   1167       402        1167      1167       1167

To Fabric FIA Errors
-----------------------
sca not pres 0          req error     xx     uni FIFO overflow 0
grant parity xx         multi req     0      uni FIFO undrflow 0
cntrl parity 0          uni req       0      crc32 lkup parity 0
multi FIFO   0          empty dst req 0      handshake error   0
cell parity  0 

The output of the show controller fia command can be used to determine if multiple line cards are reporting these errors and if a CSC switchover has taken place. In order to get this output from a specific line card, first type attach -slot nn- and then execute the show controller fia command after the LC-Slot prompt appears.

Note: The command execute-on show controller fia should not be used, since in the event that the Cisco IOS Software is unable to handle this error, this command will fail.

If multiple line cards are reporting grant parity or request parity errors and the router is still functioning, a CSC switchover has occurred. The failed CSC is the one that is currently the backup CSC (not the one listed as Master Scheduler in the show controller fia output).

If Halted is next to the heading From Fabric FIA Errors or To Fabric FIA Errors or if the router is no longer forwarding traffic, then a CSC switchover has not occurred and the failing CSC is the one listed as Master Scheduler. By default, the CSC in slot 17 is the primary and the CSC in slot 16 is the backup.

It is not likely that the CSC oscillator has failed if only one board is reporting grant parity or request parity errors.

Workaround/Solution

The recommended solution to this issue is to utilize the software fix which assures that a CSC switchover does take place should an oscillator failure occur in the master CSC and grant parity errors result. Routers with a CSC at risk for this hardware failure should be upgraded to a version of Cisco IOS Software with a fix to CSCdw10748.

Note: The version of Cisco IOS Software currently running on a system can be seen by the show version command as follows.

Router-12016a#show version
Cisco Internetwork Operating System Software
IOS (tm) GS Software (GSR-P-M), Version 12.0(18)S1, EARLY DEPLOYMENT RELEASE SO)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2001 by cisco Systems, Inc.

The fix to CSCdw10748 will be implemented in Cisco IOS Software Releases 12.0(17)ST4, 12.0(21)S, 12.0(21)ST and all of the images shown in the First Fixed-in Version field of CSCdw10748.

An alternative solution to this issue addresses the CSC hardware. There are two options for this.

  1. Replace the GSR16/80-CSC board with one that has the oscillator changes as addressed in ECO E046599. Refer to How to Identify Hardware Levels below to determine if you have a suspect CSC card. An exchange program has been set up for this purpose. Customers who wish to replace one or more of their CSC boards which are affected by the hardware problem described in this field notice should use fhe form below to request an exchange.

  2. Upgrade the Cisco 12016 2.5G fabric to the 10G fabric using the Technology Migration Program (TMP). See the following URL for information on TMP. http://www.cisco.com/offer/tic/TMP_PA.html. The TMP allows a trade in of a full 2.5G fabric towards the purchase of a full 10G fabric. For a limited time and for this trade combination only, Cisco has improved the normal trade-in credit. Please contact a Cisco sales representative for more information on the TMP.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

DDTS Description
CSCdw10748 CSC switchover does not happen after a grant parity error
> CSC switchover does not happen after a grant parity error <

How To Identify Hardware Levels

The top level assembly number (TAN) version of the GSR16/80-CSC card was changed from 800-05597-02 to 800-05597-03 with ECO E046599. Cards with a TAN of 800-05597-03 or higher are not subject to the information in this field notice. Cards with a TAN lower than 800-05597-03 AND with a deviation number of D041570 are also not subject to the information in this field notice.

A show diag command can be used to see the TAN and the deviation of the clock scheduler card in a system. The CSC in the example below is subject to the information in this field notice. Its TAN is version -01 and it does not have any deviation applied to it.

ROUTER-12016#show diag 16
SLOT 16 (CSC 0 ): Clock Scheduler Card 16XOC48
MAIN: type 23, 800-5597-01 rev A0 dev 0
HW config: 0x00 SW key: 00-00-00
PCA: 73-4196-01 rev A0 ver 1
HW version 1.0 S/N SDK043214FS
MBUS: MBUS Agent (1) 73-2146-07 rev B0 dev 0
HW version 1.2 S/N CAB0429AVSQ
Test hist: 0x00 RMA#: 00-00-00 RMA hist: 0x00
DIAG: Test count: 0x00000000 Test results: 0x00000000
FRU: Linecard/Module: GSR16/80-CSC=
MBUS Agent Software version 01.46 (RAM) (ROM version is 02.10)
Using CAN Bus A
Primary clock is CSC 1

Upgrade Program

CSC16 Exchange - oscillator issue

If you have any questions concerning the upgrade program, you can e-mail upgrades-info@cisco.com.

Note: Fields marked with an asterisk (*) are required fields.

Company Information
*Organization
*Address
Address_line2
*City
*State/Province
*ZIP/Postal Code
Country
Product
Product *Quantity
GSR16/80-CSC=
Contact Information
*First Name
*Last Name
*Phone 1 Ext.
Fax 1 Ext.
*Internet E-Mail
Please use the following format: user@domain.com
*Upgrade Order Reference Number
Please provide a number that you can use when inquiring about order status
*Serial Numbers of Affected Units 2
1 For phone and fax, include 011 and the country code outside North America.

2 When entering more than one serial number in the serial number text entry field, be sure to type a comma between each serial number to separate them.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods: