Field Notice: Fabric Parity Error on GSR CSC16 Card
Available Languages
Updated February 20, 2002
January 24, 2002
Products Affected
| Product | Comments |
|---|---|
| GSR16/80-CSC= | Cisco 12016 80 Gbps GSR Scheduler/Fabric card with TAN of 800-05597-02 or lower |
Problem Description
A small number of Clock Scheduler Cards (CSCs) for the Cisco 12016 may experience hardware failures that result in grant parity errors. The failure rate in the field for this issue is less than one percent of the installed base.
On routers running Cisco IOS® Software without the fix to software bug CSCdw10748 (see DDTS section below), grant parity errors may result in a system level failure. With the fix for CSCdw10748, a router with redundant CSCs will not experience a system level disruption if this hardware failure occurs. See the Workaround/Solution section for list of Cisco IOS Software releases which have or will have the fix for CSCdw10748.
Background
- Routers running Cisco IOS Software releases earlier than 12.0(16)S7 do not switch over to the backup CSC after grant parity errors have occurred. This has been recorded as bug CSCdw10748.
- The oscillator (P/N 19-0984-01) that provides timing for the 12016 Clock Scheduler Card (800-05597-02) has been found to fail at a rate outside of the predicted mean time between failures (MTBF). This failure can result in a grant parity error. A grant parity error occurs when a Route Processor or Line Card receives an invalid response to a request for access to the switch fabric.
Problem Symptoms
The failure symptoms are grant parity or request parity errors reported by multiple line cards. The grant parity or request parity errors will be seen in the outputs of the show log or show controller fia commands as shown.
An example of a show log output follows:
Sep 22 03:00:56.674 JST: %FABRIC-3-PARITYERR: To Fabric parity error was detected. Grant parity error Data = 0x2. SLOT 1:Sep 22 03:00:56.674 JST: %FABRIC-3-PARITYERR: To Fabric parity error was detected. Grant parity error Data = 0x1
An example of the show controller fia output follows.
12016_Router#show controller fia
Fabric configuration: Full bandwidth redundant
Master Scheduler: Slot 17 Backup Scheduler: Slot 16
From Fabric FIA Errors
-----------------------
redund fifo parity 0 redund overflow 0 cell drops 116
crc32 lkup parity 0 cell parity 0 crc32 0
Switch cards present 0x001F Slots 16 17 18 19 20
Switch cards monitored 0x001F Slots 16 17 18 19 20
Slot: 16 17 18 19 20
Name: csc0 csc1 sfc0 sfc1 sfc2
------ ------ ------ ------ ------
los 0 1 0 0 0
state Off Off Off Off Off
crc16 1167 402 1167 1167 1167
To Fabric FIA Errors
-----------------------
sca not pres 0 req error xx uni FIFO overflow 0
grant parity xx multi req 0 uni FIFO undrflow 0
cntrl parity 0 uni req 0 crc32 lkup parity 0
multi FIFO 0 empty dst req 0 handshake error 0
cell parity 0
The output of the show controller fia command can be used to determine if multiple line cards are reporting these errors and if a CSC switchover has taken place. In order to get this output from a specific line card, first type attach -slot nn- and then execute the show controller fia command after the LC-Slot prompt appears.
Note: The command execute-on show controller fia should not be used, since in the event that the Cisco IOS Software is unable to handle this error, this command will fail.
If multiple line cards are reporting grant parity or request parity errors and the router is still functioning, a CSC switchover has occurred. The failed CSC is the one that is currently the backup CSC (not the one listed as Master Scheduler in the show controller fia output).
If Halted is next to the heading From Fabric FIA Errors or To Fabric FIA Errors or if the router is no longer forwarding traffic, then a CSC switchover has not occurred and the failing CSC is the one listed as Master Scheduler. By default, the CSC in slot 17 is the primary and the CSC in slot 16 is the backup.
It is not likely that the CSC oscillator has failed if only one board is reporting grant parity or request parity errors.
Workaround/Solution
The recommended solution to this issue is to utilize the software fix which assures that a CSC switchover does take place should an oscillator failure occur in the master CSC and grant parity errors result. Routers with a CSC at risk for this hardware failure should be upgraded to a version of Cisco IOS Software with a fix to CSCdw10748.
Note: The version of Cisco IOS Software currently running on a system can be seen by the show version command as follows.
Router-12016a#show version Cisco Internetwork Operating System Software IOS (tm) GS Software (GSR-P-M), Version 12.0(18)S1, EARLY DEPLOYMENT RELEASE SO) TAC Support: http://www.cisco.com/tac Copyright (c) 1986-2001 by cisco Systems, Inc.
The fix to CSCdw10748 will be implemented in Cisco IOS Software Releases 12.0(17)ST4, 12.0(21)S, 12.0(21)ST and all of the images shown in the First Fixed-in Version field of CSCdw10748.
An alternative solution to this issue addresses the CSC hardware. There are two options for this.
- Replace the GSR16/80-CSC board with one that has the oscillator changes as addressed in ECO E046599. Refer to How to Identify Hardware Levels below to determine if you have a suspect CSC card. An exchange program has been set up for this purpose. Customers who wish to replace one or more of their CSC boards which are affected by the hardware problem described in this field notice should use fhe form below to request an exchange.
- Upgrade the Cisco 12016 2.5G fabric to the 10G fabric using the Technology Migration Program (TMP). See the following URL for information on TMP. http://www.cisco.com/offer/tic/TMP_PA.html. The TMP allows a trade in of a full 2.5G fabric towards the purchase of a full 10G fabric. For a limited time and for this trade combination only, Cisco has improved the normal trade-in credit. Please contact a Cisco sales representative for more information on the TMP.
DDTS
To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.
| DDTS | Description |
|---|---|
| CSCdw10748 | CSC switchover does not happen after a grant parity error |
How To Identify Hardware Levels
The top level assembly number (TAN) version of the GSR16/80-CSC card was changed from 800-05597-02 to 800-05597-03 with ECO E046599. Cards with a TAN of 800-05597-03 or higher are not subject to the information in this field notice. Cards with a TAN lower than 800-05597-03 AND with a deviation number of D041570 are also not subject to the information in this field notice.
A show diag command can be used to see the TAN and the deviation of the clock scheduler card in a system. The CSC in the example below is subject to the information in this field notice. Its TAN is version -01 and it does not have any deviation applied to it.
ROUTER-12016#show diag 16 SLOT 16 (CSC 0 ): Clock Scheduler Card 16XOC48 MAIN: type 23, 800-5597-01 rev A0 dev 0 HW config: 0x00 SW key: 00-00-00 PCA: 73-4196-01 rev A0 ver 1 HW version 1.0 S/N SDK043214FS MBUS: MBUS Agent (1) 73-2146-07 rev B0 dev 0 HW version 1.2 S/N CAB0429AVSQ Test hist: 0x00 RMA#: 00-00-00 RMA hist: 0x00 DIAG: Test count: 0x00000000 Test results: 0x00000000 FRU: Linecard/Module: GSR16/80-CSC= MBUS Agent Software version 01.46 (RAM) (ROM version is 02.10) Using CAN Bus A Primary clock is CSC 1
Upgrade Program
CSC16 Exchange - oscillator issueIf you have any questions concerning the upgrade program, you can e-mail upgrades-info@cisco.com.
Note: Fields marked with an asterisk (*) are required fields.
2 When entering more than one serial number in the serial number text entry field, be sure to type a comma between each serial number to separate them.
For More Information
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)