Guest

Cisco 12000 Series Routers

Field Notice: FN - 62783 - GSR PRP-1 and PRP-2 - Unable To Read PCMCIA ATA Flash (64M, 128M, 256M, 512M) - Data Loss or System Boot Hang - Replace PCMCIA ATA Flash- Workaround Fix on Failure


September 25, 2009


NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Products Affected Top Assembly Printed Circuit Assembly Comments
Part Number Revision Part Number Revision
MEM-12KRP-FD128M
16-2680-01
A0
NA
NA
-A0 or earlier
MEM-12KRP-FD256M
16-3129-01
A0
NA
NA
-A0 or earlier
MEM-12KRP-FD512M
16-3150-01
A0
NA
NA
-A0 or earlier
MEM-12KRP-FD64M
16-2756-01
A0
NA
NA
-A0 or earlier

 

Problem Description

The PCMCIA ATA Flash Disks used in specific Cisco products may contain a Manufacturing defect which reduces reliability of the Flash device. PRP-1 and PRP-2 cards used in the Cisco GSR platforms that may contain a suspect Flash device may not be able to access stored files on disk 0/1. The router may fail to reboot if it relies on an IOS image stored on the Flash device.

A portion of the PRP-1 and PRP-2 cards manufactured between Jan 2006 and Oct 2006 may have shipped with an affected PCMCIA ATA Flash disk from specific component lots. The serial number of the linecards shipped to customers are listed in the How to Identify Serial Numbers section and are eligible for a PCMCIA ATA Flash upgrade.

Refer to the How to Identify Hardware Levels section for Flash disk identification information.


The failure condition would be observed during a power cycle, reload event or during a write event such as a crash file dump. Non-redundant systems will be unable to pass traffic in the failed state. Redundant systems will switch to the backup PRP.

GSR does not allow ROMMON upgrades via software and therefore the affected Flash devices are recommended for replacement.

NOTE 1 - The GSR linecard or router is not defective; this issue is only related to the removable Flash Disk cards.

NOTE 2 - As customers may remove and re-use Flash Cards, the affected cards shipped with these products may have been re-used in different Cisco products. Please exercise caution when re-using Flash cards and visually check them against the Identification Section data provided.

Background

This issue affects specific subset of the PCMCIA ATA Disks and Compact Flash disks that were installed in specific Cisco routers manufactured between May 2005 and October 2006. However, the suspect boards affected on GSR were manufactured between Jan 2006 and Oct 2006. Cards shipped individually as spares over that same time span may also contain one of the affected units.

This issue has been linked to a component manufacturing problem. The root cause was identified and appropriate steps taken to resolve the problem.

Product impact varies due to different flash memory access methodologies and use patterns. Some products may be able to mitigate the potential effect by re-writing affected memory areas at specific intervals. This feature does not work with all products, so please check the workaround section for availability and specific instructions.

Problem Symptoms

The problem symptom is typically seen during a system reload or cold start. Under normal circumstances a router's PCMCIA card or Compact Flash card could contain Cisco IOS software, configurations or other files necessary for system boot or operation. These files are accessed by the router during the boot sequence to reach an operating state.

In the problem state, the router is unable to access the contents of the Flash disk. When the problem occurs, the system is prevented from accessing directory information stored on the Flash disk. This type of error occurs in the controller of the Flash disk. Data stored on the Flash disk should be intact but may not be accessible. The router may indicate the state with a message log below. The router will be unable to read the contents of the Flash disk that were previously read or written to.

The output below shows the system log or terminal monitor message when the router is unable to read the Flash disk contents.


%Error opening slot0:/ (No device available)

The router may stop the boot sequence and remain at a ROM Monitor (ROMmon) prompt as seen below.

Readonly ROMMON initialized
boot: cannot open "flash:"
boot: cannot determine first executable file name on device "flash:"
cxxxx processor with xxxxxx Kbytes of main memory is configured to 64 bit mode with ECC enabled
Readonly ROMMON initialized
rommon 1 >

NOTE 3 - The Router will be unaffected and continue to run with a failed Flash disk. Only read/write access such as a logfile write to the flash disk or a router reboot will cause the problem symptom to become evident.

Workaround/Solution

Workaround
There is no workaround available for PCMCIA ATA Flash disks used in the affected PRP linecards or routers.

Solution
Exchange a failed PCMCIA ATA Flash disk with a known good disk and IOS image. Replacement of the Flash Disk at the next scheduled maintenance window is recommended to prevent a possible reboot hang on router restart.

Non failed PCMCIA ATA Flash disks matching the replacement criteria should also be replaced to maximize system reliability at the next regular maintenance window.

The PRP linecards should be checked using the process described in the How to Identify Hardware Levels section.

Upgrade
An upgrade program for the PCMCIA ATA Flash Disk was previously available to replace potentially affected parts. Cisco only replaces products that actually fail. The standard RMA process must be used in order to replace failed products.


Installation Instructions

PCMCIA ATA Flash Disk installation in a GSR product

Number four (4) in the following diagram shows the locations of the PCMCIA ATA Flash disk on the PRP-1 and PRP-2 modules.

How To Identify Hardware Levels

Identifying affected PCMCIA ATA Flash disks

Please use the following flow chart as a trouble resolution guide for PCMCIA ATA Flash disks.

NOTE #4 - Suspect boards on GSR were manufactured between Jan 2006 and Oct 2006. Refer to the How to Identify Serial Numbers section for information on serial number range boundaries.

NOTE #5 - See the card section below for command and disk information.

NOTE #6 - See card sections below for pictures and code locations.

NOTE #7 - If removal of a system processor board to check serial number or Flash disk is required use ESD procedures noted in the installation link.

Remote Identification of PCMCIA disks

For PCMCIA ATA flash disks inserted into operational systems:

Enter the CLI command: show disk[0,1]: filesys

(If the Disk is unable to be read, prioritize this system for card replacement at the next service window as the disk has failed. The router will continue to run until it reboots and cannot access the failed disk. Operation should be unaffected until a reboot or logfile write event to the failed disk.)

Look for the line: Number of Cylinders xxx

64 MB PCMCIA Flash disks: If the number is 490, it may be an affected unit.
128 MB PCMCIA Flash disks: If the number is 980, it may be an affected unit.
256 MB PCMCIA Flash disks: If the number is 980, it may be an affected unit.
512 MB PCMCIA Flash disks: If the number is 980, it may be an affected unit.

If the Cylinder counts match please inspect visually and confirm card vendor type and date range at the next service window.


Visual Identification of PCMCIA disks

An example of a defective unit is shown in the following figure:

Look for
1) Date code is earlier than 060728 (7/28/06)
2) Vendor Part Number starts with "CPC"
3) Cisco Part Number is any of the following: 16-2756-01, 16-2680-01, 16-3129-01, 16-3150-01

How To Identify Serial Numbers

Product ID and Serial Number Matrix

The "S/N Location (Quadrant)" heading refers to the physical location of the serial number decal on the specific product as referenced on the "Serial Number Locations Diagram" below.


Serial Number Locations Diagram

The diagram below is a generic representation of the affected network processing engines and I/O cards that can be used to aid in the physical identification of suspect units. The top of the card representation is divided into four (4) quadrants. The above "Product ID and Serial Number Matrix" table should be referenced to determine the quadrant location of the serial number decal.

The following table details the specific GSR 12000 products that may have shipped with suspect Flash disks between Jan 2006 and Oct 2006.

S/N Location (Quadrant) Product ID Serial Number Range
2
PRP-1 SAD092303JC To SAD110101BR
2
PRP-2 SAD08490DMR To SAD111603L2

Frequently Asked Questions

Q1 - Will formatting the Compact Flash resolve this issue?
A1 - No, attempts to format the Compact Flash will not resolve the problem. When the problem occurs data should be intact and re-formatting after applying the solution should not be necessary.

Q2 - Is there a software workaround to resolve this issue?
A2 - No, the products identified within this field notice cannot have their software upgraded to resolve this issue.

Q3 - What do I do if my router is currently in ROMMON mode?
A3 - Reference the Cisco Tech Tip for instructions on booting the router using TFTP or use another compact flash.

Revision History

Revision Date Comment
1.1 25-SEPT-2009 Archiving Upgrade Program
1.0 26-APR-2007 Initial Public Release

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Cisco Notification Service - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.