Field Notice: FN - 63765 - Some 7200/7300 Products Might Fail to Boot Up After a Power Cycle - Replace on Failure

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	17-Aug-15	Initial Release
10.0	16-Oct-17	Migration to new field notice system
10.1	10-Sep-18	Updated the Defect Information Section

Products Affected

Affected Product ID	Comments
7300-CC-PA
7300-CC-PA=
7300-NSE-150
7300-NSE-150=
7304-NPE-G100
7304-NPE-G100=
C7200-VSA
C7200-VSA=
NPE-G1
NPE-G1=
NPE-G2
NPE-G2=
PA-2T3/E3-EC
PA-2T3/E3-EC=
PA-MC-2T3-EC
PA-MC-2T3-EC=
PA-MC-T3-EC
PA-MC-T3-EC=
PA-T3/E3-EC
PA-T3/E3-EC=

Defect Information

Defect ID	Headline
CSCvf34445	There were no defects filed with this field notice at the time of publication.

Problem Description

Some 7200/7300 products might fail to boot up after user actions where the board requires a power cycle operation.

Background

Cisco has been working with some customers on an issue related to memory components manufactured by a single supplier between 2005 and 2010. These memory components are widely used across the industry and are included in a number of Cisco products.

Although the majority of Cisco products using these components are experiencing field failure rates below expected levels, some components may fail earlier than anticipated. A handful of our customers have recently experienced a higher number of failures, leading us to change our approach to managing this issue.

While other vendors have chosen to address this issue in different ways, Cisco believes its approach is the best course of action for its customers. Despite the cost, we are demonstrating that we always make customer satisfaction a top priority. Customers can learn more about this topic at Memory Component Issue web page.

PLEASE NOTE - The products listed in this Field Notice have lower than expected failure rates. This assessment is based on actual usage of affected memory components, observed field failure rates and product replacements since 2012.

A degraded component will not affect the ongoing operation of a device, but will be exposed by a subsequent power cycle event. This event will result in a hard failure of the device, which cannot be recovered by a reboot or additional power cycle. For these reasons, additional caution is recommended for operational activities requiring the simultaneous power cycling of multiple devices. This issue has been observed most commonly on devices that have been in service for 24 months or more.

Problem Symptom

If the suspected hardware has been in operation for approximately 24 months, the product hardware might fail to boot up due to memory failure during a power cycle event. This is caused by one of more of these actions:

• Power cycle the chassis
• Online Insertion Removal/Replacement (OIR)

Note: This issue does not affect boards while in operation. The board failure might occur after one or more of the actions listed is executed.

These are the linecard symptoms observed:

• NPE-G1 

  The card does not boot to ROMMON.

• NPE-G2 

  The card does not boot to ROMMON, and the SYST STAT LED on the front panel repeatedly blinks yellow.

• 7300-NSE-150, 7304-NPE-G100 

  The card does not boot to ROMMON, and the Active, Standby, and System-Up LEDs remain solid yellow.

• 7300-CC-PA 

  00:00:33: %PACC-3-NOPATYPE: No port adapter type obtained for slot [x] 
  00:00:33: %LC-3-DEACTIVATED: line card in slot [x] deactivated. 

• PA-2T3/E3-EC, PA-MC-2T3-EC, PA-MC-T3-EC, PA-T3/E3-EC

  After a power up of the affected port adapter, you might see one of these error messages on the console or in the log:

  T3E3_EC-3-PA_SW_ERR: T3E3_EC on 3: incorrect number of pci devices found 
  Software error was encountered. 
  T3E3 EC image size too big 
  T3E3 EC image location not setup 
  firmware_download_verify failed for RTOS image 
  firmware_download_verify failed for APPS image 
  pa_t3e3_ec__analyze_pa t3e3_ec_port_init failed for port x 
  T3E3_EC failed to initialize T3E3_EC_TASK 

  
  Then, this message displays: 

  PA-3-DEACTIVATED: port adapter in bay [x] powered off.

• C7200-VSA 

  Nitrox Memory 
  VSA n2_verify_ddr_mem nitrox_mult_rd_ddr_ram failed (0x0, 10485760, 0) 
  VSA ERROR: DDR Verification (0x0, 0xA00000, 0x12345678) Failed 
  VSA test_nitrox_ddr(0x0608DA20) failed 
  VSA OIR FAILED, state is 1 
  
  FPGA Memory 
  When the SA RAM fails, you may notice the following CLI output: 
  %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for
  destaddr=11.0.0.41, prot=50, spi=0xCED7831(216889393), srcaddr=11.0.0.61,
  input interface=GigabitEthernet0/3

  Once you receive this CLI output, you might query the status for the SA Error Register in order to confirm that an error has occurred:

  show pas vsa info | inc ERR 
  0x800C SA_ERR_REG 0x00000020

Workaround/Solution

Normal SmartNet and warranty entitlement rules remain in place and are applied by the Cisco Technical Assistance Center (TAC) if you experience a failure in one of the products listed in the Field Notice. In other circumstances, such as out of warranty or out of contract, Cisco encourages you to raise your concern directly with your Cisco account team.

Fix on Failure Replacement Guidelines: Request Return Material Authorization (RMA) product through normal service support channels.

If you need assistance in order to determine which hardware part(s) might need replacement, consult the error messages documented in the Problem Symptom section.

PID	Action	PID Replacement	Quantity
7300-CC-PA	Replace card	7300-CC-PA=	1
7300-NSE-150	Replace card	7300-NSE-150=	1
7304-NPE-G100	Replace card	7304-NPE-G100=	1
C7200-VSA	Replace card	C7200-VSA=	1
NPE-G1	Replace card	NPE-G1=	1
NPE-G2	Replace card	NPE-G2=	1
PA-2T3/E3-EC	Replace card	PA-2T3/E3-EC=	1
PA-MC-2T3-EC	Replace card	PA-MC-2T3-EC=	1
PA-T3/E3-EC	Replace card	PA-T3/E3-EC=	1
PA-MC-T3-EC	Replace card	PA-MC-T3-EC=	1

End of Support

PID	End of Support Notice	End of Support Date
7300-CC-PA(=)	EOL6972	31-Jul-2015
7300-NSE-150(=)	EOL6972	31-Jul-2015
7304-NPE-6100(=)	EOL6972	31-Jul-2015

How To Identify Affected Products

Enter the show inventory command in order to obtain the Product ID (PID). If the CLI is not available, physically inspect the device in order to locate the PID.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.