Field Notice: FN - 64329 - APIC SSD Degradation After High Percent Utilization of Solid State Drive - Hardware Upgrade Available

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Products Affected

Defect Information

Problem Description

The endurance of Application Policy Infrastructure Controller (APIC) Solid State Drives (SSDs) is worn out over the course of high usage. This leads to slow SSD writes, and the SSD can become read-only.

When the SSD drive is degraded, it can cause CPU spikes in APIC services.

This problem exists in these two SSD parts: APIC-SD120G0KS2-EV and/or APIC-SD120GBKS4-EV.

Cisco recommends that you replace these SSDs, regardless of percent utilized, with a new Enterprise level SSD - Part Number UCS-SD200G12S3-EP. This part, UCS-SD200G12S3-EP, is included in the upgrade form contained in this field notice.

Background

SSD endurance that exceeds 90% indicates that the system is in a problem state.

Use the CLI commands in this section in order to verify the occurrence of the problem: (Note root level is required to execute this command)

• Enter the MegaCli command in order to find the SSD device, as shown here:

  MegaCli -LdpdInfo -a0 |grep 'Target\|Device Id\|Inquiry' 
  Virtual Drive: 0 (Target Id: 0)
  Device Id: 4
  Inquiry Data: 9XG4F2FAST91000640NS CC03
  Virtual Drive: 1 (Target Id: 1)
  Device Id: 5
  Inquiry Data: 1403039C3336 Micron_P400e-MTFDDAK100MAR 0257

• Enter the smartctl command in order to determine the endurance:

  smartctl -l devstat -i -A -d sat+megaraid,5 /dev/sdb
  smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.27.0.1insieme-7] (local build) 
  			
  Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
  			
  === START OF INFORMATION SECTION ===
  			
  7 0x008 1 110 Percentage Used Endurance Indicator

Problem Symptom

The observed processes that run on APIC-L1 or APIC-L2 utilize higher than expected CPU resources and slow down the APIC cluster operations.

The root cause of the symptom was identified as a specific issue of SSD degradation in a large-scale Application Centric Infrastructure (ACI) network managed by APIC-L1 or APIC-L2.

The dmesg error when the SSD is fully worn out and becomes slow and or read-only is shown here:

[492316.851464] Buffer I/O error on dev sdb, logical block 0, async page read
[492316.857818] sdb: unable to read partition table
[492316.860970] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.860975] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.860977] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.866882] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.866885] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.866888] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.872732] Buffer I/O error on dev sdb, logical block 29052656, async page read

Workaround/Solution

Use the attached replacement form to order a replacement SSD.

Complete these steps in order to replace the SSD. For details on each step, see Cisco APIC SSD Replacement.

1. Decommission the APIC.
2. If the Cisco Integrated Management Controller (CIMC) version is earlier than 2.0(9c), use Host Upgrade Utility (HUU) to upgrade the CIMC.
3. "Prepare for Removal" of the SSD from CIMC.
   Note: This step is conditional and is required if you already have a SSD installed. Perform this step before you replace an existing SSD with a new SSD. This step is not required for installation of a new SSD.
4. If there is an old SSD, physically remove it and add the new SSD.
5. Use the newly installed SSD to create RAID volume.
6. Install the APIC image with the virtual media. In this step, the SSD is partitioned and the APIC software is installed on the hard drive.
7. Recommission the APIC.

Upgrade Program

  A hardware replacement is available and you can verify if your device is impacted by using the Cisco Serial Number Validation tool at this Link: Serial Number Validation FN 64329

A Form must be filled out for each separate Ship to Address.

The Upgrade Order Reference Number should be unique for each time the Form is filled out.

Please enter a valid Serial Number in the Form.
You will receive an acknowledgement email immediately after submitting the Form with Request #. Depending on material availability, both you AND the Customer email address will receive a confirmation email with Order# in 7 to 10 days. UMPIRE orders are proactive replacements and do NOT adhere to normal SLAs or Service Contracts.

NOTE: If your Ship to Address is in the following countries, please expect delays of up to 3 months depending on importation regulations: Argentina, Brazil,, Columbia, Mexico, Venezuela, India, All countries in Asia (ei: Singapore, Malaysia, Hong Kong, China, Vietnam, Korea, Thailand, Philippines), and all non-EU countries (ie: UAE, Russia, Turkey). You will receive your Order# at that time. Thank you for your patience as this process is beneficial for the customer; it will save them the cost of vat/duty in these countries (which is very high).

If you were given a Sales Order Number for the shipment of your replacement parts, please refer to the SO Status Tool (Please note: you must have a CCO User ID and Password to access this site):https://cisco-apps.cisco.com/cisco/psn/commerce

If you were given an RMA Number for the shipment of your replacement parts, please refer to the "Service Order QuickSearch" Tool at the following location (Please note: you must have a CCO User ID and Password to access this site):http://tools.cisco.com/support/serviceordertool/home.svo

If you have not received an email with an Order# after 10 days, please send an email with your Request#(s) and Customer in the Subject line to:mailto:umpire-escalations@cisco.com
If you were given a Sales Order Number for the shipment of your replacement parts, please refer to the SO Status Tool (Please note: you must have a CCO User ID and Password to access this site): https://cisco-apps.cisco.com/cisco/psn/commerce


Note: Fields marked with an asterisk (*) are required fields.

¹ For phone and fax, include 011 and the country code outside North America.

² The serial number input field for each Product ID can hold up to 4,000 characters, including commas and white space. For longer lists of serial numbers, please submit additional requests.

³ For customers in Japan only *** please enter the building and the floor in the address field. Also, enter the contact person's name, the telephone number and the e-mail address in the appropriate fields.. 

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.