THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
Revision | Publish Date | Comments |
---|---|---|
1.0
|
30-Oct-17
|
Initial Release
|
1.0
|
01-Dec-17
|
Initial Release
|
1.1
|
01-Dec-17
|
fix missing sections
|
1.2
|
06-Dec-17
|
Updated PID list for Internal Part Numbers, Misc
|
1.2
|
25-Dec-17
|
Add verbiage to identify the SSD drive in the APIC
|
1.2
|
09-Jan-18
|
update FN Header
|
1.2
|
30-Jan-18
|
add recommendation to replace all drive regardless of percent used.
|
Affected Product ID | Comments |
---|---|
APIC-L1
|
All APIC-SD120G0KS2-EV and / or APIC-SD120GBKS4-EV need to be replaced.
Do NOT return the OLD SSD Drive, discard part. |
APIC-L2
|
All APIC-SD120G0KS2-EV and / or APIC-SD120GBKS4-EV need to be replaced. Do NOT return the OLD SSD Drive, discard part.
|
Defect ID | Headline |
---|---|
CSCvc84794 | Fault generation on SSD usage or health |
The endurance of Application Policy Infrastructure Controller (APIC) Solid State Drives (SSDs) is worn out over the course of high usage. This leads to slow SSD writes, and the SSD can become read-only.
When the SSD drive is degraded, it can cause CPU spikes in APIC services.
This problem exists in these two SSD parts: APIC-SD120G0KS2-EV and/or APIC-SD120GBKS4-EV.
Cisco recommends that you replace these SSDs, regardless of percent utilized, with a new Enterprise level SSD - Part Number UCS-SD200G12S3-EP. This part, UCS-SD200G12S3-EP, is included in the upgrade form contained in this field notice.
SSD endurance that exceeds 90% indicates that the system is in a problem state.
Use the CLI commands in this section in order to verify the occurrence of the problem: (Note root level is required to execute this command)
MegaCli -LdpdInfo -a0 |grep 'Target\|Device Id\|Inquiry' Virtual Drive: 0 (Target Id: 0) Device Id: 4 Inquiry Data: 9XG4F2FAST91000640NS CC03 Virtual Drive: 1 (Target Id: 1) Device Id: 5 Inquiry Data: 1403039C3336 Micron_P400e-MTFDDAK100MAR 0257
smartctl -l devstat -i -A -d sat+megaraid,5 /dev/sdb smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.27.0.1insieme-7] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === 7 0x008 1 110 Percentage Used Endurance Indicator
The observed processes that run on APIC-L1 or APIC-L2 utilize higher than expected CPU resources and slow down the APIC cluster operations.
The root cause of the symptom was identified as a specific issue of SSD degradation in a large-scale Application Centric Infrastructure (ACI) network managed by APIC-L1 or APIC-L2.
The dmesg error when the SSD is fully worn out and becomes slow and or read-only is shown here:
[492316.851464] Buffer I/O error on dev sdb, logical block 0, async page read [492316.857818] sdb: unable to read partition table [492316.860970] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [492316.860975] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00 [492316.860977] blk_update_request: I/O error, dev sdb, sector 232421248 [492316.866882] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [492316.866885] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00 [492316.866888] blk_update_request: I/O error, dev sdb, sector 232421248 [492316.872732] Buffer I/O error on dev sdb, logical block 29052656, async page read
Use the attached replacement form to order a replacement SSD.
Complete these steps in order to replace the SSD. For details on each step, see Cisco APIC SSD Replacement.
A hardware replacement is available and you can verify if your device is impacted by using the Cisco Serial Number Validation tool at this Link: Serial Number Validation FN 64329
A Form must be filled out for each separate Ship to Address.
The Upgrade Order Reference Number should be unique for each time the Form is filled out.
Please enter a valid Serial Number in the Form.
You will receive an acknowledgement email immediately after submitting the Form with Request #. Depending on material availability, both you AND the Customer email address will receive a confirmation email with Order# in 7 to 10 days. UMPIRE orders are proactive replacements and do NOT adhere to normal SLAs or Service Contracts.
NOTE: If your Ship to Address is in the following countries, please expect delays of up to 3 months depending on importation regulations: Argentina, Brazil,, Columbia, Mexico, Venezuela, India, All countries in Asia (ei: Singapore, Malaysia, Hong Kong, China, Vietnam, Korea, Thailand, Philippines), and all non-EU countries (ie: UAE, Russia, Turkey). You will receive your Order# at that time. Thank you for your patience as this process is beneficial for the customer; it will save them the cost of vat/duty in these countries (which is very high).
If you were given a Sales Order Number for the shipment of your replacement parts, please refer to the SO Status Tool (Please note: you must have a CCO User ID and Password to access this site):https://cisco-apps.cisco.com/cisco/psn/commerce
If you were given an RMA Number for the shipment of your replacement parts, please refer to the "Service Order QuickSearch" Tool at the following location (Please note: you must have a CCO User ID and Password to access this site):http://tools.cisco.com/support/serviceordertool/home.svo
If you have not received an email with an Order# after 10 days, please send an email with your Request#(s) and Customer in the Subject line to:mailto:umpire-escalations@cisco.com
If you were given a Sales Order Number for the shipment of your replacement parts, please refer to the SO Status Tool (Please note: you must have a CCO User ID and Password to access this site): https://cisco-apps.cisco.com/cisco/psn/commerce
Note: Fields marked with an asterisk (*) are required fields.
1 For phone and fax, include 011 and the country code outside North America.
2 The serial number input field for each Product ID can hold up to 4,000 characters, including commas and white space. For longer lists of serial numbers, please submit additional requests.
3 For customers in Japan only *** please enter the building and the floor in the address field. Also, enter the contact person's name, the telephone number and the e-mail address in the appropriate fields..
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.