Field Notice: FN64329 - APIC SSD Degradation After High Percent Utilization of Solid State Drive - Hardware Upgrade Available

Available Languages

Updated:July 24, 2023

Document ID:FN64329

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Affected Product Name	Description	Comments
APIC-L1	^APIC Appliance - Large Configurations (> 1000 Edge Ports)	All APIC-SD120G0KS2-EV and / or APIC-SD120GBKS4-EV need to be replaced. Do NOT return the OLD SSD Drive, discard part.
APIC-L2	^APIC Appliance - Large Configuration(> 1000 EdgePorts)	All APIC-SD120G0KS2-EV and / or APIC-SD120GBKS4-EV need to be replaced. Do NOT return the OLD SSD Drive, discard part.

Defect Information

Defect ID	Headline
CSCvc84794	APIC SSD Degradation After High Percent Utilization of Solid State Drive \| Fault F2730

Problem Description

The endurance of Application Policy Infrastructure Controller (APIC) Solid State Drives (SSDs) is worn out over the course of high usage. This leads to slow SSD writes, and the SSD can become read-only.

When the SSD drive is degraded, it can cause CPU spikes in APIC services.

This problem exists in these two SSD parts: APIC-SD120G0KS2-EV and/or APIC-SD120GBKS4-EV.

Cisco recommends that you replace these SSDs, regardless of percent utilized, with a new Enterprise level SSD - Part Number UCS-SD200G12S3-EP. This part, UCS-SD200G12S3-EP, is included in the upgrade form contained in this field notice.

Background

SSD endurance that exceeds 90% indicates that the system is in a problem state.

Use the CLI commands in this section in order to verify the occurrence of the problem: (Note root level is required to execute this command)

Enter the MegaCli command in order to find the SSD device, as shown here:

MegaCli -LdpdInfo -a0 |grep 'Target\|Device Id\|Inquiry' 
Virtual Drive: 0 (Target Id: 0)
Device Id: 4
Inquiry Data: 9XG4F2FAST91000640NS CC03
Virtual Drive: 1 (Target Id: 1)
Device Id: 5
Inquiry Data: 1403039C3336 Micron_P400e-MTFDDAK100MAR 0257

Enter the smartctl command in order to determine the endurance:

smartctl -l devstat -i -A -d sat+megaraid,5 /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.27.0.1insieme-7] (local build) 
			
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
			
=== START OF INFORMATION SECTION ===
			
7 0x008 1 110 Percentage Used Endurance Indicator

Problem Symptom

The observed processes that run on APIC-L1 or APIC-L2 utilize higher than expected CPU resources and slow down the APIC cluster operations.

The root cause of the symptom was identified as a specific issue of SSD degradation in a large-scale Application Centric Infrastructure (ACI) network managed by APIC-L1 or APIC-L2.

The dmesg error when the SSD is fully worn out and becomes slow and or read-only is shown here:

[492316.851464] Buffer I/O error on dev sdb, logical block 0, async page read
[492316.857818] sdb: unable to read partition table
[492316.860970] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.860975] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.860977] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.866882] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.866885] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.866888] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.872732] Buffer I/O error on dev sdb, logical block 29052656, async page read

Workaround/Solution

Use the link in Upgrade Program Information to order a replacement SSD.

Complete these steps in order to replace the SSD. For details on each step, see Cisco APIC SSD Replacement.

Decommission the APIC.
If the Cisco Integrated Management Controller (CIMC) version is earlier than 2.0(9c), use Host Upgrade Utility (HUU) to upgrade the CIMC.
"Prepare for Removal" of the SSD from CIMC.
Note: This step is conditional and is required if you already have a SSD installed. Perform this step before you replace an existing SSD with a new SSD. This step is not required for installation of a new SSD.
If there is an old SSD, physically remove it and add the new SSD.
Use the newly installed SSD to create RAID volume.
Install the APIC image with the virtual media. In this step, the SSD is partitioned and the APIC software is installed on the hard drive.
Recommission the APIC.

Upgrade Program Information

Support Case Manager must be used for ordering replacement parts for this Field Notice.

Click on the following link to open Support Case Manager in a new tab:
https://mycase.cloudapps.cisco.com/fieldnotice?fn=FN64329

Serial Numbers (SNs) must be provided and be affected.
- SN Entitlement Check will be performed.
- Order entry supports up to 50 SNs per request. If you have more than 50, you will need to submit more than 1 request.
One ship to address per request.
Service Request number (SR#) is not required, but if you have an Existing SR# please enter it for better tracking purposes.

Revision History

Version	Description	Section	Date
1.3	Migrated to Support Case Manager Ordering System	—	2023-JUL-24
1.2	Updated PID list for Internal Part Numbers, Misc	—	2023-JUL-21
1.0	Initial Release	—	2017-DEC-01
1.1	fix missing sections	—	2017-DEC-01

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Open a service request on Cisco.com
By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.