Field Notice: FN - 64329 - APIC SSD Degradation After High Percent Utilization of Solid State Drive - Hardware Upgrade Available

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	30-Oct-17	Initial Release
1.0	01-Dec-17	Initial Release
1.1	01-Dec-17	fix missing sections
1.2	06-Dec-17	Updated PID list for Internal Part Numbers, Misc
1.2	25-Dec-17	Add verbiage to identify the SSD drive in the APIC
1.2	09-Jan-18	update FN Header
1.2	30-Jan-18	add recommendation to replace all drive regardless of percent used.

Products Affected

Affected Product ID	Comments
APIC-L1	All APIC-SD120G0KS2-EV and / or APIC-SD120GBKS4-EV need to be replaced. Do NOT return the OLD SSD Drive, discard part.
APIC-L2	All APIC-SD120G0KS2-EV and / or APIC-SD120GBKS4-EV need to be replaced. Do NOT return the OLD SSD Drive, discard part.

Defect Information

Defect ID	Headline
CSCvc84794	Fault generation on SSD usage or health

Problem Description

The endurance of Application Policy Infrastructure Controller (APIC) Solid State Drives (SSDs) is worn out over the course of high usage. This leads to slow SSD writes, and the SSD can become read-only.

When the SSD drive is degraded, it can cause CPU spikes in APIC services.

This problem exists in these two SSD parts: APIC-SD120G0KS2-EV and/or APIC-SD120GBKS4-EV.

Cisco recommends that you replace these SSDs, regardless of percent utilized, with a new Enterprise level SSD - Part Number UCS-SD200G12S3-EP. This part, UCS-SD200G12S3-EP, is included in the upgrade form contained in this field notice.

Background

SSD endurance that exceeds 90% indicates that the system is in a problem state.

Use the CLI commands in this section in order to verify the occurrence of the problem: (Note root level is required to execute this command)

• Enter the MegaCli command in order to find the SSD device, as shown here:

  MegaCli -LdpdInfo -a0 |grep 'Target\|Device Id\|Inquiry' 
  Virtual Drive: 0 (Target Id: 0)
  Device Id: 4
  Inquiry Data: 9XG4F2FAST91000640NS CC03
  Virtual Drive: 1 (Target Id: 1)
  Device Id: 5
  Inquiry Data: 1403039C3336 Micron_P400e-MTFDDAK100MAR 0257

• Enter the smartctl command in order to determine the endurance:

  smartctl -l devstat -i -A -d sat+megaraid,5 /dev/sdb
  smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.4.27.0.1insieme-7] (local build) 
  			
  Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
  			
  === START OF INFORMATION SECTION ===
  			
  7 0x008 1 110 Percentage Used Endurance Indicator

Problem Symptom

The observed processes that run on APIC-L1 or APIC-L2 utilize higher than expected CPU resources and slow down the APIC cluster operations.

The root cause of the symptom was identified as a specific issue of SSD degradation in a large-scale Application Centric Infrastructure (ACI) network managed by APIC-L1 or APIC-L2.

The dmesg error when the SSD is fully worn out and becomes slow and or read-only is shown here:

[492316.851464] Buffer I/O error on dev sdb, logical block 0, async page read
[492316.857818] sdb: unable to read partition table
[492316.860970] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.860975] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.860977] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.866882] sd 0:2:4:0: [sdb] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[492316.866885] sd 0:2:4:0: [sdb] tag#0 CDB: Read(10) 28 00 0d da 77 80 00 00 08 00
[492316.866888] blk_update_request: I/O error, dev sdb, sector 232421248
[492316.872732] Buffer I/O error on dev sdb, logical block 29052656, async page read

Workaround/Solution

Use the attached replacement form to order a replacement SSD.

Complete these steps in order to replace the SSD. For details on each step, see Cisco APIC SSD Replacement.

1. Decommission the APIC.
2. If the Cisco Integrated Management Controller (CIMC) version is earlier than 2.0(9c), use Host Upgrade Utility (HUU) to upgrade the CIMC.
3. "Prepare for Removal" of the SSD from CIMC.
   Note: This step is conditional and is required if you already have a SSD installed. Perform this step before you replace an existing SSD with a new SSD. This step is not required for installation of a new SSD.
4. If there is an old SSD, physically remove it and add the new SSD.
5. Use the newly installed SSD to create RAID volume.
6. Install the APIC image with the virtual media. In this step, the SSD is partitioned and the APIC software is installed on the hard drive.
7. Recommission the APIC.

Upgrade Program

  A hardware replacement is available and you can verify if your device is impacted by using the Cisco Serial Number Validation tool at this Link: Serial Number Validation FN 64329

 

A Form must be filled out for each separate Ship to Address.

The Upgrade Order Reference Number should be unique for each time the Form is filled out.

Please enter a valid Serial Number in the Form.
You will receive an acknowledgement email immediately after submitting the Form with Request #. Depending on material availability, both you AND the Customer email address will receive a confirmation email with Order# in 7 to 10 days. UMPIRE orders are proactive replacements and do NOT adhere to normal SLAs or Service Contracts.

NOTE: If your Ship to Address is in the following countries, please expect delays of up to 3 months depending on importation regulations: Argentina, Brazil,, Columbia, Mexico, Venezuela, India, All countries in Asia (ei: Singapore, Malaysia, Hong Kong, China, Vietnam, Korea, Thailand, Philippines), and all non-EU countries (ie: UAE, Russia, Turkey). You will receive your Order# at that time. Thank you for your patience as this process is beneficial for the customer; it will save them the cost of vat/duty in these countries (which is very high).

If you were given a Sales Order Number for the shipment of your replacement parts, please refer to the SO Status Tool (Please note: you must have a CCO User ID and Password to access this site):https://cisco-apps.cisco.com/cisco/psn/commerce

If you were given an RMA Number for the shipment of your replacement parts, please refer to the "Service Order QuickSearch" Tool at the following location (Please note: you must have a CCO User ID and Password to access this site):http://tools.cisco.com/support/serviceordertool/home.svo

If you have not received an email with an Order# after 10 days, please send an email with your Request#(s) and Customer in the Subject line to:mailto:umpire-escalations@cisco.com
If you were given a Sales Order Number for the shipment of your replacement parts, please refer to the SO Status Tool (Please note: you must have a CCO User ID and Password to access this site): https://cisco-apps.cisco.com/cisco/psn/commerce


Note: Fields marked with an asterisk (*) are required fields.

Requestor Information
*Name
*E-mail Address
TAC SR Number
Customer Shipping Information
*Company
*Address
Address_line2
*City
*State/Province
*ZIP/Postal Code
*Country	Select a Country AFGHANISTAN ALBANIA ALGERIA AMERICAN SAMOA ANDORRA ANGOLA ANGUILLA ANTARCTICA ANTIGUA ARGENTINA ARMENIA ARUBA AUSTRALIA AUSTRIA AZERVBAIJAN BAHAMAS BAHRAIN BANGLADESH BARBADOS BELARUS BELGIUM BELIZE BENIN BERMUDA BHUTAN BOLIVIA BOSNIA BOTSWANA BOUVET ISLAND BRAZIL BRITISH INDIES BRUNEI DARUSSALAM BULGARIA BURKINA FASO BURUNDI CAMBODIA CAMEROON CANADA CAPE VERDE CAYMAN ISLANDS CENTRAL AFRICA CHAD CHILE CHINA CHRISTMAS ISLAND COCOS COLOMBIA COMOROS CONGO CONGO, THE DEMOCRATIC REPUBLIC OF THE COOK ISLANDS COSTA RICA COTE D IVOIRE CROATIA CUBA CYPRUS CZECH REPUBLIC DENMARK DIJIBOUTI DOMINICA DOMINICAN REPUBLIC ECUADOR EGYPT EL SALVADOR EQUATORIAL ERITREA ESTONIA ETHIOPIA FALKLAND ISLANDS FAROE ISLANDS FIJI FINLAND FRANCE FRENCH GUIANA FRENCH POLYNESIA FRENCH SOUTHERN TERR GABON GAMBIA GEORGIA GERMANY GHANA GIBRALTAR GREECE GREENLAND GRENADA GUADELOUPE GUAM GUATEMALA GUINEA GUINEA-BISSAU GUYANA HAITI HEARD ISLAND HONDURAS HONG KONG HUNGARY ICELAND INDIA INDONESIA IRAN, ISLAMIC REPUBLIC OF IRAQ IRELAND ISRAEL ITALY JAMAICA JAPAN JORDAN KAZAKHSTAN KENYA KIRIBATI KOREA, DEMOCRATIC PEOPLE'S REPUBLIC OF KOREA, REPUBLIC OF KUWAIT KYRGYZSTAN LAO PEOPLE LATVIA LEBANON LESOTHO LIBERIA LIBYAN ARAB LIECHTENSTEIN LITHUANIA LUXEMBOURG MACAO MACEDONIA, THE FORMER YUGOSLAV REPUBLIC OF MADAGASCAR MALAWI MALAYSIA MALDIVES MALI MALTA MARSHALL ISLAND MARTINQUE MAURITANIA MAURITIUS MAYOTTE MEXICO MICRONESIA, FEDERATED STATES OF MOLDOVA MONACO MONGOLIA MONTSERRAT MOROCCO MOZAMBIQUE MYANMAR NAMBIA NAURU NEPAL NETHERLANDS NETHERLANDS ANTILLES NEW CALEDONIA NEW ZEALAND NICARAGUA NIGER NIGERIA NIUE NORFOLK ISLAND NORTHERN COMMONWEALTH NORWAY OMAN PAKISTAN PALAU PALESTINIAN TERRITORY,OCCUPIED PANAMA PAPUA NEW GUINEA PARAGUAY PERU PHILIPPINES PITCAIRN POLAND PORTUGAL PUERTO RICO QATAR REUNION ROMANIA RUSSIAN FEDERATION RWANDA SAINT HELENA SAINT KITTS SAINT LUCIA SAINT PIERRE SAINT VINCENT SAMOA SAN MARINO SAO TOME SAUDI ARABIA SENEGAL SERBIA AND MONTENEGRO SEYCHELLES SIERRA LEONE SINGAPORE SLOVAKIA SLOVENIA SOLOMON ISLAND SOMALIA SOUTH AFRICA SOUTH GEORGIA AND THE SOUTH SANDWICH ISLANDS SPAIN SRI LANKA SUDAN SURINAME SVALBARD AND JAN MAYEN SWAZILAND SWEDEN SWITZERLAND SYRIAN ARAB TAIWAN TAJIKISTAN TANZANIA THAILAND TIMOR-LESTE TOGO TOKELAU TONGA TRINIDAD AND TOBAGO TUNISIA TURKEY TURKMENISTAN TURKS AND CAICOS TUVALU UGANDA UKRAINE UNITED ARAB EMIRATES UNITED KINGDOM UNITED STATES UNITED STATES MINOR OUTLYING ISLANDS URUGUAY UZBEKISTAN VANUATU VATICAN CITY VENEZUELA VIET NAM VIRGIN ISLANDS, BRITISH VIRGIN ISLANDS, U.S. WALLIS AND FUTUNA WESTERN SAHARA YEMEN ZAMBIA ZIMBABWE
Product	Affected Product *Quantity *Serial# 2 Replacement PID APIC-L1 UCS-SD200G12S3-EP= APIC-L2 UCS-SD200G12S3-EP=
Customer Contact Information
*First Name
*Last Name
*Phone 1	Ext.
Fax 1	Ext.
*E-Mail	Please use the following format: user@domain.com
*Upgrade Order Reference Number	Please provide a number that you can use when inquiring about order status
Notes

¹ For phone and fax, include 011 and the country code outside North America.

² The serial number input field for each Product ID can hold up to 4,000 characters, including commas and white space. For longer lists of serial numbers, please submit additional requests.

³ For customers in Japan only *** please enter the building and the floor in the address field. Also, enter the contact person's name, the telephone number and the e-mail address in the appropriate fields.. 

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.