Field Notice: FN - 70234 - SED Drive Failure Might Cause the HyperFlex Cluster to Go Down - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	28-Jun-18	Initial Release
1.1	16-Aug-18	Updated for HyperFlex Content
1.2	18-Oct-18	Updated the Product Hierarchy Metatags
1.3	28-Nov-18	Updated the Defect Information, Background, and Workaround/Solution Sections for HX Release 3.0(1i)
1.4	04-Dec-18	Updated the Background, Workaround/Solution, and How to Identify Affected Products Sections
1.5	04-Jan-19	Updated the Workaround/Solution Section
1.6	11-Feb-19	Updated the Workaround/Solution and How to Identify Affected Products Sections
1.7	25-Mar-19	Updated the Workaround/Solution Section
1.8	02-Apr-20	Updated the Defect Information, Problem Description, Background, Problem Symptom, and Workaround/Solution Sections
1.9	20-Jul-20	Updated Terminology

Products Affected

Affected Product ID	Comments
UCS-M2-240GB=	Part Alternate
UCS-M2-240GB	Part Alternate
HX-M2-240GB
HX-SD38TBM1K9=
HX-SD38TBM1K9
HX-SD960GBM1K9=
HX-SD960GBM1K9
HX-SD240GBM1K9=
HX-SD240GBM1K9
HX-SD38TBE1NK9=
HX-SD38TBE1NK9
HX-SD960GBE1NK9=
HX-SD960GBE1NK9
HX-SD240GBE1NK9=
HX-SD240GBE1NK9
HX-M2-240GB=

Defect Information

Defect ID	Headline
CSCvj66157	SED drive failure may cause the UCS/HX cluster to go down
CSCvm66552	Multiple simultaneous 3.8TB SED SSD drive failures may cause the HX cluster to go offline
CSCvk17250	Cluster instability when disks of different sector size placed in HX node

Problem Description

A drive firmware issue on select Self-Encrypting Drives (SEDs) might cause an operational issue for some HyperFlex clusters.

An increased rate of blocked drives might occur, which requires frequent drive replacements and in some instances a potential for a HyperFlex cluster outage.

During drive replacement or addition, there is a potential for a HyperFlex cluster outage. The remediation needs to be performed prior to the drive replacement or addition.

All existing clusters running HXDP version 3.5.2a and below with SED drives should be upgraded to the latest star release in 3.5 that is available on cisco.com before any disks are added/replaced or cluster expansion.

Background

An operational bug in the drive firmware might be triggered when the drive is subjected to a specific workload, which could result in uncorrectable drive-level errors. Software upgrades are recommended in order to mitigate potential risks associated with uncorrectable errors. A couple of newer issues have also been addressed where data that is read in one location can affect data stored in an adjacent location and during drive replacement or addition there is a potential block size mismatch of drive sector size which leads to the cluster outage.

HyperFlex blocks the drive when the involved errors are encountered. The blocked drive state is when a disk is not utilized by the cluster due to either a software error or an I/O error. This could be a transitional state while the cluster attempts to repair the disk, if the disk is still available, before the state transitions to "repairing". After repeated I/O errors the drive might be permanently blocked, which could trigger frequent drive replacements. While the HyperFlex HX Data Platform (HXDP) software protects against drive failures, there is a potential for the cluster to fail after multiple, simultaneous drive failures.

Problem Symptom

In order to handle the errors, HXDP software puts the drive in a blocked state. When there are several drive errors, the drive is permanently blocked as shown in the How To Identify Affected Products section. Blocked drives appear as shown in the How To Identify Affected Products section.

Note: All clusters that have the affected parts need to be upgraded as soon as possible and the upgrade recommendation is not limited to clusters that show this symptom.

Workaround/Solution

HyperFlex blocks the drive when the involved errors are encountered. The "blocked" drive state is when a disk is not utilized by the cluster due to either a software error or an I/O error. This could be a transitional state while the cluster attempts to repair the disk, if the disk is still available, before the state transitions to "repairing". After repeated I/O errors the drive might be permanently blocked, which could trigger frequent drive replacements. While the HXDP software protects against drive failures, there is a potential for the cluster to fail after multiple, simultaneous drive failures.

The action required for HyperFlex nodes is listed in this table.

Configuration	Action Required
Systems with only HX-M2-240GB boot drive (no SED in system)	Perform a combined upgrade - HXDP to Version 3.5(2b) or later*, and Unified Computing System Manager (UCS Manager) to Version 4.0(1c) or later. Note: Do NOT upgrade UCS Manager only.
Clusters not created	Create the cluster with UCS Manager Version 4.0(1c) or later*, and HXDP Version 3.5(2b) or later*.
Clusters created (with SEDs)	Upgrade HXDP to Version 3.5(2b) or later*. Upgrade UCS Manager to Version 4.0(1c) or later*.

* This is the minimum upgrade version. The recommended version to be used is listed in Recommended Cisco HyperFlex HX Data Platform Software Releases - for Cisco HyperFlex HX-Series Systems.

See Cisco HyperFlex Systems Upgrade Guides for instructions on how to upgrade your system.

UCS Manager software images are available at UCS Infrastructure and UCS Manager Software Release 4.0(1C).

HyperFlex software images are available at HyperFlex HX Data Platform Release 3.5(2b).

Cisco recommends that you enable autosupport in order to enhance the supportability of HyperFlex clusters.

All HyperFlex releases posted are supported, however the recommended release is designated with a "*" next to the release name on the Software Download page.

How To Identify Affected Products

HyperFlex Systems

The Products Affected section lists the systems with the affected drive Product IDs.

Note: The upgrade and the remediation process post upgrade is required irrespective of whether or not the system currently shows blocked drives.

Blocked drives can be seen in the drive inventory on the HyperFlex Connect user interface as follows:

1. Click System Information in the left column.
2. Click the Disks tab.
3. Look to see if any disks show "Blacklisted" in the Status column.

   

   Blocked drives also appear in the System Overview tab. Click on any slot with a red circle as shown in this example.

   

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.