Field Notice: FN - 70620 - Moving Existing HyperFlex ESXi Hosts to vCenter 7.0 Update 1 Can Unexpectedly Power Off and Delete the HyperFlex Controller VMs - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	16-Oct-20	Initial Release
1.1	23-Oct-20	Updated the Problem Description, Background, Problem Symptom, and Workaround/Solution Sections
2.0	08-Jan-21	Updated the Problem Description, Background, Problem Symptom, and Workaround/Solution Sections

Products Affected

Affected Product ID	Comments
HX220C-M4S=	Part Alternate
HX220C-M4S
HX240C-M4SX=	Part Alternate
HX240C-M4SX
HXAF240C-M4SX=	Part Alternate
HXAF220C-M4S=	Part Alternate
HXAF240C-M4SX
HXAF220C-M4S
HXAF220C-M5SX=	Part Alternate
HXAF240C-M5SX=	Part Alternate
HX240C-M5SX=	Part Alternate
HX240C-M5SX
HX220C-M5SX
HXAF240C-M5SX
HXAF220C-M5SX
HX220C-M5SX=	Part Alternate
HX240C-M5L
HX240C-M5L=	Part Alternate
HXAF220C-M5SX=	Part Alternate
HXAF240C-M5SX=	Part Alternate
HX240C-M5SX=	Part Alternate
HX240C-M5SX
HX220C-M5SX
HXAF240C-M5SX
HXAF220C-M5SX
HX220C-M5SX=	Part Alternate
HX-E-240M5SX=
HXAF-E-240M5SX=
HXAF-E-240M5SX
HX-E-240M5SX
HXAF-E-220M5SX
HX-E-220M5SX

Defect Information

Defect ID	Headline
CSCvw01432	HyperFlex Controller VMs Are Deleted After Being Added to vCenter 7.0 U1

Problem Description

An interoperability issue between HyperFlex and vCenter 7.0 Update 1 (U1) has been identified that impacts HyperFlex controller virtual machines (VMs) that are managed by vSphere ESX Agent Manager (EAM). At the time of initial publishing of this notice, vCenter and ESXi 7.0 and 7.0 U1 were not supported with HyperFlex, and this notice was issued out of an abundance of caution.

HyperFlex Data Platform (HXDP) Software Releases 4.0(2) and 4.5(1) and later now support vCenter Server 7.0 U1c (build 17327517) and later. Refer to the HyperFlex product documentation links in the Background section for full interoperability details.

Background

HXDP software adheres to a list of compatible ESXi and vCenter versions in all published software release notes. vCenter Server 7.0 U1c can be used with HyperFlex 4.0(2) and 4.5(1) and later releases. No vCenter Server 7 version earlier than U1c can be used at any time. Refer to HyperFlex 4.0 Release Notes, Table 6. ESXi Versions and Cisco HX Release 4.5(x) – Software Requirements, Table 7. Software Requirements for VMware ESXi for full details.

Note: ESXi and vCenter compatibility with the HXDP are independently qualified and listed separately in the product documentation. This field notice strictly covers vCenter Server 7.0 interoperability and does not address ESXi 7.0 interoperability.

vCenter Server 7.0 Behavior Prior to U1c

Some HyperFlex clusters use a service in vCenter Server that is called ESX Agent Manager (EAM). This service is responsible for the lifecycle of the HyperFlex controller VMs (named stCtlVM in vCenter).

When existing HyperFlex ESXi hosts are moved from an existing vCenter Server to a new vCenter Server 7.0 U1, the HyperFlex controller VMs that reside on these hosts will be powered off and deleted. This issue impacts HyperFlex clusters that were initially installed with an HXDP release earlier than 4.0(1a). By default, all clusters that were deployed initially on a release pre-4.0(1a) will continue to use the EAM service, even after an upgrade to Release 4.0(1a) of HXDP or later.

Example scenarios are listed here:

• HyperFlex cluster initially deployed on 2.6(1a) and then upgraded to 4.0(2c).
  • Although the cluster is upgraded to a release later than 4.0(1a), it was initially deployed on a release  pre-4.0(1a) and is therefore susceptible to this issue.
• HyperFlex cluster initially deployed on 4.0(1b).
  • This cluster is not impacted because it was initially deployed on release 4.0(1a) or later.
• HyperFlex cluster initially deployed on 3.5(2a), then upgraded to 4.0(2c), and then manually modified in order to remove the EAM configuration in accordance with Cisco documentation.
  • This cluster is still susceptible. The manual removal procedure does not resolve the problem.

If a susceptible cluster is removed from the current vCenter and the corresponding ESXi hosts are added to a new vCenter 7.0 U1 instance, the controller VMs will be permanently deleted. As a result, the cluster becomes unavailable and, in some cases, the cluster data becomes unrecoverable.

Problem Symptom

HyperFlex Controller VMs can power off suddenly and be deleted from disk by the EAM service in vCenter when adding the ESXi hosts that host the controller VMs to a vCenter earlier than 7.0 U1c. This results in a loss of cluster availability. In some cases, the HyperFlex storage cluster can become unrecoverable.

Workaround/Solution

VMware has enhanced the default EAM behavior in vCenter Server 7.0 U1c and later to prevent orphaned VM cleanup automatically for non-vSphere Clustering Service (vCLS) VMs. Fresh and upgraded vCenter Server installations will no longer encounter an interoperability issue with HXDP controller VMs when running vCenter Server 7.0 U1c and later.

When used with HyperFlex, vCenter Server should never be manually configured to auto-cleanup all orphaned VMs automatically. For further details, refer to VMware KB 81352.

vCenter Server 7.0 earlier than the U1c release should never be used with Cisco HyperFlex.

vCenter Server	EAM Behavior	HyperFlex Support Stance
All 6.x releases	No EAM interoperability issue.	Supported per HyperFlex release notes.
7.0 (15952498)	No EAM interoperability issue. Unqualified by Cisco.	Unsupported.
7.0a (16189094)	No EAM interoperability issue. Unqualified by Cisco.	Unsupported.
7.0b (16386292)	No EAM interoperability issue. Unqualified by Cisco.	Unsupported.
7.0c (16620007)	No EAM interoperability issue. Unqualified by Cisco.	Unsupported.
7.0d (16749653)	No EAM interoperability issue. Unqualified by Cisco.	Unsupported.
7.0 U1 (16860138)	EAM cleanup results in interoperability issue with HyperFlex.	Unsupported.
7.0 U1a (17004997)	EAM cleanup results in interoperability issue with HyperFlex.	Unsupported.
7.0 U1c (17327517) and later	Issue resolved.	Supported per HyperFlex release notes.

 

Note: Manual removal of the EAM configuration in accordance with the documented procedure on Cisco.com will not prevent this issue.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.