Cisco HyperFlex Software Advisory for HX Release 3.5(2e)

Cisco HyperFlex Software Advisory for HX Release 3.5(2e)

Cisco engineering has identified an issue with Cisco HyperFlex Release 3.5(2e) release that may affect your use of this software. Please review the Software Advisory notice here to determine if the issues apply to your environment and the steps required to address the issue.

Contents:

 

 

 

 

 

Software Advisory for CSCvw01432: Moving Existing HyperFlex ESXi Hosts to vCenter 7.0 Update 1 Can Unexpectedly Power Off and Delete the HyperFlex Controller VMs

 

Updated: January 8, 2021

 

 

Dear Cisco Customer,

 

Cisco engineering has identified the following software issues with the release that you have selected that may affect your use of this software. Please review the Software Advisory notice here to determine if the issues apply to your environment. You may proceed to download this software if you have no concerns with the issue described.

 

For more comprehensive information about what is included in this software, refer to the Cisco software Release Notes, available from the Product Selector tool. From this page, select the product you are interested in. Release Notes are under "General Information" on the product page.

 

Reason for Advisory:

An interoperability issue between HyperFlex and vCenter 7.0 Update 1 (U1) has been identified that impacts HyperFlex controller VMs that are managed by vSphere ESX Agent Manager (EAM). At the time of initial publishing of this notice, vCenter and ESXi 7.0 and 7.0 U1 were not supported with HyperFlex, and this notice was issued out of an abundance of caution. HyperFlex Data Platform software versions 4.0(2) & 4.5(1) and later now support vCenter Server 7.0 U1c (build 17327517) and higher.

HyperFlex Data Platform software adheres to a list of compatible ESXi and vCenter versions in all published software release notes. vCenter Server 7.0 U1c may be used with HyperFlex 4.0(2) and 4.5(1) and later releases. No vCenter Server 7 version prior to U1c may be used at any time. Refer to the HyperFlex 4.0 Release Notes, table 6 & Cisco HX Release 4.5(x) – Software Requirements, table 7 for full details.

Note: ESXi & vCenter compatibility with the HyperFlex Data Platform are independently qualified and listed separately in the product documentation. This field notice strictly covers vCenter Server 7.0 interoperability and does not address ESXi 7.0 interoperability.

vCenter Server 7.0 behavior prior to U1c:

Some HyperFlex clusters use a service in vCenter Server that is called EAM. This service is responsible for the lifecycle of the HyperFlex controller VMs (named stCtlVM in vCenter).

 

When existing HyperFlex ESXi hosts are moved from an existing vCenter Server to a new vCenter Server 7.0 U1, the HyperFlex controller VMs that reside on these hosts will be powered off and deleted. This issue impacts HyperFlex clusters that were initially installed with a HyperFlex Data Platform (HXDP) version earlier than 4.0(1a). By default, all clusters that were deployed initially on a version pre-4.0(1a) will continue to use the EAM service, even after an upgrade to version 4.0(1a) of HXDP or later.

 

Example scenarios are listed here:

o   HyperFlex cluster initially deployed on 2.6(1a) and then upgraded to 4.0(2c).

·       Although the cluster is upgraded to a version later than 4.0(1a), it was initially deployed on a version pre-4.0(1a) and is therefore susceptible to this issue.

o   HyperFlex cluster initially deployed on 4.0(1b).

·       This cluster is not impacted because it was initially deployed on version 4.0(1a) or later.

o   HyperFlex cluster initially deployed on 3.5(2a), then upgraded to 4.0(2c), and then manually modified in order to remove the EAM configuration in accordance with Cisco documentation.

·       This cluster is still susceptible. The manual removal procedure does not resolve the problem.

 

If a susceptible cluster is removed from the current vCenter and the corresponding ESXi hosts are added to a new vCenter 7.0 U1 instance, the controller VMs will be permanently deleted. As a result, the cluster becomes unavailable and, in some cases, the cluster data becomes unrecoverable.

 

Affected Hardware Platforms:

All HyperFlex converged nodes except for new hardware that requires HXDP version 4.0(1a) or later.

 

Symptom:

HyperFlex Controller VMs can power off suddenly and be deleted from disk by the EAM service in vCenter when adding the ESXi hosts that host the controller VMs to a vCenter prior to 7.0 U1c. This results in a loss of cluster availability. In some cases, the HyperFlex storage cluster can become unrecoverable.

 

Workaround:

VMware has enhanced the default EAM behavior in vCenter Server 7.0 U1c and later to prevent orphaned VM cleanup automatically for non-vCLS VMs. Fresh and upgraded vCenter Server installations will no longer encounter an interoperability issue with HyperFlex Data Platform controller VMs when running vCenter Server 7.0 U1c and later.

When used with HyperFlex, vCenter Server should never be manually configured to auto-cleanup all orphaned VMs automatically. For further detail, refer to VMware KB 81352 available at https://kb.vmware.com/s/article/81352.

vCenter Server 7.0 prior to the U1c release should never be used with Cisco HyperFlex.

Note: Manual removal of the EAM configuration in accordance with the documented procedure on Cisco.com will not prevent this issue.

 

More Information:

See Cisco Field Notice FN70620.

 

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

·       Open a service request on Cisco.com

·       By email or telephone

 

 

 

 

Software Advisory for Cisco HyperFlex Stretched Cluster Operations

 

July 13, 2020

 

 

Dear Cisco Customer,

 

Cisco engineering has identified issues with Stretched Cluster configurations with HX 3.5(2e) release that may affect your use of this software. Please review the Software Advisory notice here to determine if the issues apply to your environment and the steps required to address the issue.

For more comprehensive information about what is included in this software, refer to the Cisco software Release Notes, available from the Product Selector tool. From this page, select the product you are interested in. Release Notes are under "General Information" on the product page.

 

 

Reason for Advisory:

This software advisory is a notification for issues found in HXDP 3.5(2e) that impacts all stretched cluster configurations. For this reason, HXDP 3.5(2e) should not be used for stretched cluster configurations.

 

Stretched Cluster Configuration Users:

If you have installed or already upgraded a Stretched Cluster configuration to HXDP 3.5(2e) please reinstall or upgrade to HXDP Recommended release for 3.5(2x)2. See the Recommended Cisco HyperFlex HX Data Platform Software Releases for current recommendations.

 

Non-Stretched Cluster Configuration Users:

HyperFlex Standard HX clusters are not affected by these defects.

 

 

Affected Hardware Platforms:

HX220C-M5SX

HX240C-M5SX 

HXAF220C-M5SX 

HXAF240C-M5SX

HX240C-M5L

Symptoms & Conditions

 To be susceptible to the issues listed, the HyperFlex cluster must be running HXDP 3.5(2e) and configured in a Stretched Cluster configuration. HyperFlex Standard HX clusters are not affected by this defect. 

 

 

Better Split-Brain Handling and Failover

 

Workaround: If a Stretched Cluster configuration has been installed or upgraded to HXDP 3.5(2e) (The affected release) - please upgrade to the HXDP Recommended release for 3.5(2x). See the “Recommended Cisco HyperFlex HX Data Platform Software Releases” for the current recommendations.

 

 

 

 

 

 

Software Advisory for CSCvq06952 and CSCvp88515: Snapshot Creation on Change Block Tracking (CBT) Enabled Fails

 

February 27, 2020

 

 

Dear Cisco Customer,

 

Cisco engineering has identified an incompatibility issue with the Cisco HyperFlex Native Snapshot API and VMware ESXi 6.7 U2 and U3. This combination may affect your use of this software. Please review the Software Advisory notice here to determine if the issues apply to your environment, and the steps required to address the issue.

For more comprehensive information about what is included in this software, refer to the Cisco software Release Notes, available from the Product Selector tool. From this page, select the product you are interested in. Release Notes are under "General Information" on the product page.

 

 

Reason for Advisory:

This software advisory is a notification about an incompatibility found when using Cisco HyperFlex Data Platform Snapshot API and VMware ESXi 6.7 U2 and U3 Hypervisor. This combination is causing the snapshot creation on Change Block Tracking (CBT) enabled VM to fails with the error “Failed in vmreparent vmkfstools clone1"

 

Solution:

Users running Cisco HyperFlex 3.5 release should upgrade to 3.5(2g) to resolve this issue.

Users running Cisco HyperFlex 4.0 release should upgrade to 4.0(2a) to resolve this issue.

 

 

Affected Hardware Platforms:

HX220C-M5

HX220C-M5SX

HX220C-M5 Edge

HX240C-M5

HX240C-M5SX

HX220C-AF M5

HX240C-AF M5

HX240C-M5L

HXAF220c-M5SN

HXAF220C-M5SX

HXAF240C-M5SX

HX220C All NVMe M5

 

Symptom:

Due to a technical incompatibility between Cisco HyperFlex Data Platform’s Native Snapshot APIs and the VMWare ESXi 6.7 U2 and U3 hypervisor, when Cisco HyperFlex Native Snapshot API is invoked for VMs that have Change Block Tracking (CBT) enabled, snapshots fail. After the snapshot failure, the backup process that initially called the native snapshot API will also fail.

 

Conditions:

The issue is limited to users with the following combination of Cisco HyperFlex Data Platform versions and VMWare ESXi hypervisor versions

·       Clusters running one of the Cisco HyperFlex Data Platform versions listed above when a HyperFlex Native Snapshot is attempted for VMs that have CBT enabled.

·       Cisco HyperFlex cluster that have upgraded to or fresh installed with the hypervisor version VMWare ESXi 6.7 U2 and U3.

Workaround:

None. See Solution.

 

More Info:

None.