ESXi upgrade to 7.0 U2 via HX Connect or Intersight fails with "CPU_SUPPORT WARNING"

Introduction

This document describes the process to upgrade ESXi to 7.0 U2 which can fail on servers running older generation Central Processing Units (CPU).

Background Information

The upgrade initiated through HX Connect or through Intersight displays the failure message: "Failed to apply profile HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5" or "CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly" .

The Hyperflex Data Platform (HXDP) versions required to hit this issue are:

HXDP 4.5(2a)

HXDP 4.5(1a)

HXDP 4.0(2f) and below that support ESXi 7.0 U2

Problem

Upgrade attempt to ESXi 7.0 U2 via HX Connect or Intersight on servers running older generation CPUs, notably the Sandy Bridge-EP (E5-2600), Ivy Bridge-EP (E5-2600v2), and Haswell EP (E5-2600v3) CPU families. See VMware KB for full details on the processors impacted by this compatibility check: VMware knowledge base.

HX connect error message

Solution

Scenario A. HXDP + ESXi Combined Upgrade From HX Connect User Interface

For this scenario, you upgrade HXDP to 4.5(2a) and ESXi to 7.0 U2 Build 17867351.

Step 1. Upgrade HXDP Only first.

Step 1.1. Download HXDP package via https://software.cisco.com/download/home/286305544/type/286305994/release/4.5(2a) as shown in the image:

Step 1.2. Open the HX-Connect, choose Upgrade, check the HX Data Platform checkbox, upload the package, enter the Username and Admin Password for vCenter Credentials and click Upgrade as shown in the image:

HX connect upgrade HXDP

Once the upgrade is complete, proceed to Step 2.

Step 2. Open the HX Connect, choose System information > Nodes, choose the node which you want to upgrade Esxi on and click Enter HX Maintenance Mode as shown in the images.

HX connect nodes

HX connect maintenance mode popup

You can see the activity complete under the Activity tab as shown in the image:

HX connect Activity tab

Step 3. Secure Shell (SSHl) into the ESXi host placed into Maintenance Mode as root.

Step 4. Create a folder and upload ESXi package into it.

Step 4.1. Run the command: mkdir /hxpatch

Step 4.2. Create a ramdisk to store package with the command: esxcli system visorfs ramdisk add -m 500 -M 500 -n hxpatch -t /hxpatch -p 0755.

Step 5. Download ESXi-7.2.0.5 package from cisco downloads to a local repo with the link https://software.cisco.com/download/home/286305544/type/286305994/release/4.5(2a) as shown in the image:

Step 5.1. Copy package from the local repo to /hxpatch folder. The example below uses WinSCP:

Step 6. SSH back into the host in maintenance mode.

Step 7. Run this command in order to view the package profile details: esxcli software sources profile list --depot=/hxpatch/HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5-upgrade-bundle.zip as shown in the image:

Step 8. Run this command to upgrade ESXi package to 7.0 U2:

esxcli software profile update --allow-downgrades -d /hxpatch/HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5-upgrade-bundle.zip -p HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5 --no-hardware-warning

Step 9. Reboot the host.

Step 10. SSH into the host and verify the version with the command: vmware -v.

Step 11. Exit the node from maintenance mode and wait for the cluster to become healthy.

Step 11.1. Open the HX-Connect, choose System Information> Nodes, choose the node in maintenance mode and click Exit HX Maintenance Mode as shown in the images:

HX connect Nodes

HX connect Exit MM

Step 11.2.  The HX Connect Dashboard shows you the Operational Status, it must be Online, and the Resiliency Health must be Healthy.

Step 12. Repeat steps 2-11 on all nodes which are part of the cluster, one at a time.

Once the ESXi upgrade is complete, you can see the version details on the Keyboard, Video, Mouse (KVM) console of the HX Server as shown in the image:

You can also verify within HX Connect that the new ESXi version is present on the System Information tab as shown in the image:

HX connect updated Esxi

Scenario B. HXDP + ESXi Combined Upgrade From Intersight

For this scenario, you upgrade HXDP to 4.5(2a) and ESXi to 7.0 U2 Build 17867351 from Intersight.

Step 1. If one of the HX servers is in maintenance mode, then reboot this server through vCenter or ucs KVM console.

Step 2. Once the Server has been rebooted, exit HX Maintenance Mode from the HX Connect UI. Open the HX-Connect, select System Information > Nodes, choose the node in maintenance mode and click Exit HX Maintenance Mode.

Step 3. Allow a few minutes for the Hyperflex Cluster to complete healing. The HX Connect Dashboard shows you the Operational Status, it must be Online, and the Resiliency Health must show Healthy.

Step 4. Open Intersight, choose Upgrade, click Expert Mode > Upgrade HyperFlex Data Platform as shown in the images:

Step 5. You can observe the Upgrade start and then finish:

Step 6. Start the ESXi upgrade for each server. Copy the ESXi 7.0 U2 upgrade bundle to the ESXi hosts.

Step 6.1. Download the ESXi 7.0U2 upgrade bundle from cisco downloads with the link: https://software.cisco.com/download/home/286305544/type/286305994/release/4.5(2a):

Step 6.2. On each ESXi host, copy the ESXi 7.0U2 upgrade bundle into an available datastore directory (use methods such as scp, wget). Once complete, you must see:

[root@ucs-221:/vmfs/volumes] pwd
/vmfs/volumes

[root@ucs-221:/vmfs/volumes] ls
MyDataStore                                  60ee5dfc-6f05dc8b-05cf-0025b5b1a116         60ef8a79-63aadd3f-ee0d-0025b5b1a116         BOOTBANK2                                   SpringpathDS-FCH2008V1M1
..
[root@ucs-221:/vmfs/volumes] cd MyDataStore /
[root@ucs-221:/vmfs/volumes/ac08f584-9a2f8a67] ls -al
total 0
drwx------    1 root     root           686 Jul 14 05:19 .
drwxr-xr-x    1 root     root           512 Jul 15 01:21 ..
-rw-r--r--    1 root     root     391417386 Jul 15 00:55 HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5-upgrade-bundle.zip
drwxr-xr-x    1 root     root          2793 Jul 14 20:03 ucs221-VM

[root@ucs-221:/vmfs/volumes/ac08f584-9a2f8a67] pwd
/vmfs/volumes/ MyDataStore

Step 7. Open the HX-Connect, choose System information > Nodes, choose the node which you want to upgrade Esxi on and click Enter HX Maintenance Mode as shown in the images:

HX connect Nodes

You can see the activity complete under the Activity tab as shown in the image:

Step 8. SSH into the ESXi host which is in HX Maintenance Mode.

Step 8.1. Find the upgrade bundle profile name:

[root@abcd-221:/vmfs/volumes/ac08f584-9a2f8a67] esxcli software sources profile list --depot=/vmfs/volumes/0714051143/HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5-upgrade-bundle.zip
Name                                         Vendor  Acceptance Level  Creation Time        Modification Time
-------------------------------------------  ------  ----------------  -------------------  -------------------
HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5  Cisco   PartnerSupported  2021-04-29T18:21:43  2021-04-29T18:21:43

Step 8.2. Execute the upgrade command with the no-hardware-warning parameter specified:

[root@abcd-221:/vmfs/volumes/ac08f584-9a2f8a67] esxcli software profile update --depot=/vmfs/volumes/0714051143/HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5-upgrade-bundle.zip --profile HX-ESXi-7.0U2-17867351-Cisco-Custom-7.2.0.5 --no-hardware-warning

Step 8.3. You see a success message once the upgrade completes:

Update Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true

Step 9. Reboot the host.

Step 10. Log in into the ESXi host and verify the new version with the command: vmware -v.

Step 11. Once the Server has been rebooted, exit HX Maintenance Mode from the HX Connect UI. Open the HX-Connect, choose System Information > Nodes, choose the node in maintenance mode and click Exit HX Maintenance Mode as shown in the images:

HX connect exit MM

Step 12. Allow a few minutes for the Hyperflex Cluster to complete healing. The HX Connect Dashboard shows you the Operational Status, it must be Online, and the Resiliency Health must show Healthy.

Step 13. Repeat Steps 6.-12. for each ESXi host 1 at a time. Ensure that the cluster state shows Healthy before you move on to the next node.