All endpoints in a Cisco HyperFlex domain must be fully functional and all processes must be complete before you begin a firmware upgrade on those endpoints. For example, the firmware on a server that has not been discovered cannot be upgraded or downgraded. Each endpoint is a component in the Cisco HyperFlex domain that requires firmware to function.

Follow the listed the best practices while upgrading the endpoints.

1. Review the Release Notes and the relevant Hardware and Software Interoperability Matrix to ensure the operating systems on all HX servers have the right driver levels for the release to which you plan to upgrade.

   1. Verify the desired UCS bundle version is in the HX HCL list.

   2. Ensure the operating systems on all servers have the right driver levels for the release of Cisco UCS to which you plan to upgrade.

2. Back up the configuration into an All Configuration backup file.

3. Before you perform firmware updates, use the UCS Manager Image management interfaces in order to download relevant images to the fabric interconnect. Images are stored in bootflash partition in the fabric interconnect.

4. Verify vSphere licensing satisfies the HyperFlex requirements.

   1. Verify each HX node is licensed for vMotion, HA and DRS.

   2. If DRS is not supported, not enabled, or not in fully automated mode, then user VMs must be manually migrated to other nodes in the cluster.

5. Verify that the HX Cluster is healthy. See section HX Node Upgrade Validations for further details.

6. Verify vMotion is enabled for all nodes in the HX Cluster. For further details see the Verify vMotion Configuration for HX Cluster section.

7. Disable cluster rebalance. Enter the following on the HX Data PLatform command line:

   # stcli rebalance disable

   Verify the cluster rebalance is disabled by checking the attribute in the CLI.

   # stcli rebalance status
   rebalanceStatus:
   percentComplete: 0
   rebalanceState: cluster_rebalance_not_running
   rebalanceEnabled: False

8. Disable vCenter HA.

   From the vSphere Web Client, click HX Cluster > Settings > vSphere HA and turn it off if it is not already off.

   Note	This is only applicable if the cluster is being upgraded from HX Data Platform 1.7.0 or 1.7.1

9. Verify the VMware cluster is healthy.

   1. Verify there are no major alarms reported for the cluster.

   2. Verify DRS is enabled.

   3. Verify DRS is set to "fully automated". For other DRS options such as manual or partial, follow the VMware best practices.

   4. Verify vSphere services are running and there are no issues with ESX Agent manager (EAM).

      • Log in to the vSphere Web Client with a vCenter Single Sign-on administrator account.

      • Navigate to Administration > Deployment > System Configuration.

      • Click on Services to view the list of all services within the vCenter Server system.

10. Verify the UCS cluster is healthy.

    1. Verify that the high availability status of the fabric interconnects shows that both are up and running.

    2. Verify that the data path is up and running. For more information, see the Cisco UCS Manager Firmware Management Guide.

    3. Verify the HX servers have no faults.

    4. Verify that all servers have been discovered.

11. Verify NTP is configured on the fabric interconnects.

12. Verify the vNIC templates are configured symmetrically on Fabric Interconnect A and B.

    1. Determine if vNIC A and B are configured symmetrically.

    2. If vNIC A and B are not configured symmetrically, reconfigure the vNICs to ensure vNICs A and B carry the same VLANs

HX Cluster Health

The following validations must be performed on a each HX node before moving on to upgrading the next node in the cluster.

1. Verify the HX Cluster is healthy and online.

   1. GUI

      To view the Cluster summary, access the summary tab from vSphere Web Client Navigator. Select Home > vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > Cluster > Summary. The storage cluster status is displayed on the top left of the page and in the Cluster widget.

   2. CLI

      Login to any controller VM in the storage cluster. Run the command - # stcli cluster info

      Example response that indicates the storage cluster is online and heathy:

      locale: English (United States)
      state: online
      upgradeState: ok
      healthState: healthy
      state: online
      state: online

      The HX cluster health status takes into consideration the number of disks that have been black listed or failed.

2. Verify all HX Cluster nodes are connected in vCenter and are online.

   From vSphere Web Client Navigator. Select Home > Inventory> Host and Cluster > Datacenter . Select the cluster. The State of all the HX nodes in the cluster is displayed on the Host tab on the top.

3. Verify the VMware cluster is healthy.

   1. Verify that no major alarms are reported for the HX Cluster.

   2. Verify DRS is enabled.

   3. Verify DRS is set to fully automated

   4. Verify vSphere services are running and ESX Agent Manager health is Normal.

VMware Cluster Health

1. Verify that no major alarms are reported for the HX Cluster.

   1. Login to vSphere Web Client. Navigate to Inventory > Hosts and Clusters > Datacenter

2. Verify DRS is enabled and set to fully automated.

   1. Login to vSphere Web Client. Navigate to Inventory > Hosts and Clusters > Datacenter.

   2. Select the HX Cluster. Click on Summary tab to check DRS is On.

   3. Click DRS tab to check Migration Automation Level is set to Fully Automated.

3. Verify vSphere services are running and ESX Agent Manager (EAM) health is Normal.

   1. Login to vSphere Web Client. Navigate to Administration > vCenter Solutions Manager > vSphere ESX Agent Manager > Summary

When upgrading multiple software and firmware components, perform the upgrade in the following order. Specific instructions for upgrading components are described later in this document. If a particular upgrade below is not required, the step can be skipped.

1. Upgrade vCenter

2. Upgrade Cisco UCS Infrastructure firmware

3. Upgrade HX Data Platform

4. Upgrade Cisco UCS server firmware

5. Upgrade ESXi Hypervisor software

The HX node firmware can be upgraded with no disruption to the HyperFlex workloads. This is achieved by performing a rolling upgrade of each node in the HX cluster. Prior to the upgrade process, update the firmware version on the HX servers. Follow the listed steps to have the correct firmware version.

Procedure

1. Login to the Cisco UCS Manager

   • Open a web browser and enter the IP address of UCS Manager. For example: 10.193.211.120

   • Enter the admin level login credentials for User Name and Password.

   • Click Continue. The UCS Manager Configuration page displays Servers, LAN Configuration, and Advanced fields.

2. Select Servers > Policies > Sub-Organizations.

3. Expand the hx-cluster sub-organization.

4. Expand Host Firmware Packages.

5. Select the HyperFlex Host Firmware Package.

6. Click Modify package Versions.

7. On each sub-tab, complete the following steps for each type of firmware you want to include in the package:

   1. In the Select column, ensure that the check box for the appropriate lines are checked.

   2. In the Vendor, Model, and PID columns, verify that the information matches the nodes you want to update with this package. The model and model number (PID) must match the servers that are associated with this firmware package. If you select the wrong model or model number, Cisco UCS Manager cannot install the firmware update.

   3. In the Version column, choose the desired firmware version to which you want to upgrade the firmware for Blade package.

   4. Do the same for desired version of Rack package.

8. Click OK. If HX Service Profiles are currently associated, a popup is displayed indicating that the change requires user acknowledgment.

9. Click OK.

10. Verify Pending Activities.

    1. In the Pending Activities, HX service profiles that require user acknowledgment are listed. Do not acknowledge service profiles yet. Each Service Profile must be acknowledged sequentially to perform a non-disruptive rolling upgrade

    2. If there are multiple HX Clusters, all nodes will require user acknowledgment. Cisco UCS Manager will list all HX nodes in the "pending activities".