Upgrade Procedures

Recommended Upgrade Method

For both combined upgrade and split upgrade, Cisco recommends upgrading the HyperFlex components in the following order for optimizing the upgrade time:

  1. Upgrade Cisco UCS Infrastructure

  2. Upgrade Cisco HX Data Platform

  3. Upgrade Cisco customized VMware ESXi

  4. Upgrade Cisco UCS firmware

Upgrading Your HyperFlex Cluster Using the HX Connect UI


Note

Hypercheck Health Check Utility— Cisco recommends running this proactive health check utility on your HyperFlex cluster prior to upgrade. These checks provide early visibility into any areas that may need attention and will help ensure a seamless upgrade experience. For more information see the Hyperflex Health & Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck.


Important

Use the HX Connect UI when upgrading from a current HX Data Platform version of 2.5(1a) or later releases.

Procedure

Step 1

If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. See Downloading Software for more details.

Step 2

Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. See Test Upstream Network Connectivity for more details.

Step 3

Upgrade Cisco UCS Infrastructure bundle as required. See Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager for more details.

Note 

It is important that you manually upgrade the UCS infrastructure first before initiating the upgrade sequence of the HyperFlex components as outlined in Recommended Upgrade Method. The upgrade feature of the HX Platform software will not upgrade the UCS infrastructure bundle. This upgrade is a manual process.

Step 4

Bootstrap to upgrade Cisco HX Data Platform.

Note 

If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process).

Step 5

Log in to HX Connect.

  1. Enter the HX Storage Cluster management IP address in a browser. Navigate to https://<storage-cluster-management-ip>.

  2. Enter the administrative username and password.

  3. Click Login.

Step 6

In the Navigation pane, select Upgrade.

Step 7

Choose the type of upgrade from the Select Upgrade Type page.

Caution 

After manual bootstrap, validation will fail if you perform UCS only, ESXi only, or UCS and ESXi combined upgrade. For successful upgrade, Cisco recommends the following upgrade types:

  • HX Data Platform only upgrade, followed by UCS Firmware and/or Hypervisor Software upgrade

  • HX Data Platform and UCS Firmware

  • HX Data Platform and Hypervisor Software

  • HX Data Platform, UCS Firmware, and Hypervisor Software
Step 8

Depending on the type of upgrade you want to perform, complete the following fields on the Enter Credentials tab.

UCS Server Firmware

Field

Essential Information

UCS Manager Connectivity

UCS Manager FQDN/IP field

Enter the Cisco UCS Manager FQDN or IP address. Example: 10.193.211.120.

User Name field

Enter the Cisco UCS Manager <admin> username.

Admin Password field

Enter the Cisco UCS Manager <admin> password.

HX Server Firmware

Discover button

Click Discover to view the current UCS firmware package version, in the Current Version field.

HX Data Platform

UI Element Essential Information

Drag the HX file here or click to browse

Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform.

Sample file name format: storfs-packages-3.5.2a-31601.tgz.

Current version

Displays the current HyperFlex Data Platform version.

Current cluster details

Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state.

Bundle version

Displays the HyperFlex Data Platform version of the uploaded bundle.

(Optional) Checksum field

The MD5 Checksum number is stored in a separate text file at the /tmp directory where the upgrade package was downloaded.

This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.

ESXi

Note 

The ESXi upgrade option is supported in the HyperFlex Connect UI for HyperFlex release 3.5(1a) or later.

UI Element

Essential Information

Drag the ESXi file here or click to browse field

Upload the latest Cisco HyperFlex Custom Image Offline Bundle for upgrading existing ESXi hosts from Download Software - HyperFlex HX Data Platform.

Example: HX-ESXi-6.5U2-10884925-Cisco-Custom-6.5.2.4-upgrade-bundle.zip.

Current version field

Displays the current ESXi version.

Current hypervisor details field

Lists the HyperFlex cluster details like the Hypervisor version and Cluster upgrade state.

Bundle details field

Displays the ESXi version of the uploaded bundle.

vCenter Credentials

UI Element Essential Information

User Name field

Enter the vCenter <admin> username.

Admin Password field

Enter the vCenter <admin> password.

Step 9

Click Upgrade to begin the cluster upgrade process.

Step 10

The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete.

When upgrade is in progress, you may see an error message, 'Websocket connection failed. Automatic refresh disabled'. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message.

During cluster upgrade, if the orchestration node reboots or power cycles due to power issue, the cluster upgrade will be stuck. Once the node is up, restart the cluster upgrade process after cleaning the cluster system using the following command: 

stcli cluster upgrade --components hxdp –clean

If the clean-up command fails, restart the stMgr service on all control VMs (ctrlVM) by running the following command:

#restart stMgr

Then, clean the cluster system by rerunning the following command:

stcli cluster upgrade --components hxdp –clean

Here is a sample code:

root@ucs-stctlvm-385-1:~# stcli cluster upgrade --clean --components hxdp
##Forcefully cleaned up upgrade progress
root@ucs-stctlvm-385-1:~# stcli cluster upgrade --status
##No active upgrade found. Upgrade progress available after triggering an upgrade

Post vCenter Upgrade Tasks

Perform the following steps after you upgrade your vCenter to ensure functionality:

Procedure

Step 1

Unregister the extensions.

Example:

com.springpath.sysmgmt.domain-<id>
com.springpath.sysmgmt

Use the mob browser https://<vCenter IP or FQDN>/mob (content > extensionManager path and Invoke UnregisterExtension method).

Step 2

Re-register the Springpath plug-in using:

Example:

stcli cluster reregister
Note 

You can use stcli cluster reregister –-h for help and then continue with the reregistration.

Online Upgrade Process Workflow


Attention

If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process).

When using the online upgrade process workflow, consider the following:

  • First upgrade Cisco UCS infrastructure to the latest version and then use the automated upgrade workflow for a combined upgrade of Cisco UCS firmware and Cisco HX Data Platform. Online upgrade uses host firmware packages to upgrade all server endpoints.

  • During online upgrade, as one node is being upgraded (placed into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and the Access Policy settings.

    See Entering Cisco HyperFlex Maintenance Mode for the procedures on how to access Cisco HyperFlex Maintenance Mode.

  • If upgrading both HXDP and UCS firmware, combined upgrade can be selected through HX Connect depending on the length of the maintenance window.

  • Do not use Firefox browser. It is not supported due to an outdated version of flash that is bundled with the browser.


Note

Cisco UCS Manager infrastructure upgrade is only supported using AutoInstall and the direct server firmware upgrade should be performed only through the upgrade orchestration framework provided by the HX Data Platform Plug-in.

The following table summarizes the online upgrade workflow:

Step

Description

Reference

1.

If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C.

Downloading Software

2.

Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts.

Test Upstream Network Connectivity

3.

Upgrade Cisco UCS Infrastructure bundle as required.

Note 

It is important that you manually upgrade the UCS infrastructure first before initiating the upgrade sequence of the HyperFlex components as outlined in Recommended Upgrade Method. The upgrade feature of the HX Platform software will not upgrade the UCS infrastructure bundle. This upgrade is a manual process.

Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager

4.

Bootstrap to upgrade Cisco HX Data Platform.

Manual Bootstrap Upgrade Process

5.

Disable snapshot schedule, on the bootstrapped storage controller VM.

Note 

It is enough to run this script on one of the controller nodes.

Run the command stcli snapshot-schedule --disable.

6.

Log in to HX Connect with administrator credentials.

7.

Start combined upgrade of:

  • HX Data Platform and UCS Firmware

  • HX Data Platform and Hypervisor Software

Upgrading Your HyperFlex Cluster Using the HX Connect UI
Attention 

To perform a split upgrade, you must upgrade HX Data Platform first. After HX Data Platform is upgraded to 3.5(1x), you can perform a split upgrade of UCSM only and/or ESXi only.

When only UCS firmware is being upgraded, you may see the upgrade process pause at the validation screen after the fabric interconnect discovery. It may be a network connectivity failure issue, however, in most cases, it just requires waiting for the process to finish.

Upgrading Cisco HyperFlex Data Platform Using HX Connect

Upgrading Cisco UCS Server Firmware Using the HX Connect UI

Upgrading ESXi Using the HX Connect UI

8.

Confirm that the upgrade task is complete.

Post Upgrade Tasks

9.

On the same controller VM, enable snapshot schedule.

Run the command stcli snapshot-schedule -–enable.

Offline Upgrade Process Workflow

The following table summarizes the offline upgrade workflow:

Step

Description

Reference

1.

If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C.

Downloading Software

2.

Ensure that the hx-storage-data and vMotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts.

Test Upstream Network Connectivity

3.

Upgrade Cisco UCS Infrastructure bundle as required.

Note 

It is important that you manually upgrade the UCS infrastructure first before initiating the upgrade sequence of the HyperFlex components as outlined in Recommended Upgrade Method. The upgrade feature of the HX Platform software will not upgrade the UCS infrastructure bundle. This upgrade is a manual process.

Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager

4.

Launch the vSphere Web Client and power down all user VMs residing on HX servers and all user VMs running on HX datastores. This includes VMs running on compute-only nodes. After the VMs have been shut down, verify the health state of the cluster and perform a graceful shutdown.

Important 

HyperFlex controller VMs (stCtlVMs) must remain powered on.

Graceful Shutdown of a HX Cluster

5.

Manually stage the correct firmware version before starting the upgrade process.

Modifying Host Firmware Package Using Cisco UCS Manager

6.

Shutdown the HyperFlex Controller VMs (stCtlVMs).

In vCenter, right-click on each HX Controller VM (stCtlVM) and select Power > Shut Down Guest OS.

7.

Once the Controller VMs are shutdown, place the ESXi hosts into Maintenance Mode.

In vCenter, right-click on each ESXi host select Maintenance Mode > Enter Maintenance Mode.

8.

Acknowledge the pending reboot on the servers that comprise your HX cluster nodes, including both converged nodes and compute-only nodes connected to the cluster. Wait until all nodes are upgraded. Confirm that correct firmware packages have been installed before proceeding.

9.

Once the ESXi hosts have booted, take them out of Maintenance Mode. Now the controller VM should come back online.

In vCenter, right-click on each ESXi host select Maintenance Mode > Exit Maintenance Mode.

10.

Bootstrap to upgrade the Cisco HX Data Platform Plug-in.

Important 

  • Be sure to copy the bootstrap file to the controller VM /tmp directory.

  • Ensure that you confirm the version of the plug-in in the vCenter Administration > Client Plug-Ins page.

Manual Bootstrap Upgrade Process

11.

Disable snapshot schedule, on the bootstrapped storage controller VM.

Note 

It is enough to run this script on one of the controller nodes.

Run the command stcli snapshot-schedule --disable.

12.

From the same controller VM, begin the upgrade.

Upgrading Your HyperFlex Cluster Using the HX Connect UI

Offline Upgrade Using CLI

13.

Confirm that upgrade is complete.

Post Upgrade Tasks

14.

After the upgrade is complete, start the cluster and power on VMs.

Start Cluster and Power On VMs

15.

On the same controller VM, enable snapshot schedule.

Run the command stcli snapshot-schedule -–enable.

Offline Upgrade Guidelines


Important

  • --ucsm-host and --ucsm-user parameters are required when you are upgrading from 1.7x to 1.8x. These parameters must not be used when moving up from 1.8(1a)/1.8(1b) to 2.0(1a) as we are not changing the Cisco UCS server firmware version.

    Cisco recommends using HX Connect UI to upgrade HyperFlex clusters over CLI for ease of use and better reporting. You can safely ignore Cluster Invalid State alerts in HX Connect.

Before you proceed, with either combined or split upgrade consider the following guidelines:

  • The package name must match the file that you uploaded to the controller VM.

  • Enter passwords when prompted.

  • Nodes are upgraded with the new version of the Cisco HX Data Platform software and rebooted one at a time.

  • Offline cluster upgrades with nested vCenter is not supported.

Offline Upgrade Using CLI


Important

If you need to perform a split upgrade, you must upgrade HX Data Platform first. After HX Data Platform is upgraded to Release 3.5(1x), you can perform a split upgrade of UCSM only and/or ESXi only.

Combined Upgrade of Cisco HX Data Platform, ESXi and Cisco UCS Firmware

M5 Servers
# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location/tmp/
<storfs package name,ESXi package name> --ucsm-host <IP/FQDN of UCSM> 
--ucsm-user <UCSM User> --ucsm5-fw-version <UCSM Firmware Version>

Example for M5 Servers:

~# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location 
/tmp/storfs-packages-3.5.1a-19712.tgz 
--ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucs5fw-version '3.1(2g)'
M4 Servers
# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location/tmp/
<storfs package name, ESXi package name> --ucsm-host <IP/FQDN of UCSM> 
--ucsm-user <UCSM User> --ucsfw-version <UCSM Firmware Version>

Example for M4 Servers:

~# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location 
/tmp/storfs-packages-3.5.1a-19712.tgz 
--ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucsfw-version '3.1(2g)'

Combined Upgrade of Cisco HX Data Platform and ESXi

M5 Servers
# stcli cluster upgrade --components hxdp,hypervisor --location /tmp/hxupgrade_bundle.tgz --hypervisor-bundle /tmp/esxiupgrade_bundle.zip

Example for M5 Servers:

~# stcli cluster upgrade --components hxdp,hypervisor --location /tmp/hxupgrade_bundle.tgz --hypervisor-bundle /tmp/esxiupgrade_bundle.zip
M4 Servers
# stcli cluster upgrade --components hxdp,hypervisor --location /tmp/hxupgrade_bundle.tgz --hypervisor-bundle /tmp/esxiupgrade_bundle.zip

Example for M4 Servers:

~# stcli cluster upgrade --components hxdp,hypervisor --location /tmp/hxupgrade_bundle.tgz --hypervisor-bundle /tmp/esxiupgrade_bundle.zip

Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware

M5 Servers
# stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/
<storfs package name> --vcenter-user <vcuser> --ucsm-host <IP/FQDN of UCSM> 
--ucsm-user <UCSM User> --ucsm5-fw-version <UCSM Firmware Version>
M4 Servers
# stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/
<storfs package name> --vcenter-user <vcuser> --ucsm-host <IP/FQDN of UCSM> 
--ucsm-user <UCSM User> --ucsfw-version <UCSM Firmware Version>

Example for M4 Servers:

~# stcli cluster upgrade --components hxdp,ucs-fw --location 
/tmp/storfs-packages-1.8.1c-19712.tgz --vcenter-user administrator@vsphere.local 
--ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucsfw-version '3.1(2b)'

Start Cluster and Power On VMs

After the upgrade is complete and the cluster has been upgraded, log out and log back in to vCenter to see upgrade changes.

Procedure

Step 1

After the upgrade is complete, start your cluster.

Step 2

Login to any controller VM through SSH.

# stcli cluster start

Example: 

HyperFlex StorageController 1.8(1c)
Last login: Wed Sept 21 23:54:23 2016 from pguo-dev.eng.storvisor.com
 root@ucs-stclivm - 384 -1;~# stcli cluster upgrade-status
Cluster upgrade succeeded. Cluster version: 1.8(1c)
root@ucs-stctlvm-384;~# stcli cluster start
waiting for Cluster to start on nodes: [ucs-383, ucs-384, ucs-385, ucs-386]
This will start the cluster and mount the HX datastores. Wait for cluster to come online. You will see the prompt: 
Started cluster on nodes; [ucs-383, ucs-384, ucs-385, ucs-386]
Cluster is online
root@ucs-stctlvm-384-1;~#
Step 3

Wait for cluster to become healthy before starting the VMs. Run command:

~# stcli clustr info| grep health

Example: 


root@SpringpathControllerZRVF040451;~# stcli cluster info | grep health
healthState: healthy
state: healthy
 storage cluster is healthy
Step 4

After the cluster is healthy, launch vSphere Web Client or Thick Client, navigate to Hosts and Cluster > Datacenter > Cluster > . Right click, select Power> Power On to start the VMs.