Stretched Cluster Upgrade

Overview

This section provides information related to upgrading a Cisco HyperFlex Stretched Cluster. The procedure for performing a Stretched Cluster upgrade is similar to the regular HyperFlex cluster upgrade procedure.

Upgrade Guidelines for Stretched Cluster

  • Review the Cisco HyperFlex Upgrade Guidelines in the Recommended Cisco HyperFlex HX Data Platform Software Releases - for Cisco HyperFlex HX-Series Systems.

  • Combined HXDP and UCS FW is not supported.

  • UCS FW upgrade from HX-Connect is not supported.

  • Manual cluster bootstrap is required for upgrade from a pre-3.5 release to 3.5(1a).

    Auto bootstrap is supported for upgrade from 3.5(1a) to later releases.

  • When upgrading to any release from 3.0.x to 3.5.x or later releases:

    • If you manually bootstrap only one node, the ESXi checkbox will not appear under the Select Upgrade Type section of the Cluster Upgrade page. The ESXi upgrade option appears only after upgrading HX Data Platform to release 3.5.x or later releases.

    • If you manually bootstrap all the nodes to 3.5.x or later release, the ESXi checkbox will appear under the Select Upgrade Type section of the Cluster Upgrade page. However, you cannot perform an ESXi only upgrade at this point. You can do a combined upgrade of HX Data platform + ESXi.

  • HyperFlex Witness node version 1.0.2 is supported from 3.5(1a) or later releases. An upgrade of the HyperFlex Witness node is not required when upgrading stretched clusters to 3.5(1a) or later releases.

  • Hypercheck Health Check Utility— Cisco recommends running this proactive health check utility on your HyperFlex cluster prior to upgrade. These checks provide early visibility into any areas that may need attention and will help ensure a seamless upgrade experience. For more information see the Hyperflex Health & Pre-Upgrade Check Tool TechNote for full instructions on how to install and run Hypercheck.

Upgrading HyperFlex Stretched Cluster Using HX Connect

Before you begin

  • Complete pre-upgrade validation checks.

  • Download the latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters from previous releases, from Software Download .

  • Upgrade Cisco UCS Infrastructure.

  • Disable snapshot schedule, on the storage controller VM. SSH to HyperFlex cluster IP and run the command stcli snapshot-schedule –disable snapshot schedule.

  • If DRS is Enabled and set to fully automatic mode, the VMs are automatically migrated to other hosts with vMotion.


    Note

    If DRS is Disabled, vMotion the VMs manually to continue the upgrade process when prompted. For more information, see VMware Documentation for Migration with vMotion.

Procedure

Step 1

Log in to HX Connect.

  1. Enter the administrative username and password.

  2. Click Login.

Step 2

In the Navigation pane, select Upgrade.

Step 3

On the Select Upgrade Type page, select HX Data Platform and ESXi and complete the following fields:

Step 4

On the Select Upgrade Type page, select HX Data Platform and complete the following fields:

UI Element

Essential Information

Drag the HX file here or click to browse

Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform.

Sample file name format: storfs-packages-4.5.1a-31601.tgz.

Current version

Displays the current HyperFlex Data Platform version.

Current cluster details

Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state.

Bundle version

Displays the HyperFlex Data Platform version of the uploaded bundle.

(Optional) Checksum field

The MD5 Checksum number is available by hovering over the filename in the Cisco.com Software Download section.

This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle.
Step 5

Upload the VMware ESXi custom image offline upgrade bundle.
Step 6

Provide vCenter login credentials:

Essential Information

Essential Information

User Name field

Enter the vCenter <admin> username.

Admin Password field

Enter the vCenter <admin> password.
Step 7

Click Upgrade to begin the combined upgrade process.

Step 8

The Validation screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any.

Note 

At this point, all pre-upgrade checks and validations are running, along with the initial upgrade stage. Within a few minutes, HX Connect returns and prompts you to confirm and start the second stage of the upgrade. The upgrade is not complete until both steps are performed in the UI. The system should never be left in a state where only the first step of the upgrade is complete.

Note 

Do not manually acknowledge servers in UCS Manager. While the servers will enter a pending-ack state, the administrator should not manually intervene. The HyperFlex platform automatically acknowledges each server at the correct time.

Step 9

The HyperFlex Connect UI refreshes after the first step of the upgrade, and a banner pops up prompting you to provide the UCS and vCenter credentials and start the second stage of the upgrade process. Monitor the upgrade page and confirm that the upgrade is complete.

When upgrade is in progress, you may see an error message Websocket connection failed. Automatic refresh disabled. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message.

Note 

Perform post upgrade tasks once the upgrade is complete. If the upgrade fails, you can re-try the upgrade or contact Cisco TAC for further assistance.

Upgrading a Witness VM

Before you begin

Select the Witness VM version that supports the HXDP version you are upgrading to. For supported versions see the HX Data Platform Software Versions for HyperFlex Witness Node for Stretched Cluster section of the Cisco HyperFlex Recommended Software Release and Requirements Guide.

  • For supported versions see the HX Data Platform Software Versions for HyperFlex Witness Node for Stretched Cluster section of the Cisco HyperFlex Recommended Software Release and Requirements Guide

  • Upgrade HyperFlex Stretched Cluster.

  • The upgraded HyperFlex Stretched Cluster must be in healthy state. To check the health state of Stretched Cluster after upgrade, run the following command:
    root@StCtlVM:~# stcli cluster info | grep healthy

Procedure

Step 1

Log in to the witness VM using SSH and execute the following command to stop the service exhibitor.

root@WitnessVM:~# service exhibitor stop
Step 2

Copy the exhibitor.properties file available in the /usr/share/exhibitor/ path to a remote machine from where you can retrieve the exhibitor.properties file. 

scp root@<Witness-VM-IP>:/usr/share/exhibitor/exhibitor.properties user@
<Remote-Machine>:/directory/exhibitor.properties
Step 3

Log out from the Witness VM. Power off and rename the Witness VM to WitnessVM.old.

Note 

Confirm that the IP address of the old Witness VM is unreachable, using the ping command.

Step 4

Deploy a new Witness VM and configure the same IP address as the old Witness VM.

Note 

If the IP address is not reachable, the Witness OVA deployment may contain stale entries in the /var/run/network directory. You must manually remove these entries and reboot the VM to have the assigned IP address become reachable on the network.

To reboot the VM, open the VM console in vCenter/vSphere and execute the following command:

rm -rf /var/run/network/*
reboot
Step 5

Log in to the new witness VM using SSH and execute the following command to stop the service exhibitor.

root@WitnessVM:~# service exhibitor stop
Step 6

Copy the exhibitor.properties file from the remote machine (copied in Step 2) to the /usr/share/exhibitor/ path of the new Witness VM. 

scp /directory/exhibitor.properties root@<Witness-VM-IP>:
/usr/share/exhibitor/exhibitor.properties
Step 7

Verify if the following symlinks are preserved in the new Witness VM: 

root@Cisco-HX-Witness-Appliance:~# cd /etc/exhibitor/
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ls -al
total 8
drwxr-xr-x 2 root root 4096 Sep 11 13:00 .
drwxr-xr-x 88 root root 4096 Sep 11 12:55 ..
lrwxrwxrwx 1 root root 41 Sep 11 13:00 exhibitor.properties 
lrwxrwxrwx 1 root root 37 Jul 24 16:49 log4j.properties

If the symlinks are not available, execute the following command:

root@Cisco-HX-Witness-Appliance:/etc/exhibitor# 
ln -s /usr/share/exhibitor/exhibitor.properties exhibitor.properties 
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# 
ln -s /usr/share/exhibitor/log4j.properties log4j.properties r
oot@Cisco-HX-Witness-Appliance:/etc/exhibitor# 
ls -al total 8 drwxr-xr-x 2 root root 4096 Sep 11 13:00 . drwxr-xr-x 88 root root 4096 
Sep 11 12:55 .. lrwxrwxrwx 1 root root 41 
Sep 11 13:00 exhibitor.properties -> /usr/share/exhibitor/exhibitor.properties lrwxrwxrwx 
1 root root 37 Jul 24 16:49 log4j.properties -> /usr/share/exhibitor/log4j.properties
Step 8

Run the /usr/share/springpath/storfs-misc/setexhibitorconfig.sh command to upgrade the Witness exhibitor configuration.

Note 

This step is required for users that are moving to Witness VM Node version 1.1.1 and later, if the Witness VM being upgraded is a version previous to 1.1.1.

Note 

The setexhibitorconfig.sh automates the process of editing the exhibitor.properties file, and replaces all of the data IP addresses with the management IP addresses for each corresponding controller VM.

Note 

It is normal for this command to not show any output when upgrading from a Witness VM that is older than 1.1.1.

Step 9

Start the service exhibitor by executing the following command:

root@Cisco-HX-Witness-Appliance:~# service exhibitor start
exhibitor start/running, process <ID>

Manually Upgrading ESXi for Cisco HyperFlex Stretch Cluster 4.0(x)

Procedure

Step 1

Select one of the hosts and put it in HX maintenance mode using the vSphere Web Client. After the host enters maintenance mode, complete the following steps.

Step 2

Copy files using SCP, start the SSH service in the destination ESXi hosts as well.

Note 

  • On HX240, you can use the local SpringpathDS datastore or a mounted HX datastore.

  • On HX220, you can use either a mounted HX datastore or create a temporary RAM disk.

 
scp local_filename user@server:/path/where/file/should/go
Step 3

Log in to ESXi, and execute the following command to query the list of available image profiles and for profile name verification.

esxcli software sources profile list -d <location_of_the_esxi_zip_bundle_on_the_datastore>

Attention 

Full path must be used when running the esxcli software command.

Example:

[root@localhost:~] esxcli software sources profile list -d /vmfs/volumes/5d3a21da-7f370812-ca58-0025
b5a5a102/HX-ESXi-6.0U3-13003896-Cisco-Custom-6.0.3.9-upgrade-bundle.zip
Name                                         Vendor  Acceptance Level  Creation Time        Modification Time
-------------------------------------------  ------  ----------------  -------------------  -------------------
HX-ESXi-6.0U3-13003896-Cisco-Custom-6.0.3.9  Cisco   PartnerSupported  2019-04-02T00:14:56  2019-04-02T13:38:34
Step 4

Run the following command to perform the upgrade.

esxcli software profile update -d <path_to_profile_ZIP_file> -p < profile name>

Example:

[root@HX-ESXi-01:/vmfs/volumes/1a234567-89bc1234] esxcli software profile update -d
/vmfs/volumes/1a234567-89bc1234/HX-Vmware-ESXi-60U2-4192238-Cisco-Custom-Bundle-6.0.2.3.zip
-p HX-ESXi-6.0U3-13003896-Cisco-Custom-6.0.3.9
Step 5

After the ESXi host comes up, verify that the host has booted up with the correct version. 

vmware -vl
Step 6

Exit maintenance mode using the vSphere Web Client.
Step 7

Ensure that the cluster becomes healthy between each ESXi upgrade.

stcli cluster storage-summary --detail
Step 8

Repeat this process for all hosts in the cluster in a sequence.

Note 

Make sure that the cluster becomes healthy between each ESXi upgrade.

Configuring Stretched Cluster for UCS FW Upgrade

During upgrade, the following customized UCS policies are validated and adjusted for HyperFlex:

  • HFP (Host Firmware Package) - Host Firmware Packages provide consistent firmware files for the multiple components of a HyperFlex node. This includes CIMC, BIOS, HBA and SAS Expander firmware, VIC and other components. Unlike typical UCS Host Firmware Packages, they also control disk firmware, due to the criticality of this to Hyperflex Data Platform. Note that Self Encrypting Drives (SED) firmware is controlled by HyperFlex Data Platform directly and not UCS Manager policies.

  • VNIC Templates - Virtual NIC (VNIC) templates provide consistent configuration of VNIC's between UCS fabrics. HyperFlex VNIC Templates are configured as redundancy pairs to ensure changes to Hyperflex VNIC's on one UCS fabric is applied to the other.

  • Ethernet Adaptor Policies - Ethernet Adaptor Policies provide performance related properties for HyperFlex VNIC's.

  • BIOS Policies - BIOS policies control configuration and performance of key hardware resources on a HyperFlex node, such as CPU and Memory. HyperFlex uses specific configuration to provide consistent high performance.

  • VNIC/VHBA Placement Policies - VNIC/VHBA placement policies determine the PCI addresses presented to the HyperFlex node for a given VNIC/VHBA. HyperFlex sets this in a consistent manner so further configuration can proceed succesfully.

Procedure

Step 1

SSH to any CVM on a site and change directory into /tmp
Step 2

Run the following command: /usr/local/bin/hx.py --upgrade-cluster-config. This generates a file called "customer_site_config.json" and saves it in the /tmp directory.

Step 3

Edit the customer_site_config.json file to change the firmware version and the org name appropriately. For example:

Example:

{
    "id": "Advanced",
    "collapse": true,
    "label": "Advanced",
    "groups": [
      {
        "id": "firmware",
        "label": "UCS Firmware",
        "items": [
          {
            "id": "version",
            "label": "UCS Firmware Version",
            "type": "text",
            "description": "UCS Firmware Version to be used on the HX servers",
            "placeholder": "ex: 3.2(2d)",
            "defaultValue": "3.2(2d)",
            "value": "4.1(1d)"  #<<<<<----------------- Change this 
          },
          {
            "id": "version-m5",
            "label": "UCS Firmware Version",
            "type": "text",
            "description": "UCS Firmware Version to be used on the M5 HX servers",
            "placeholder": "ex: 3.2(2d)",
            "defaultValue": "3.2(2d)",
            "value": "4.1(1i)" #<<<<<----------------- Change this 
          }
        ]
      },
      {
        "id": "org",
       "items": [
                  {
                        "id": "name",
                        "label": "Hyperflex Org name",
                        "type": "text",
                        "value": "Faridabad", #<<<<<----------------- Change this 
                        "description": "The name of the org in ucsm which is to be used for creation of all the policies and profiles for this Hyperflex cluster"
                  }
        ]
      }
    ]
Step 4

Execute the command again and enter the UCSM IP and credentials.

For example:
/usr/local/bin/hx.py --upgrade-cluster-config

Example:

[root@SpringpathControllerVP0RX5DWTC:/# /usr/local/bin/hx.py --upgrade-cluster-config
                [UCS Manager] [in_progress][  0.00%][ETA:  0:18:00] Login to UCS API
                 UCS host name or virtual IP address: 10.42.17.11
                 Connecting to admin@10.42.17.11...
                Password:
Step 5

Ensure that the command runs without any error. If there is an error, contact Cisco TAC.

Note 

Note that this command (hx.py) is being run for the first site FI domain. You need to run the same steps for the second site FI domain later.

Step 6

Perform the following steps in vCenter and UCSM:

  1. Verify that Pending reboot appears in the pending activities of the UCSM.

  2. Put one host in maintenance mode.

  3. Reboot the server and then wait for the server to come online and cluster to be online/healthy.

  4. Perform the same steps for the remaining nodes.
Step 7

Repeat Steps 4, 5 and 6 again for the other site.