2.5(1d)

May 8, 2018

Added "Cluster Expansion Guidelines" section.

February 9, 2018

Updated support for VMware ESXi 6.5 U1 in the "Supported VMware vSphere Versions and Editions" table.

February 7, 2018

Revised workaround description for CSCvh54563 to add support for VMware ESXi 6.5 U1.

January 30, 2018

Added CSCvh54563 to the list of open caveats for 2.5(1d).

October 28, 2017

Created release notes for Cisco HX Data Platform Software, Release 2.5(1d).

2.5(1c)

October 13, 2017

Added section on "Upgrade Advisory for Hybrid Clusters with Replication Factor of 2".

September 8, 2017

Created release notes for Cisco HX Data Platform Software, Release 2.5(1c).

2.5(1b)

August 22, 2017

Corrected note regarding Cisco TAC support in the "Upgrade Guidelines" section.

August 16, 2017

Updated "Upgrade Guidelines" section to specify 2.5(1b) upgrade requirements.

July 31, 2017

Created release notes for Cisco HX Data Platform Software, Release 2.5(1b).

2.5(1a)

July 24, 2017

Created release notes for Cisco HX Data Platform Software, Release 2.5(1a).

CSCvf61822

Retry of the upgrade after upgrading some components, may not start storfs successfully.

2.5(1a)

2.5(1c)

CSCvf58104

One or more Storage Controller VMs may be deleted due to insufficient resources.

2.5(1a)

2.5(1c)

CSCvf44793

Single node kernel upgrade fails due to panic in the VM.

2.5(1a)

2.5(1c)

CSCvf5944

Upgrade from release 2.1(1b) to 2.5(1b) failed.

2.5(1a)

2.5(1c)

CSCvf23255

ASUP email notification does not work when multiple Notification Settings are open.

2.5(1a)

2.5(1c)

CSCuy87775

Datastore failed to mount on all ESX hosts. This is a VMware issue.

1.7.1

2.5(1c)

CSCvf40007

When any of the out-of-box HX virtual switches - vMotion or vm-network - are changed to use the vSphere distributed switch (vDS), upgrades to 2.5(1a) fail.

2.5(1a)

2.5(1b)

CSCuy87828

Performance chart display is not formatted at 100% zoom.

Selecting an optional metric and a smaller resolution at the same time shows a chart that is not formatted correctly.

1.7.1

2.5(1a)

CSCvb30039

When SeSparse disks are used in the virtual machine, native snapshots and ReadyClones fails with error:

"Unable to rename VMDK_FILE_NAME.flat.vmdk"

1.8(1a)

2.5(1a)

CSCvb81393

Firmware upgrade fails from Server Storage Controller with unsupported board.

1.8(1c)

2.5(1a)

CSCvb91799

Controller VM may fail to power on during UCS upgrade.

1.8(1c)

2.5(1a)

CSCvc39250

Performance charts could display skewed data if the CPU utilization on nodes in the cluster is very high.

2.0(1a)

2.5(1a)

CSCvd00100

Performance charts show zero or a white space gap or performance IO counters drop to zero during converged node going down or entering/exiting maintenance mode.

1.8(1e)

2.5(1a)

CSCvd37521

A datastore is not mounted on all the nodes in a storage cluster (partially mounted datastore).

2.0(1a)

2.5(1a)

CSCvd37534

In some rare cases, restarting online upgrade on a hyperflex cluster with the previous upgrade in failed state may fail again. Even though the cluster recovered from failure and in a healthy state.

2.0(1a)

2.5(1a)

CSCve01880

During upgrade, if the upgrade procedure fails or is interrupted for any reason, the cleaner process might not restart.

2.1(1b)

2.5(1a)

Install, Upgrade, Expand

CSCvh54563

Generating HyperFlex logs on HX240 platform causes an all paths down state for the cluster.

This issue is caused by the vmw_ahci driver in ESXi 6.5 GA release.

• (Recommended) Upgrade to VMware ESXi 6.5 Update 1.

• Disable the vmw_ahci driver on ESXi host one node at a time, making sure that the cluster is healthy before moving to the next node. Use the following steps:

  1. Run the following command on the ESXi nodes:

     # esxcli system module set --enabled=false --module=vmw_ahci

  2. Reboot the node.

2.5(1d)

CSCvf84968

Combined upgrade using stcli command fails without Cisco UCS Manager and vCenter credentials.

Provide the Cisco UCS Manager and vCenter credentials as part of CLI command. They are not optional.

2.5(1c)

CSCvf82238

After upgrade to a cluster with compute only nodes, some VMs are not managed by EAM.

This occurs when a cluster with compute nodes has been re-registered on a vCenter cluster.

Add the compute node to the HX cluster.

2.5(1c)

Customer created the cluster without any vCenter option and intended to:

1. Create the cluster.

2. Create/mount a datastore.

3. Install vCenter on HyperFlex cluster (nested vCenter).

However, the create/mount datastore step failed with “Mount status failure”.

If you see this issue, do the following:

1. Using SSH, connect to all controller VMs and identify the node that contains the stcli_create_cluster.shfile by running the following command:

   #ls /bin/stcli_create_cluster.sh

2. On that node, secure copy (SCP) the

   /etc/springpath/secure/springpath_keystore.jceks

   file to every other node in the cluster, replacing the original springpath_keystore.jceks file. We recommend that you back up the existing keystore file by running the following command prior to the secure copy operation:

   #mv /etc/springpath/secure/springpath_keystore.jceks /etc/springpath/secure/springpath_keystore.jceks.bak

3. Restart stMgr service on all nodes by running the following command:

   #restart stMgr

For further information, see Troubleshooting TechNote: "How to Deploy vCenter on the HX Data Platform".

2.5(1c)

CSCvf12501

Sometimes, after fresh cluster creation, controller VM memory usage is high and in a critical state.

This is a known VMware issue. See the article, https://kb.vmware.com/s/article/2149787.

2.5(1a)

CSCve73004

UCS Manager does not update the disk firmware status, if a firmware upgrade from 2.1(1b) to 2.5 was initiated by the HX Data Platform.

Perform a soft reset:

# CIMC-soft-rest

2.5(1a)

CSCvc62266

After an offline upgrade, due to a VMware EAM issue, sometimes all the controller VMs do not restart. The stcli start cluster command returns an error: "Node not available".

Manually power on the controller VM and start the cluster.

1. Manually power on the controller VMs.

   • Log in to the vSphere Web Client.

   • Locate the controller VMs that are not powered on. From the vCenter Navigator select, Inventory Lists > Virtual Machines > vm. Storage controller VMs have the prefix, stCtlVM.

   • From the Actions menu, select Power > Power On.

2. Restart the storage cluster.

   • Log in to the command line of any controller VM.

   • Run the command:

     # stcli cluster start

2.0(1a)

CSCvc32497

Cluster creation or expansion fails when UCSM configuration is not chosen as an advanced option in the Cisco HX Data Platform Installer. This happens due to non-reachability of ESX.

In the Cisco HX Data Platform Installer's configuration page, you will see that the default VLAN for hx-inband-mgmt 3091 is tagged to the ESX and not the user-specified VLAN.

Correct the VLAN and retry deploy.

1. Tag the correct VLAN on the ESX by launching KVM console from Cisco UCS Manager.

2. In Cisco HX Data Platform Installer, retry deploy validation.

2.0(1a)

CSCvb94112

HX Installer may be stuck at Cluster Expansion Validation screen during the cluster expansion process.

1. Check logs to verify that the expansion workflow is hung.

2. In your browser, type http://ip_of_installer/api/reset to restart the workflow.

1.8(1c)

CSCvb91838

Cluster expansion failed with no operational DNS server from the list.

If the DNS server becomes non-operational after deployment or cluster creation, add a new operational DNS to the controller. Use the following commands:

# stcli services dns remove --dns <non_operational_dns_ip>

# stcli services dns add --dns <operational_dns_ip>

1.8(1c)

CSCvb29790

Cluster creation fails due to failure to locate vCenter server.

In the vSphere Web Client, change the vCenter host name to an IP address in the config.vpxd.sso.sts.uri variable.

1.8(1a)

Management

CSCvf25130

HX Connect times out after 30 minutes.

When left idle for more than 30 minutes, the HX Connect Virtual Machine page times out. When you return to a page and click anywhere, refreshed data might be incomplete or you might receive the following error: VI SDK invoke exception:; nested exception is: com.vmware.vim25. NotAuthenticated.

Retry refresh HX Connect through the browser or HX Connect buttons. Alternatively, log out of HX Connect and log back in.

This is a known VMware issue. See VMware KB, vCenter Server logs report the error: SOAP session count limit reached (2004663).

2.5(1a)

CSCve17284

Performance charts show a gap for several minutes during an All Flash cluster upgrade.

This is expected behavior because the reporting services are taken down during the upgrade. Only the Reporting Chart is affected, not the actual performance.

2.5(1a)

CSCvd88557

When creating many datastores through the stcli command-line, a temporary error displays indicating that some datastores fail to mount.

This is temporary. However, as the number of datastores being created increases, it takes longer to complete the task and clear the mount error.

2.0(1a)

Replication

CSCvb54848

vSphere Replication Plug-in fails after HX Plug-in is deployed.

To prevent the issue, first install the vSphere Replication plug-in, and then install the HX Data Platform plug-in.

For complete steps for uninstalling required elements and reinstalling them in the supported order, see the 2.5 Release Troubleshooting guide.

1.7.1

CSCvf29202

Recovery might not include disks that are not in the same folder on a datastore as the virtual machine being protected.

If any virtual machine disk resides outside the same folder and datastore of a protected virtual machine:

1. Move the disk to the same folder on the datastore.

2. Then add (re-add) the disk to the virtual machine.

This ensures protection and recovery work successfully.

2.5(1a)

CSCvf27609

Query for recovery job returns summary_step_state and state fields.

Refer to the state field only. Ignore the information in the summary_step_state field.

2.5(1a)

Encryption

CSCvf17183

If a CIMC reboots while a modify-security command is in-progress, and the server is secured with local key management, a subsequent disable-security command may fail, because the server doesn't know the correct key to use.

CIMC was rebooted while a modify-security command was in-progress.

Log in to the controller VM and use the sed-client to update the physical drive keys to match the server's key.

2.5(1a)

CSCvf06510

UCSM might indicate partially disabled encryption security.

No action required. This is a sync issue between reporting interfaces.

To verify from HX Connect, select System Information > Disks > Security. All disks and the controller VM should indicate Security Disabled.

2.5(1a)

CSCvf04240

Encryption may not be enabled on the new node, after it is added to the cluster.

One of the potential causes is that the serial number was not reported correctly from the ESX host.

Restart hostd service on the ESX host, and enable encryption on the cluster from the HX Connect UI. All the nodes that already have encryption enabled are not impacted.

2.5(1a)

CSCve91866

Cannot modify encryption KMIP policy on UCSM to clear an IP address.

UCSM does not allow this behavior. On UCSM, delete the KMIP policy, adjusting for the IP addresses as needed, and retry the task.

2.5(1a)

Install, Upgrade, Expand

CSCvf84968

Combined upgrade using stcli command fails without Cisco UCS Manager and vCenter credentials.

Provide the Cisco UCS Manager and vCenter credentials as part of CLI command. They are not optional.

2.5(1c)

CSCvf82238

After upgrade to a cluster with compute only nodes, some VMs are not managed by EAM.

This occurs when a cluster with compute nodes has been re-registered on a vCenter cluster.

Add the compute node to the HX cluster.

2.5(1c)

Customer created the cluster without any vCenter option and intended to:

1. Create the cluster.

2. Create/mount a datastore.

3. Install vCenter on HyperFlex cluster (nested vCenter).

However, the create/mount datastore step failed with “Mount status failure”.

If you see this issue, do the following:

1. Using SSH, connect to all controller VMs and identify the node that contains the stcli_create_cluster.shfile by running the following command:

   #ls /bin/stcli_create_cluster.sh

2. On that node, secure copy (SCP) the

   /etc/springpath/secure/springpath_keystore.jceks

   file to every other node in the cluster, replacing the original springpath_keystore.jceks file. We recommend that you back up the existing keystore file by running the following command prior to the secure copy operation:

   #mv /etc/springpath/secure/springpath_keystore.jceks /etc/springpath/secure/springpath_keystore.jceks.bak

3. Restart stMgr service on all nodes by running the following command:

   #restart stMgr

For further information, see Troubleshooting TechNote: "How to Deploy vCenter on the HX Data Platform".

2.5(1c)

CSCvf12501

Sometimes, after fresh cluster creation, controller VM memory usage is high and in a critical state.

This is a known VMware issue. See the article, https://kb.vmware.com/s/article/2149787.

2.5(1a)

CSCve73004

UCS Manager does not update the disk firmware status, if a firmware upgrade from 2.1(1b) to 2.5 was initiated by the HX Data Platform.

Perform a soft reset:

# CIMC-soft-rest

2.5(1a)

CSCvc62266

After an offline upgrade, due to a VMware EAM issue, sometimes all the controller VMs do not restart. The stcli start cluster command returns an error: "Node not available".

Manually power on the controller VM and start the cluster.

1. Manually power on the controller VMs.

   • Log in to the vSphere Web Client.

   • Locate the controller VMs that are not powered on. From the vCenter Navigator select, Inventory Lists > Virtual Machines > vm. Storage controller VMs have the prefix, stCtlVM.

   • From the Actions menu, select Power > Power On.

2. Restart the storage cluster.

   • Log in to the command line of any controller VM.

   • Run the command:

     # stcli cluster start

2.0(1a)

CSCvc32497

Cluster creation or expansion fails when UCSM configuration is not chosen as an advanced option in the Cisco HX Data Platform Installer. This happens due to non-reachability of ESX.

In the Cisco HX Data Platform Installer's configuration page, you will see that the default VLAN for hx-inband-mgmt 3091 is tagged to the ESX and not the user-specified VLAN.

Correct the VLAN and retry deploy.

1. Tag the correct VLAN on the ESX by launching KVM console from Cisco UCS Manager.

2. In Cisco HX Data Platform Installer, retry deploy validation.

2.0(1a)

CSCvb94112

HX Installer may be stuck at Cluster Expansion Validation screen during the cluster expansion process.

1. Check logs to verify that the expansion workflow is hung.

2. In your browser, type http://ip_of_installer/api/reset to restart the workflow.

1.8(1c)

CSCvb91838

Cluster expansion failed with no operational DNS server from the list.

If the DNS server becomes non-operational after deployment or cluster creation, add a new operational DNS to the controller. Use the following commands:

# stcli services dns remove --dns <non_operational_dns_ip>

# stcli services dns add --dns <operational_dns_ip>

1.8(1c)

CSCvb29790

Cluster creation fails due to failure to locate vCenter server.

In the vSphere Web Client, change the vCenter host name to an IP address in the config.vpxd.sso.sts.uri variable.

1.8(1a)

Management

CSCvf25130

HX Connect times out after 30 minutes.

When left idle for more than 30 minutes, the HX Connect Virtual Machine page times out. When you return to a page and click anywhere, refreshed data might be incomplete or you might receive the following error: VI SDK invoke exception:; nested exception is: com.vmware.vim25. NotAuthenticated.

Retry refresh HX Connect through the browser or HX Connect buttons. Alternatively, log out of HX Connect and log back in.

This is a known VMware issue. See VMware KB, vCenter Server logs report the error: SOAP session count limit reached (2004663).

2.5(1a)

CSCve17284

Performance charts show a gap for several minutes during an All Flash cluster upgrade.

This is expected behavior because the reporting services are taken down during the upgrade. Only the Reporting Chart is affected, not the actual performance.

2.5(1a)

CSCvd88557

When creating many datastores through the stcli command-line, a temporary error displays indicating that some datastores fail to mount.

This is temporary. However, as the number of datastores being created increases, it takes longer to complete the task and clear the mount error.

2.0(1a)

Replication

CDCvb54848

vSphere Replication Plug-in fails after HX Plug-in is deployed.

To prevent the issue, first install the vSphere Replication plug-in, and then install the HX Data Platform plug-in.

For complete steps for uninstalling required elements and reinstalling them in the supported order, see the 2.5 Release Troubleshooting guide.

1.7.1

CSCvf29202

Recovery might not include disks that are not in the same folder on a datastore as the virtual machine being protected.

If any virtual machine disk resides outside the same folder and datastore of a protected virtual machine:

1. Move the disk to the same folder on the datastore.

2. Then add (re-add) the disk to the virtual machine.

This ensures protection and recovery work successfully.

2.5(1a)

CSCvf27609

Query for recovery job returns summary_step_state and state fields.

Refer to the state field only. Ignore the information in the summary_step_state field.

2.5(1a)

Encryption

CSCvf17183

If a CIMC reboots while a modify-security command is in-progress, and the server is secured with local key management, a subsequent disable-security command may fail, because the server doesn't know the correct key to use.

CIMC was rebooted while a modify-security command was in-progress.

Log in to the controller VM and use the sed-client to update the physical drive keys to match the server's key.

2.5(1a)

CSCvf06510

UCSM might indicate partially disabled encryption security.

No action required. This is a sync issue between reporting interfaces.

To verify from HX Connect, select System Information > Disks > Security. All disks and the controller VM should indicate Security Disabled.

2.5(1a)

CSCvf04240

Encryption may not be enabled on the new node, after it is added to the cluster.

One of the potential causes is that the serial number was not reported correctly from the ESX host.

Restart hostd service on the ESX host, and enable encryption on the cluster from the HX Connect UI. All the nodes that already have encryption enabled are not impacted.

2.5(1a)

CSCve91866

Cannot modify encryption KMIP policy on UCSM to clear an IP address.

UCSM does not allow this behavior. On UCSM, delete the KMIP policy, adjusting for the IP addresses as needed, and retry the task.

2.5(1a)

Install, Upgrade, Expand

CSCvf12501

Sometimes, after fresh cluster creation, controller VM memory usage is high and in a critical state.

This is a known VMware issue. See the article, VM Memory Usage heuristic over-reporting on ESXi 6.5.

2.5(1a)

CSCve73004

UCS Manager does not update the disk firmware status, if a firmware upgrade from 2.1(1b) to 2.5 was initiated by the HX Data Platform.

Perform a soft reset:

# CIMC-soft-rest

2.5(1a)

CSCvc62266

After an offline upgrade, due to a VMware EAM issue, sometimes all the controller VMs do not restart. The stcli start cluster command returns an error: "Node not available".

Manually power on the controller VM and start the cluster.

1. Manually power on the controller VMs.

   • Log in to the vSphere Web Client.

   • Locate the controller VMs that are not powered on. From the vCenter Navigator select, Inventory Lists > Virtual Machines > vm. Storage controller VMs have the prefix, stCtlVM.

   • From the Actions menu, select Power > Power On.

2. Restart the storage cluster.

   • Log in to the command line of any controller VM.

   • Run the command:

     # stcli cluster start

2.0(1a)

CSCvc32497

Cluster creation or expansion fails when UCSM configuration is not chosen as an advanced option in the Cisco HX Data Platform Installer. This happens due to non-reachability of ESX.

In the Cisco HX Data Platform Installer's configuration page, you will see that the default VLAN for hx-inband-mgmt 3091 is tagged to the ESX and not the user-specified VLAN.

Correct the VLAN and retry deploy.

1. Tag the correct VLAN on the ESX by launching KVM console from Cisco UCS Manager.

2. In Cisco HX Data Platform Installer, retry deploy validation.

2.0(1a)

CSCvb94112

HX Installer may be stuck at Cluster Expansion Validation screen during the cluster expansion process.

1. Check logs to verify that the expansion workflow is hung.

2. In your browser, type http://ip_of_installer/api/reset to restart the workflow.

1.8(1c)

CSCvb91838

Cluster expansion failed with no operational DNS server from the list.

If the DNS server becomes non-operational after deployment or cluster creation, add a new operational DNS to the controller. Use the following commands:

# stcli services dns remove --dns <non_operational_dns_ip>

# stcli services dns add --dns <operational_dns_ip>

1.8(1c)

CSCvb29790

Cluster creation fails due to failure to locate vCenter server.

In the vSphere Web Client, change the vCenter host name to an IP address in the config.vpxd.sso.sts.uri variable.

1.8(1a)

Management

CSCvf25130

HX Connect times out after 30 minutes.

When left idle for more than 30 minutes, the HX Connect Virtual Machine page times out. When you return to a page and click anywhere, refreshed data might be incomplete or you might receive the following error: VI SDK invoke exception:; nested exception is: com.vmware.vim25. NotAuthenticated.

Retry refresh HX Connect through the browser or HX Connect buttons. Alternatively, log out of HX Connect and log back in.

This is a known VMware issue. See VMware KB, vCenter Server logs report the error: SOAP session count limit reached (2004663).

2.5(1a)

CSCvf23255

ASUP email notification does not work when multiple Notification Settings are open.

Disable, then re-enable notification settings.

2.5(1a)

CSCve17284

Performance charts show a gap for several minutes during an All Flash cluster upgrade.

This is expected behavior because the reporting services are taken down during the upgrade. Only the Reporting Chart is affected, not the actual performance.

2.5(1a)

CSCvd88557

When creating many datastores through the stcli command-line, a temporary error displays indicating that some datastores fail to mount.

This is temporary. However, as the number of datastores being created increases, it takes longer to complete the task and clear the mount error.

2.0(1a)

CSCuy87775

Datastore failed to mount on all ESX hosts. This is a VMware issue.

Try mounting the datastore one at a time.

1.7.1

Replication

CSCvf29202

Recovery might not include disks that are not in the same folder on a datastore as the virtual machine being protected.

If any virtual machine disk resides outside the same folder and datastore of a protected virtual machine:

1. Move the disk to the same folder on the datastore.

2. Then add (re-add) the disk to the virtual machine.

This ensures protection and recovery work successfully.

2.5(1a)

CSCvf27609

Query for recovery job returns summary_step_state and state fields.

Refer to the state field only. Ignore the information in the summary_step_state field.

2.5(1a)

Encryption

CSCvf17183

If a CIMC reboots while a modify-security command is in-progress, and the server is secured with local key management, a subsequent disable-security command may fail, because the server doesn't know the correct key to use.

CIMC was rebooted while a modify-security command was in-progress.

Log in to the controller VM and use the sed-client to update the physical drive keys to match the server's key.

2.5(1a)

CSCvf06510

UCSM might indicate partially disabled encryption security.

No action required. This is a sync issue between reporting interfaces.

To verify from HX Connect, select System Information > Disks > Security. All disks and the controller VM should indicate Security Disabled.

2.5(1a)

CSCvf04240

Encryption may not be enabled on the new node, after it is added to the cluster.

One of the potential causes is that the serial number was not reported correctly from the ESX host.

Restart hostd service on the ESX host, and enable encryption on the cluster from the HX Connect UI. All the nodes that already have encryption enabled are not impacted.

2.5(1a)

CSCve91866

Cannot modify encryption KMIP policy on UCSM to clear an IP address.

UCSM does not allow this behavior. On UCSM, delete the KMIP policy, adjusting for the IP addresses as needed, and retry the task.

2.5(1a)