Troubleshooting

Verifying and Recreating the HX User Account

Procedure

Step 1

Check for missing hxuser account:

  1. Secure Shell (SSH) to all esxi hosts.

  2. Type esxcli system account list.

  3. Verify that the hxuser account is present on all esxi hosts. If the hxuser account is missing, proceed to step 2.
Step 2

Secure Shell (SSH) to any controller:

  1. Type cd /opt/springpath/storfs-mgmt/hxtoolbox-1.0/bin.

  2. Type ./hxtoolbox –u.

    Note 
    If you see an error, open a TAC case.

  3. Verify that the huxser is created on the esxi hosts (Repeat step 1).
Step 3

Rerun the upgrade.

Verifying and Recreating the Springpath User Account

Procedure

Step 1

Check for missing Springpath user account:

  1. Secure Shell (SSH) to all esxi hosts.

  2. Type esxcli system account list.

  3. Verify that the Springpath user account is present on all esxi hosts. If the Springpath user account is missing, proceed to step 2.

Step 2

On the ESXi hosts where springpath user is missing:

  1. Type esxcli system account add -i springpath -p spr!n9p@th -c spr!n9p@th.

  2. Type esxcli system permission set -i springpath -r Admin.

  3. Verify that the Springpath User is present on missing nodes (Repeat step 1).
Step 3

Rerun the upgrade.

After Upgrade Cluster Shows Offline from CLI/nodes Show Offline in vCenter HXDP Plugin Summary

Procedure

Step 1

Recreate Springpath_security.properties link:

  1. Secure Shell (SSH) to all controller VMs.

  2. Type ls /usr/share/springpath/storfs-misc/ and confirm the springpath_security.properties link exists on all controllers.

  3. Type ls /etc/springpath/secure to confirm that the springpath_security.properties file exists on all nodes.

Step 2

If the file is missing, copy the file from another node to the missing node:

Example (from controller with the file): 
scp /etc/springapth/secure/springapth_security.properties root@10.10.10.5:
/etc/springpath/secure/springpath_security.properties
Step 3

If the alias link is missing, you must recreate the link on the missing node:

Example:
/etc/springpath/secure/springpath_security.properties

  1. Type ln -s /usr/share/springpath/storfs-misc/springpath_security.properties /etc/springpath/secure/springpath_security.properties.

  2. Type restart stMgr.

Step 4

Verify the fix:

  1. Type stcli cluster info and confirm that the vCluster state is online.

  2. In the HXDP vCenter plugin, confirm that the nodes all show as online in the summary section.

Update Net.Team Policy

Perform the following steps on ALL ESXi Hosts:

Procedure

Step 1

Confirm current value is set to 100. 

[root@JR-NODE-1:~] esxcli system settings advanced list -o /Net/TeamPolicyUpDelay
Step 2

Change /Net/TeamPolicyUpDelay --int-value from 100 (Default value) to 30000. 

[root@JR-NODE-1:~] esxcli system settings advanced set -o /Net/TeamPolicyUpDelay --int-value 30000
Step 3

Confirm the new value of /Net/TeamPolicyUpDelay --int-value is 30000. 

[root@JR-NODE-1:~] esxcli system settings advanced list -o /Net/TeamPolicyUpDelay 
 Path: /Net/TeamPolicyUpDelay 
 Type: integer
 Int Value: 30000 <— Value set in Step-1 above from 100(Default value) to 30000
 Default Int Value: 100
 Min Value: 0
 Max Value: 600000 String Value:
 Default String Value:
 Valid Characters:
 Description: Delay (ms) before considering an 'uplink up' event relevant

Precheck Validation Failure Due to an Algorithm Change in 6.0 U3

While upgrading HX, with version 6.0 U3 and later, we see following error: Failed upgrade validations : Checking vCenter configuration. Reason: Upgrade validations failed. Failed to query ESX version on host X.X.X.X

root@SpringpathControllerXXXX:~# stcli cluster upgrade --components hxdp --location 
/tmp/storfs-packages-3.5.2d-31738.tgz --vcenter-user administrator@XXX.com 
…Waiting for upgrade validations to finish... ['Checking vCenter configuration']
Failed upgrade validations : Checking vCenter configuration. Reason: Upgrade validations
 failed. Failed to query ESX version on host: X.X.X.X

From ESXi  /var/run/log/auth.log

2019-07-03T19:56:18Z sshd[757807]: Connection from X.X.X.Xport 42416
2019-07-03T19:56:18Z sshd[757807]: Unable to negotiate with X.X.X.Xport 42416: 
no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,
diffie-hellman-group-exchange-sha1 [preauth]

Procedure

Step 1

Workaround: On each ESXI host in the cluster, edit the /etc/ssh/sshd_config file. Add KexAlgorithms diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1 to the end of the file.

Step 2

Rerun the upgrade command or resume from UI.

Pycrypto Minimum Version

Verify the minimum package version is 2.6.

EXAMPLE:
 $ pip show pycrypto
---
Name: Jinja2
Version: 2.6.1

Upgrading a Version before HyperFlex Release 1.8(1a)

Contact TAC to upgrade the package.

Extra or Duplicate stNode is Present in stcli after Upgrade

Contact TAC.

Duplicate pnode is Present in stcli cluster info after Upgrade

Contact TAC.