Check if hxuser is present in Hyperflex nodes & if it doesn't exist how to add using hxtoolbox or manually add hxuser to Hyperflex Nodes

Introduction

Hyperflex nodes require the special user "hxuser" to be present for certain cluster functionality. 

A missing hxuser will frequently cause problems during upgrades.

Use below ESXi commands to verify whether the hxuser exists with the correct permissions in each ESXi node:

ESXi# esxcli system account list
User ID  Description
-------  -------------------------------------------
root     Administrator
dcui     DCUI User
vpxuser  VMware VirtualCenter administration account
hxuser   HX Platform Internal User Account    


ESXi# esxcli system permission list
Principal  Is Group  Role   Role Description
---------  --------  -----  ------------------
dcui          false  Admin  Full access rights
hxuser        false  Admin  Full access rights
root          false  Admin  Full access rights
vpxuser       false  Admin  Full access rights

If hxuser does not exist in any node, the hxuser can be added back to Hyperflex nodes using the below methods.

Note: Try the methods explained below in the same order as they appear and only move to the next method should the preceding one fail

Option#1 :

Add hxuser by using hxtoolbox from HX Cluster management VIP

This is the preferred method always instead of  the manual procedure  Option # 2.

Open ssh connection to HX cluster mgmt VIP as root user and run the following commands:

SCVM# cd /opt/springpath/storfs-mgmt/hxtoolbox-1.0/bin 
SCVM# ./hxtoolbox -u <esx-username> <esx-password>

Example:
SCVM# ./hxtoolbox -u root abcdefg

If successful, this will create the hxuser on all nodes. Cross check if created on all nodes & permissions are set alright as explained in the Introduction section above. 

Option#2 :

Manually add the user in each ESXi node

Perform this step only if adding hxuser via hxtoolbox method doesn't work.

Open an ssh connection the HX cluster mgmt VIP as root user. 

Run the following commands (ignore any errors displayed after entering the first line)

PYTHONPATH=/opt/springpath/storfs-mgmt-cli/stCli-1.0-py2.7.egg:/usr/share/thrift-0.9.1.a-py2.7-linux-x86_64.egg python -i -m stCli.stCli
>>> s = StCli()
>>> creds = s.stMgrBase.getEsxCredential()
>>> print creds

After entering the last line, the output will be seen similar to the following

EsxCredential(username='hxuser', password='=nfiARX(yZ|[#9Ab')

Note down the Password (everything between the single quotes). In this example, the password is =nfiARX(yZ|[#9Ab

The hxuser can then be manually created on each Hyperflex node (in ESXi) by running the following commands on ESXi host as root user

Note: Single quotes or double quotes may be required to delimit the password. You may have to change which quotation is used if a single quote or double quote exists in the password

ESXi# esxcli system account add -i hxuser -p <password from above output> -c <password from above output> -d "HX Platform Internal User Account"
Example :
[root@node1:~] esxcli system account add -i hxuser -p '=nfiARX(yZ|[#9Ab' -c '=nfiARX(yZ|[#9Ab' -d "HX Platform Internal User Account"
ESXi# esxcli system permission set -i hxuser -r Admin

ssh <IP Address of ESXi node>
login as: hxuser
Password:                 
[hxuser@node1:~]