Introduction
This document describes the failure issue of HyperFlex vCenter registration after VCSA certification regeneration.
After the regeneration/replacement of the certificate on vCenter Server, the EAM service is unaware of the new certificate and is unable to log in.
Contributed by Ignacio Orozco, Cisco TAC Engineer.
Background Information
When you attempt to reregister the Hyperflex cluster to vCenter after the regeneration/replacement of the vCenter certificate, you can see this error:
"Connection error between vCenter and ESX Agent Manager (EAM). From vSphere, restart the EAM services. Then click Retry Cluster Creation."
In the stMgr.log you can see this:
017-12-20-21:51:17.015 [opId=6b747f93e3224869, operationId=6b747f93e3224869.6b747f93e3224869<:6b747f93e3224869] [pool-4-thread-2] ERROR c.s.s.v.VirtPlatformImpl$ - ESX Agent Manager error
java.rmi.RemoteException: VI SDK invoke exception:; nested exception is:
com.vmware.eam.NoConnectionToVCenter
at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:157) ~[vijava-eam-5.5.6sp.jar:na]
at com.vmware.eam.ws.EamService.queryAgency(EamService.java:144) ~[vijava-eam-5.5.6sp.jar:na
In the VCSA: /var/log/vmware/eam/eam.log file OR the VC on Windows server C:\ProgramData\VMware\vCenterServer\logs\eam\eam.log file for the ESX Agent Manager (EAM), you see entries similar to:
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80
YYYY-MM-DDTHH:MM:SS.MSZ | DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-7502fb4c-3521-48c7-93ed-3d1865e0fff1, subject domain: vsphere.local
YYYY-MM-DDTHH:MM:SS.MSZ | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error
(vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
at sun.reflect.GeneratedConstructorAccessor82.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFau
Workaround
Reference document: https://kb.vmware.com/s/article/2112577
On vCenter Server for Windows
- Connect to vCenter Server through a Console or Remote desktop session.
- Open an elevated command prompt.
- Create a temporary directory named c:\certificate.
- Run this command to retrieve the vpxd-extension solution user certificate and key:
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt
"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key
- Navigate to C:\Program Files\VMware\vCenter Server\vpxd\scripts:
cd C:\Program Files\VMware\vCenter Server\vpxd\scripts
Note: The path listed is for a default install of vCenter Server. If you have customized the install location of vCenter Server, change the directory accordingly.
- Run this command to update the extension's certificate with vCenter Server:
"%VMWARE_PYTHON_BIN%" updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificate\vpxd-extension.crt -k C:\Certificate\vpxd-extension.key -s localhost -u Administrator@domain.local
Note: The default user and domain are administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment.
- When prompted, enter the Administrator@domain.local password.
- Navigate to the C:\Program Files\VMware\vCenter Server\bin folder:
cd C:\Program Files\VMware\vCenter Server\bin
- Restart the VMware ESX Manager service with these commands:
- service-control --stop EsxAgentManager
- service-control --start EsxAgentManager
On the vCenter Server Appliance
- Log in to the vCenter Server Appliance using SSH.
- Run this command to enable access the Bash shell:
shell.set --enabled true
- Type shell and press Enter.
- Run this command to retrieve the vpxd-extension solution user certificate and key:
mkdir /certificate
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
- Run this command to update the extension's certificate with vCenter Server.
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u Administrator@domain.local
Note: The default user and domain are Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.localpassword.
- Restart the VMware ESX Manager service with these commands:
service-control --stop vmware-eam
service-control --start vmware-eam
Feedback