ERROR [HydraGatekeeper] [ValidateChallengeKey] could not validate hydra gateway challenge while collecting data using Splunk addon for VMware.

Applicability

Description

This article explains the issues that occur when the admin user is absent on the DCN and DCS instances, which result in the following error: "[HydraGatekeeper] [ValidateChallengeKey] could not validate hydra gateway challenge." Below are the error that are observed in hydra_gatekeeper.log.

ERROR [HydraGatekeeper] [ValidateChallengeKey] could not validate hydra gateway challenge: <urlopen error [Errno 111] Connection refused>

Investigation

Below are the following errors which are present in the logs:

06-19-2023 11:55:39.694 +0200 ERROR ExecProcessor [2106 ExecProcessor] - Invalid user admin, provided in passAuth argument, attempted to execute command /apps/splunk/bin/python3.7 /apps/splunk/etc/apps/Splunk_TA_vmware/bin/ta_vmware_hierarchy_agent.py06-19-2023 13:08:44.746 +0200 ERROR ExecProcessor [2029 ExecProcessor] - Invalid user admin, provided in passAuth argument, attempted to execute command /apps/splunk/bin/python3.7 /apps/splunk/etc/apps/SA-Hydra/bin/bootstrap_hydra_gateway.py

During investigation, it was observed that certain scripts in SA-Hydra and Splunk_TA_VMware have the passAuth parameter value hardcoded to admin. If the admin user is deleted or renamed, the above error occurs, and the script execution is blocked.

Resolution

User can resolve the error using the steps mentioned below on DCS machines:

• SSH to the DCS machine and navigate to the $SPLUNK_HOME/etc/apps/Splunk_TA_vmware/local/.

• Edit the inputs.conf and add the passAuth parameter value to a existing user which has the splunk_vmware_admin role for eg. splunkadmin. The inputs.conf will look like as below:

  [ta_vmware_collection_scheduler://puff] 
  disabled = 0 [script://$SPLUNK_HOME/etc/apps/Splunk_TA_vmware/bin/ta_vmware_hierarchy_agent.py] disabled = 0 
  passAuth = splunkadmin

• Save the inputs.conf and restart the splunk.

Below are the and steps that needs to be performed on DCN machines:

• SSH to the DCN machine and navigate to the $SPLUNK_HOME/etc/apps/SA-Hydra/local/.

• Edit the inputs.conf and add the passAuth parameter value to a existing user which has the splunk_vmware_admin role for eg. splunkadmin. The inputs.conf will look like as below:

  [script://$SPLUNK_HOME/etc/apps/SA-Hydra/bin/bootstrap_hydra_gateway.py] 
  disabled = 0 
  passAuth = splunkadmin

• Save the inputs.conf and restart the splunk.

Related Links