SSL certificate check failed

Applicability

Description

After adding HFs as search peers to cluster master in order the following can be seen from the monitoring console:

[subsearch]: [PGZUGCHSPLF01] Connection failed with SSL certificate check failed
[subsearch]: [PGZUGCHSPLF01] The REST request on the endpoint URI /services/server/info?count=0 returned HTTP 'status not OK': code=502, Error connecting: SSL certificate check failed.
[subsearch]: [PGZUGCHSPLF01] Unable to determine response format from HTTP Header

Investigation

Look for the following logs in splunkd.log. These errors are related to certificate issues.

06-13-2022 06:01:54.590 +0200 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='unknown CA'.
06-13-2022 06:01:54.590 +0200 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:xxxx. error:xxxxxxx:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.

Splunk is unable to verify the authenticity of the certificate and will refuse connections. Check the sslCommonNameToCheck and sslAltNameToCheck in server.conf:

splunk btool server list sslConfig --debug

The issue experienced in this case is that the heavy forwarder doesn't have its own common name/alt name included in sslCommonNameToCheck or sslAltNameToCheck causing server certificate validation to fail and creating an error during the REST search.

Resolution

Add the appropriate hostname values into the sslCommonNameToCheck and sslAltNameToCheck in server.conf using the following syntax:

[sslConfig]
sslCommonNameToCheck = <commonName1>, <commonName2>, ...
sslAltNameToCheck = <alternateName1>, <alternateName2>, ...

Restart Splunk after making these changes.

Reference document (server.conf):
SSL/TLS Configuration details

Related Links

SSL/TLS Configuration details