SSL Certificate Verification Failure on Splunk Forwarder Due to Common Name Conflict and Unknown CA; SSL ERROR X509Verify - X509 certificate

Applicability

Description

This issue typically arises in the following scenarios:

1. A brand new installation of a Universal Forwarder.
2. Migration from on-prem to Splunk Cloud.
3. Re-installation of the Splunk credential package.

The error might occur after successfully installing multiple forwarders (e.g., 10 forwarders), but only one reports this issue. With a fresh installation, all configurations are usually copy-pasted, and the installation process is repeated identically. As a result, when reviewing the configuration, everything appears to be the same and all other forwarders have successfully connected to Splunk except just this one. 

Splunk can flag an issue with a certificate that was actually provided by Splunk itself. In this situation, you are not using a 3rd party certificate, but rather the Splunk certificate package that was downloaded directly.

The primary error in this issue is ERROR X509Verify - X509 certificate, which can be easily overlooked amidst multiple errors and warnings.  The errors and warnings that are usually reported with this issue:

WARN SSLCommon [1114401 FwdDataReceiverThread] - Received fatal SSL3 alert. ssl_state='error', alert_description='handshake failure'.
WARN SSLCommon [1114458 FwdDataReceiverThread] - Received fatal SSL3 alert. ssl_state='SSLv3 read client certificate A', alert_description='bad certificate' 
WARN SSLCommon [18101 TcpOutEloop] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'.

ERROR TcpOutputFd [3622596 TcpOutEloop] - Connection to host=XX.XX.XX.XX:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.
ERROR X509Verify [18101 TcpOutEloop] - Server X509 certificate (CN=Root CA) failed validation; error=19, reason="self signed certificate in certificate chain"

When users see the error about the server and CA certificate having the same Common Name, it can be misleading. This error, while detailed, tends to distract from the real issue. The most critical error — the failure to validate a self-signed certificate — frequently goes unnoticed in these situations, even though it's the root cause of the problem. It's essential to focus on the actual certificate validation failure rather than getting sidetracked by other descriptive errors.

Investigation

To effectively diagnose the problem, it's crucial to gather specific information to determine whether the issue is due to a failed certificate check during the SSL handshake.

The answer to both questions below must be YES

1) Is the installed certificate a self-signed certificate provided by Splunk? - YES

2) On the forwarder is the setting sslVerifyServerCert set to true? - YES

Example on the UF:

../etc/system/default/server.conf 
[applicationsManagement]                       
sslVerifyServerCert = true

Run this command on the forwarder to check.

splunk $ splunk btool --debug server list |grep sslVerifyServerCert

To double-check if the instance's side is not blocking the SSL connection (e.g firewall proxy) run the following command :

UNIX: curl -v -k https://http-inputs-<stackname>.splunkcloud.com/services/collector/raw -H 'Authorization: Splunk <token>' -d '{"event":"Hello, World!"}'Win: curl -v -k https://http-inputs-<stackname>.splunkcloud.com/services/collector/raw -H "Authorization: Splunk <token>" -d "{\"event\":\"Hello, World!\"}" 

Why do I get this error when configuring the universal forwarder: SSL routines: :certificate verify failed? 

Resolution

The root cause of this issue is that the self-signed certificate is failing to pass Splunk's certificate validation process. This is a rare issue where Splunk does not accept its own self-signed certificate, even though it works fine on other forwarders. 

A self-signed certificate essentially means that the entity is verifying itself — like saying, "I am who I say I am because I say so." This is different from a certificate signed by a private certificate authority (CA), which has a root certificate (likely self-signed) that establishes a trust anchor for any other certificates it issues.

In this case, the error "ssl3_get_server_certificate:certificate verify failed" occurs because the certificate can't be validated. If sslVerifyServerCert is set to true and Splunk can't verify the certificate, the connection fails. The best course of action is to review the certificate settings and potentially adjust the verification parameters to address this issue.

Change:
Depending on whether or not it is self-signed (Splunk uses a self-signed cert), make this change to the settings. This doesn't apply if it's a 3rd party certificate. 

../etc/system/default/server.conf 
[applicationsManagement]                       
sslVerifyServerCert = false

Verify
Run this command on your forwarder to check.

splunk $ splunk btool --debug server list | grep sslVerifyServerCert

Note: As the certificate comes from Splunk and is self-signed, and Splunk is managing its self-signed certificate,  it is safe and that's why you can turn off the setting. This doesn't raise any risk nor impact the high level of encryption. It can be turned off until it's decided to use a 3rd party certificate