Logs not getting indexed in Splunk (Certificate errors).

Applicability

Description

Certificate errors can prevent log data from flowing and can potentially cause several other issues with communication. Ensuring the certificates work properly on the systems they are deployed to is important for proper operation. 

Investigation

When running "./splunk list inputstatus" on a universal forwarder installed on a Linux server, a message was seen stating "splunkd is not running and Splunk is down". When running ./splunk status, splunkd is actually running. The following errors may be seen in the logs:

error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please 
check the output of the `openssl verify`splunkd ERROR HttpClientRequest 11-30-2020 14:07:11.907 +0100 ERROR HttpClientRequest - 
Caught exception while parsing HTTP reply: Unexpected character while looking for value: splunkd ERROR IndexerDiscoveryHeartbeatThread 11-30-2020 14:07:11.907 +0100 ERROR 
IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify 
that the pass4SymmKey set under [indexer_discovery:primary_indexers] in 'outputs.conf' 
matches the same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. 
[uri=https://X.X.X.X:8089/services/indexer_discovery http_code=502 
http_response="Unauthorized"]splunkd ERROR TcpOutputFd 11-30-2020 14:06:50.955 +0100 ERROR TcpOutputFd - Connection 
to host=X.X.X.X:9998 failed. sock_error = 0. SSL Error = error:14090086:SSL 
routines:ssl3_get_server_certificate:certificate verify failed - please check the output 
of the `openssl verify` command for the certificates involved; note that if certificate 
verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA 
certificate and the server certificate should not have the same Common Name. splunkd ERROR X509Verify 11-30-2020 14:06:50.955 +0100 ERROR X509Verify - X509 certificate 
(emailAddress=something@domain.com,CN=domain.com,OU=test,O=test,
L=California,ST=California,C=com) failed validation; error=10,reason="certificate has expired"splunkd ERROR HttpClientRequest 12-01-2020 10:32:49.622 +0100 ERROR HttpClientRequest - 
Caught exception while parsing HTTP reply: Unexpected character while looking for value: splunkd ERROR IndexerDiscoveryHeartbeatThread 12-01-2020 10:32:49.622 +0100 ERROR 
IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the 
pass4SymmKey set under [indexer_discovery:primary_indexers] in 'outputs.conf' matches the 
same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. 
[uri=https://X.X.X.X:8089/services/indexer_discovery http_code=502 
http_response="Unauthorized"]splunkd WARN SSLCommon 12-01-2020 10:32:50.827 +0100 WARN SSLCommon - Received fatal SSL3 
alert. ssl_state='SSLv3 read server certificate B', alert_description='certificate expired'.splunkd ERROR TcpOutputFd 12-01-2020 10:32:50.827 +0100 ERROR TcpOutputFd - Connection to 
host=X.X.X.X:9998 failed. sock_error = 0. SSL Error = error:14090086:SSL 
routines:ssl3_get_server_certificate:certificate verify failed - please check the output of 
the `openssl verify` command for the certificates involved; note that if certificate 
verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA 
certificate and the server certificate should not have the same Common Name.

Resolution

The issue occurs when Splunk intercepts/decodes the network packets and then re-encrypts the SSL stream with its own internal CA Root keys. To solve the issue, follow the instructions below:

1. Find the appsCA.pem file in $SPLUNK_HOME/etc/auth on the Splunk instance.
2. Back this file up before making any changes.
3. Open the file with a text editor and copy the internal CA Root to the appsCA.pem, for example:
   

   -----BEGIN CERTIFICATE-----
   SPLUNK CERT HERE
   -----END CERTIFICATE-----
   -----BEGIN CERTIFICATE-----
   CUSTOMER INTERNAL CA ROOT HERE
   -----END CERTIFICATE-----

4. Restart Splunk to apply these changes.