Logs are not flowing to splunk cloud from Heavy Fowarder - SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

Applicability

Description

$ cat $SPLUNK_HOME/var/log/splunk/splunkd.log| egrep “ERROR|WARN”
WARN SSLCommon [23564 TcpOutEloop] - Received fatal SSL3 alert. ssl_state='error', alert_description='unknown CA'.

Investigation

Upon investigation, it was identified that the UF  has credential package which is directly copied from Splunk Cloud,  while the  HF is getting UF credentials package from Deployment Server(DS)

1. Checked the connectivity from HF to Splunk Cloud which has no issues. 

$SPLUNK_HOME/splunk cmd openssl s_client -connect inputs1.XXXXX.splunkcloud.com:9997

2. Executed the command to check pem file expiry which looks good too

$SPLUNK_HOME/bin/openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem

3. Followed steps mentioned in "Install the forwarder credentials on many forwarders using a deployment server in Windows"  but it didn't resolve the issue

Install and configure the Splunk Cloud Platform universal forwarder credentials package

3. It was identified that the problem is the HF using a cloud cert but also uses cert signed by an internal CA to connect to the DS

Execute the btool command on server.conf and outputs.conf to know about certificate details which clearly shows that it has self signed internal CA to connect to the DS

$SPLUNK_HOME/bin/splunk btool outputs list --debug
/opt/splunk/etc/system/default/outputs.conf [syslog]
/opt/splunk/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunk/etc/system/default/outputs.conf priority = <13>
/opt/splunk/etc/system/default/outputs.conf type = udp
/opt/splunk/etc/apps/100_enbridge_splunkcloud2/local/outputs.conf [tcpout]
/opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunk/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunk/etc/apps/100_enbridge_splunkcloud2/local/outputs.conf channelReapInterval = 60000
/opt/splunk/etc/apps/100_enbridge_splunkcloud2/local/outputs.conf channelReapLowater = 10
/opt/splunk/etc/apps/100_enbridge_splunkcloud2/local/outputs.conf channelTTL = 300000$SPLUNK_HOME/bin/splunk btool server list --debug
/opt/splunk/etc/system/default/server.conf                             [app_backup]
/opt/splunk/etc/system/default/server.conf                             backup_path = $SPLUNK_HOME/var/backup
/opt/splunk/etc/apps/100_<app_name>_splunkcloud2/default/server.conf     sslRootCAPath = $SPLUNK_HOME/etc/apps/100_enbridge_splunkcloud2/default/<name>_cacert.pem
/opt/splunk/etc/system/default/server.conf                             [applicationsManagement]
/opt/splunk/etc/system/default/server.conf                             allowInternetAccess = true
/opt/splunk/etc/system/default/server.conf                             caCertFile = $SPLUNK_HOME/etc/auth/appsCA.pem

Resolution

Steps to resolve:

1. Create a new custom TA to manage the SSL configuration for the UF to DS connection.

2. Combine the Splunk Cloud root cert with the custom CA root cert in one file and include it in this TA.

cat theirRootCA ourRootCA > newCombinedRootCA.pem

and set the permissions to 600:

chmod 600 newCombinedRootCA.pem

3. Add a server.conf file with the SSL config, referencing the custom cert in "serverCert" and the combined root CA cert file in the "sslRootCAPath".

Please make sure the ---BEGIN--- and ---END LINES-- are present after each cert or it can break and set the combined Root CA in  server.conf of UF creds package

Below is the sample path where modification needs to be done.

Please make the  $SPLUNK_HOME/etc/apps/100_<app_name>_splunkcloud2/default/server.conf changes in below stanza

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/apps/<yourapp>/certs/combinedCA.pem

4. Added values for "sslVerifyServerCert" and "sslCommonNameToCheck", although this isn't required for the setup to work. If the requirement is to force the Universal Forwarder(UFs) to check they are connecting only to the correct DS FQDN.

Please be noted that the /opt/splunk/etc/apps/100_<app_name>_splunkcloud2/local/outputs.conf file didn't need to be adjusted to reference the new combined root CA file.

Related Links

Install and configure the Splunk Cloud Platform universal forwarder credentials package