Splunk SSL Certificate Issue - TCPOutAutoLB-0

Applicability

Description

The following error message shows up in the Splunk UI alert section:

TCPOutAutoLB-0
• Root Cause: More than 70% of forwarding destinations have failed. 
Ensure your hosts and ports in outputs.conf are correct. 
Also ensure that the indexers are all running, and that any SSL certificates being used 
for forwarding are correct.

Investigation

The best way to identify the issue is as follow:

This error is seen in Splunk Web:

 there are 3 certificates to check:

openssl x509 -noout -modulus -in <your_cert>.crt | openssl md5

(stdin)= 509c1a81eaXXXXXXXXXXXXXb471 - these all need to match

openssl rsa -noout -modulus -in <your_key>.key | openssl md5

(stdin)= 509c1a81eaXXXXXXXXXXXa21b471 - these all need to match

openssl x509 -noout -modulus -in <your_cert>.pem | openssl md5

(stdin)= 3d8acXXXXc12bcXXXXXba8987c8c0 - these all need to match

This error is commonly caused because there are expired certificates, check the SSL certificates on the server.conf and output.conf from the forwarder and indexers.

Also verify:

***Check the expiration date of the cert

openssl x509 -enddate -noout -in cacert.pem

***Verify that your Root Cert did, in fact, issue your Server Certificate:
 

openssl verify -verbose -CAfile cacert.pem server.pem

Resolution

If the certificates are expired they will need to be renewed and installed.

Renew existing TLS certificates

and

How do I renew an expired Splunk Certificate?