KVStore failing to upgrade - SSL/TLS certificate error - unable to get local issuer certificate

Applicability

Description

This article describes a scenario where KVStore upgrade is failing to upgrade due to the following SSL/TLS certificate error indicading unable to verify the first certificate present in $SPLUNK_HOME/var/log/splunk.mongod.log:

I NETWORK [conn778] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unable to verify the first certificate.

Investigation

A failed upgrade of kvstore due to a SSL/TLS certificate error indicading unable to verify the first certificate will show the following in the mongod log:

E  NETWORK [conn778] SSL peer certificate validation failed: unable to verify the first certificate
I  NETWORK  [conn778] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: unable to verify the first certificate.

If these logs are seen, you can run the following command to verify the CAfile:

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>

If it returns unable to get local issuer certificate

This error means that there is something wrong with the CAfile (maybe that the server.pem was renewed but the CAfile was not changed so they don't match)

Resolution

1. Verify that the certificate is not expired: 

   $SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in <file path to serverCert>
   

2. Verify the CAfile:

   $SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>

   If it returns unable to get local issuer certificate, verify it the certificate was recently changed or if the correct CAfile is being used. 
3. Once the correct CAfile is being used, push the KVStore upgrade manually: 

   splunk start-standalone-upgrade kvstore -version 7.0