KV store fails to start with custom or self-signed certificates

Applicability

Description

This issue is related to an incorrectly signed certificate where the x509 extensions is set incorrectly for Splunk KV store to work.

Check the Splunk KV Store status with this command:

$SPLUNK_HOME/bin/splunk show kvstore-status --verbose

And it will show that it has failed to start:

 This member:
		           backupRestoreStatus : Ready
		                      disabled : 0
		   featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: X.X.X.X:8191]
		                          guid : XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
		                          port : 8191
		                    standalone : 1
		                        status : failed
		                 storageEngine : wiredTiger
		      versionUpgradeInProgress : 0

The following warning and error are also found in $SPLUNK_HOME/var/log/splunk/splunkd.log:

WARN MongoClient [3578138 KVStoreConfigurationThread] - Disabling TLS hostname validation for localhostERROR KVStoreConfigurationProvider [3578138 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=301816ms

Investigation

Verify if the certificate is correctly signed:

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile  <CA_FILE.pem> <certificate.pem>

Expected output:

/opt/splunk/etc/auth/mycerts/enterprise.pem: OK 

Inspect the Splunk KV store certificate information using the following command:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in <certificate.pem> -text -noout

Examine the X509v3 extensions section in the output:

        X509v3 extensions:
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication

The X509v3 Extended Key Usage only includes TLS Web Server Authentication. This is causing Splunk KV Store to fail to start.

Resolution

As per the MongoDB documentation on certificate requirements:

If the certificate used as the certificateKeyFile includes extendedKeyUsage, the value must include both clientAuth ("TLS Web Client Authentication") and serverAuth ("TLS Web Server Authentication").

Verify the certificate used for Splunk KV store meets the requirements with the following command:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in <path to certificate file> -text -noout

The output should show the certificate includes both TLS Web Server Authentication, TLS Web Client Authentication under the X509v3 Extended Key Usage:

        X509v3 extensions:
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication

Related Links

KV store Prerequisites

Preparing custom certificates for use with KV store

MongoDB certificate requirements