KVStore in failed state due to SSL certificate expiry

Applicability

Description

• When observing the following error message on Splunk Enterprise instance, it may be related to certificate exipry.

“Search peer XXXXXX.domain.local has the following message: Failed to start KV Store process. See mongod.log and splunkd.log for details.”

• Under the splunkd.log file, look for mongodRunner process and check if the mongod is exited abnormally with exit code 14 as shown in the sample log below:

ERROR MongodRunner - mongod exited abnormally (exit code 14, status: exited with code 14) - look at mongod.log to investigate

• Upon further investigation, the mongod.log file reveals that the SSL certificate is expired as the root cause for KV Store failure as shown in the sample log below: 

No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
NETWORK  The provided SSL certificate is expired or not yet valid.
Fatal Assertion 28652

Investigation

• This can happen if the certificate used by Splunkd to communicate with Mongod has expired. Verify the certificates in use are valid. For example, to validate the expiration date for server.pem use this command:

  From $SPLUNK_HOME/etc/auth/

openssl x509 -enddate -noout -in ./server.pem

• The output display would look similar to the sample shown below:

notAfter=Dec  10 14:017:25 2015 GMT

• If the date displayed is before the present date, the certificate is expired.

Resolution

• If the cert has the date that is expired. A new certificate can be created using the creatssl command. Review the syntax and more using the following help command:

$SPLUNK_HOME/bin/splunk help createssl

• An example of the createssl command is shown below as a sample:

$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048

• Make sure to adjust the environment requirements/settings before the certificate is in place.
• Once the new certificate is in place, check its validity by running the following command:

           From $SPLUNK_HOME/etc/auth/

openssl x509 -enddate -noout -in ./server.pem

• The output display would look similar to the sample shown below:

notAfter=Aug 22 15:30:45 2019 GMT

• If the output displayed suggests a valid date in future, restart splunk process on the instance using the following command:

/opt/splunk/bin/Splunk restart

• Validate if the KV Store is up and running using the following command:

ps -ax | grep mongod

• A sample output similar to the one below must be displayed:

ps -ax | grep mongod26108 ?        Ssl   62:11 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=50D25A40-7DD2-4017-A223-732705AD4A96 --sslAllowInvalidHostnames --sslMode=preferSSL --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --nounixsocket

•  Another way to check is to use rest call via search using either of the two commands as mentioned below:

| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus

or

splunk _internal call /services/server/info |grep -i kvstore

• The output must look like as shown in the sample output below:

<s:key name="kvStoreStatus">ready</s:key>​​​​​​​

​​​​

Related Links

Related Knowledge Articles

ERROR: KV Store changed status to failed, KV Store process terminated abnormally (exit code 14, status exited with code 14), F NETWORK The provided SSL certificate is expired or not yet valid" is seen in mongod.log

Expired TLS certificate and KV Store will not start, exit code 14 fatal assertion 28652

Splunk Answers

KV Store Fails To Start