KV Store initialization has failed - KVStore crashes/stop working with multiple network connections messages after changing SSL Certificates

Applicability

Splunk Enterprise Splunk Search Heads Splunk Heavy Forwarder

Description

Receiving the following error during initial Splunk booting phase

08-26-2022 18:36:04.568 +0000 ERROR SearchScheduler [1646665 AlertNotifierWorker-0] - Error in 'outputlookup' command: External command based lookup'em_entity_cache' is not available because KV Store initialization has failed. Contact your system administrator., search='| eval _dimensions_kv_lookup=lower(dimensions_kv) \n| outputlookup em_entity_cache key_field=key'

After checking the KVSTORE status this one shows STARTING for a minute then it changes to FAILED and Splunk stops working.

Investigation

Enable the Splunk environment, so the commands use the OpenSSL version that comes with Splunk.

source /opt/splunk/bin/setSplunkEnvie:
source /opt/splunk/bin/setSplunkEnv
Tab-completion of "splunk <verb> <object>" is available.

Verify the status of the KVStore

$SPLUNK_HOME/bin/splunk show kvstore-statusie:
/opt/splunk/bin/splunk show kvstore-status
 This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : 09BA2836-D80B-4A3B-8EAD-C0A204605691
                                          port : 8191
                                    standalone : 1
                                        status : failed
                                 storageEngine : mmapv1

Make sure the server.pem cert is valid by running the following command from CLI

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in $SPLUNK_HOME/etc/auth/<full_chain_cert_name>.pem -noout -textie:
/opt/splunk/bin/splunk cmd openssl x509 -in /opt/splunk/etc/auth/my_certs/server.pem -noout -text
Certificate:
(...)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=CA, L=San Francisco, O=Splunk, CN=SplunkCommonCA/emailAddress=support@splunk.com
        Validity
            Not Before: Dec  8 17:36:36 2022 GMT
            Not After : Dec  7 17:36:36 2025 GMT
        Subject: CN=SplunkServerDefaultCert, O=SplunkUser
(...)

Verify the $SPLUNK_HOME/var/log/splunk/mongod.log, on a duplicated terminal while booting the splunkd.service at the same time.

2022-09-13T18:52:41.551Z I CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist
2022-09-13T19:01:45.690Z I NETWORK [listener] connection accepted from 127.0.0.1:xxxxx #1 (1 connection now open)
2022-09-13T19:01:45.691Z I NETWORK [connl] end connection 127.0.0.1:xxxxx (0 connections now open)
2022-09-13T19:01:45.770Z I NETWORK [listener] connection accepted from 127.0.0.1:xxxxx #1 (1 connection now open)
2022-09-13T19:01:45.771Z I NETWORK [connl] end connection 127.0.0.1:xxxxx (0 connections now open)
(...)
2022-09-13T19:06:45.111Z I CONTROL  [signalProcessingThread] shutting down with code:0

Found a 1000 messages loop of open and closed connections from the NETWORK component, after this Splunk crashes and the KVSTORE goes from STARTING to FAILED.

Verify the sslConfig and kvstore settings on the instance.

$SPLUNK_HOME/bin/splunk btool server list --debug sslConfig
$SPLUNK_HOME/bin/splunk btool server list --debug kvstoreie:
/opt/splunk/bin/splunk btool server list --debug sslConfig
/opt/splunk/etc/system/local/server.conf   [sslConfig]
/opt/splunk/etc/system/default/server.conf caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem
/opt/splunk/etc/system/default/server.conf serverCert = $SPLUNK_HOME/etc/auth/server.pem
/opt/splunk/etc/system/local/server.conf   sslRootCAPath = etc/auth/mycerts/myCACertificate.pem
/opt/splunk/etc/system/local/server.conf   sslVersions = tls1.2,tls1.3
/opt/splunk/etc/system/default/server.conf sslVersionsForClient = tls1.2
/opt/splunk/etc/system/local/server.conf   supportSSLV3only = false

In this particular case there were some changes made to the sslRootCAPath, sslVersions, and supportSSLV3only settings of the sslConfig stanza while the settings under the kvstore stanza are all default values, and after disabling each one individually the only one that made a positive impact was the sslVersions setting.

Please refer to SSL/TLS Configuration regarding sslVersions setting.

Resolution

Change the sslVersions in the sslConfig stanza of $SPLUNK_HOME/etc/system/local/server.conf as per the following

[sslConfig]
#sslVersions = tls1.2,tls1.3OR [sslConfig] 
sslVersions = tls

After making the following change a restart of Splunk will be required

$SPLUNK_HOME/bin/splunk restart 

Use btool and confirm the setting is properly set up based on the documentation or it matches the current default.

$SPLUNK_HOME/bin/splunk btool server list --debug sslConfig/opt/splunk/etc/system/local/server.conf   [sslConfig]
/opt/splunk/etc/system/default/server.conf caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem
/opt/splunk/etc/system/default/server.conf serverCert = $SPLUNK_HOME/etc/auth/server.pem
/opt/splunk/etc/system/local/server.conf   sslRootCAPath = etc/auth/mycerts/myCACertificate.pem
/opt/splunk/etc/system/default/server.conf sslVersions = tls1.2
/opt/splunk/etc/system/default/server.conf sslVersionsForClient = tls1.2
/opt/splunk/etc/system/local/server.conf   supportSSLV3only = false

Related Links

SSL/TLS Configuration details
Identify a stale KV store member
Configuration file examples for configuring TLS certificates on receiving indexers

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release