KV Store will not start after upgrading to Splunk Enterprise Version 9.4

Available Languages

Updated:October 1, 2026
Document ID:SPLK052626

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise 9.4.x

Description

The KV store reports "Specified /opt/splunk/var/lib/splunk/kvstorebackup archive is not found" after upgrading Splunk Enterprise from 9.3.x to 9.4.x.


Investigation

The following errors were observed.

mongod.log

"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: self signed certificate in certificate chain"}}
 
$SPLUNK_HOME/bin/splunk show kvstore status --verbose
Specified /opt/splunk/var/lib/splunk/kvstorebackup archive is not found
 
A custom certificate is being used, however only for forwarding. Splunk to Splunk communication is using the default Splunk certificate. A custom application had the following settings.
 
server.conf:
[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/apps/.../.../root.pem
 
outputs.conf:
[tcpout:app]
clientCert = $SPLUNK_HOME/etc/apps/.../.../server.pem
sslVerifyServerCert = true
 

Resolution

Version 9.4 included upgrading the KV store to version 7.0 or higher. This KV store version requires a Certificate Authority (CA) verification check. If this check fails, the KV store will not function properly. In this case, the CA verification fails because the KV store is attempting to use the Splunk default certificate in combination with the CA configured in the custom app. To resolve this:
 
  1. Create a CA certificate that contains both the Splunk CA and the application's CA.
    cat $SPLUNK_HOME/etc/auth/cacert.pem $SPLUNK_HOME/etc/apps/.../.../root.pem > combinedCA.pem
  2. Place the combined CA certificate in the same location as the application's CA.

  3. Modify the application's server.conf, changing the CA certificate name to match the new combined certificate name.
    [sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/apps/.../.../combinedCA.pem

Preparing custom certificates for use with KV store

 

Revision History

Revision Publish Date Comments
1.0
01-Oct-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products