KV store status failed to start after upgrade to Splunk 9.4.x and 10.0.x.

Applicability

Description

When upgrading Splunk to version 9.4 or 10.0, it is common to encounter errors regarding the KV store in failed status; this article will explain a common issue with this new version and the default certificates configured in cloud under the app: 100_<stackname>_splunkcloud. 

The error is below mentioned :

In handler 'kvstoreupgrade': KVStore not in ready state. Upgrade cannot proceed. Current Status :: failedFailed to check current FCV value prior to KV Store upgrade. Error= An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No
suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]. Topology type: ReplicaSetNoPrimary .Please check KVStore Status

Investigation

When upgrading to Splunk 9.4 or 10.0, it's crucial to address Key-Value (KV) store compatibility and prerequisites to prevent future issues. Splunk Enterprise versions 9.4 and higher require KV store server version 7.0 for continued support and security. If the KV store is at version 4.2.x, it will automatically upgrade to 7.0 during the Splunk 9.4 upgrade. However, direct upgrades from Splunk versions before 9.3.x to 9.4 can lead to critical KV store issues.

Pre-Upgrade and Compatibility Checks

• KV Store Version: Ensure the KV store is running version 4.2 or higher before upgrading to Splunk 9.4.
• Prerequisites: Verify that the deployment meets all prerequisites for KV store server version 7.0. These include:
  • Supported Architecture: Intel x86_64 (Sandy Bridge or higher Core processor with SSE4.2, AVX, and AES-NI) or AMD x86_64 (Bulldozer or higher processor with AVX).
  • AVX Support: Confirm that the system's CPU settings have Advanced Vector Extensions (AVX) support enabled.
  • Disk Space: Ensure more than 50% of the disk space is available on the partition where KV store is present.
  • Custom Certificates/IPv6: If using custom certificates or IPv6, review the documentation on "Preparing custom certificates for use with KV store" for compatibility with KV store server version 7.0. Note that custom certificates cannot be used while migrating to MongoDB 7.0 in some cases, and IPv6 might also cause issues during migration.
• Backup: Always back up all KV store databases before upgrading Splunk Enterprise.
• App Compatibility: Not all Splunk apps and add-ons are compatible with Splunk Enterprise 9.4. Check the Splunk products version compatibility matrix and Splunkbase to confirm compatibility before upgrading.

Checking KV Store Status

To check the health and version of the KV store, use the following command: 

splunk show kvstore-status --verbose

Example output :

If the KV store is experiencing issues, output similar to this will be observed:

This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : xxxxxxxxxxxxxxxxxxxx
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

This error indicates that the KV store process failed to start or connect, often due to underlying issues like certificate problems or an unhealthy MongoDB instance. 

Checking MongoDB Status

The KV store relies on MongoDB. It's important to verify its version and ensure it's running correctly.

To check the MongoDB version:

splunk cmd mongod --version

Sample Output of a Healthy MongoDB Service: 

db version v7.0.14
Build Info: {
"version": "7.0.14",
"gitVersion": "ce59cfc6a3c5e5c067dca0d30697edd68d4f5188",
"openSSLVersion": "OpenSSL 1.0.2zk-fips 3 Sep 2024",
"modules": [
"enterprise"
],
"allocator": "tcmalloc",
"environment": {
"distmod": "rhel70",
"distarch": "x86_64",
"target_arch": "x86_64"

Checking Certificate Expiry

Expired certificates are a common cause of KV store failures. Check the expiration date of  server.pem certificate with this command:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/<.pem file>

If the output date is before the current date, the certificate is expired.

Checking Certificate Validity

Old but not expired certificates can cause KV Store to fail. They might not be expired, but if they were created before 9.4.x they will not meet KV Store requirements. Renewing them is the way to fix this. 

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>

Common Log Errors

Troubleshooting often involves reviewing splunkd.log and mongod.log for specific error messages.

splunkd.log Example:

WARN MongoClient [1234 KVStoreUpgradeStartupThread] - Disabling TLS hostname validation for localhost
ERROR KVStorageProvider [1234 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

mongod.log Example: 

SSL peer certificate validation failed: self signed certificate in certificate chain

These errors frequently point to issues with SSL certificate validation or connectivity to the MongoDB instance. Specifically, MongoDB requires that SSL certificates include both serverAuth and clientAuth within the extendedKeyUsage field.

Root Cause: Certificate Override

A common issue identified during investigation is that an application, such as 100_<stackname>_splunkcloud, might override the sslRootCAPath attribute in server.conf. This prevents the KV store from reading the correct sslCAFile, leading to certificate validation failures.

More specifically, the problematic line can be found in:

$SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/server.conf
sslRootCAPath = $SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_cacert.pem

This configuration can cause the KV store to incorrectly use certificates, especially if the Splunk Cloud Credentials Package contained incorrect settings for [sslConfig] in server.conf in older releases.

Splunk 10.0 Migration Note

When migrating to Splunk 10.0 from versions 9.2 or 9.3,  errors related to the MongoDB version might be encountered. It's important to note that generally, manually updating MongoDB version should not be done. The resolution steps for the certificate issues will often automatically update the MongoDB version as well.

Windows-Specific Troubleshooting

If the users are using a Windows or Windows Server environment, additional steps might be required to be performed. After creating combined certificates,  delete the existing SSL certificates from the Microsoft Management Console (MMC). The specific steps for this process would typically be detailed in the resolution section of the troubleshooting guide.

Resolution

UPDATE: An older release of the Splunk Cloud Credentials Package contained incorrect certificate settings for [sslConfig] in server.conf. These resulted in the certificates being used globally causing the KV store to incorrectly use them. The KV store will not work with these certificates. The latest Splunk Cloud Credentials Package has been updated to remove these server.conf entries and move them to the correct location in outputs.conf which are needed for securing the splunktcp communications between the OnPrem HF to Cloud. This allows the KV store to use either the default Splunk certificates or custom certificates. The details below should no longer be needed but are being left for reference purposes.

The long term fix, is waiting until a new fix version of the app splunkclouduf.spl is released due to the big issue is having a cacert that overrides entirely the Splunk configuration.

In the meantime, the next workaround can be applied to get the KV store to ready state:

1. Combine CA Certificates

Files to Combine:

$SPLUNK_HOME/etc/auth/cacert.pem
$SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_cacert.pem

- Combine the two certificate files.

Save the combined certificate as
/home/splunker/splunk/etc/auth/test/combined_cacert.pem

2. Update server.conf

File to Edit: $SPLUNK_HOME/etc/system/local/server.conf

Under the [sslConfig] stanza

- Configuration Change: Add or update the following attribute:

sslRootCAPath = /home/splunker/splunk/etc/auth/test/combined_cacert.pem

3. Adjust KVStore SSL Settings

- Copy the sslRootCAPath config from $SPLUNK_HOME/etc/apps/100_<stack_name>/default/server.conf.

- Removal: Remove the sslRootCAPath entry from the global configuration file located at: $SPLUNK_HOME/etc/apps/100_<stack_name>/default/server.conf

- Copy the [tcpout:splunkcloud_…] stanza from $SPLUNK_HOME/etc/apps/100_<stack_name>/default/outputs.conf and paste it in $SPLUNK_HOME/etc/apps/100_<stack_name>/local/outputs.conf. 

Next, add the sslRootCAPath config copied in the previous steps below this stanza.

Below is an example of how it should be: $SPLUNK_HOME/etc/apps/100_<stack_name>/local/outputs.conf

[tcpout:splunkcloud_…]
sslRootCAPath = <root_cert>

4. Removing certificates from Windows MMC (optional):

5. Final Checks

Remove any comments or outdated settings related to sslRootCAPath from previous configurations.

6. Restart

Save all changes and restart Splunk to ensure that the KVStore starts as expected. 

7 . Additionally, the suggested steps helped to resolve the issue :

- Reinstall the latest credential package on the affected server :

      1. Log in to the affected Splunk server. 

      2. Navigate to :
       

$SPLUNK_HOME/etc/apps/

 
      3. Identify the Splunk Cloud credential app:  100_<stackname>_splunkcloud

      4. Stop Splunk before making changes:
                

$SPLUNK_HOME/bin/splunk stop

      5. Remove or move the existing credential app (backup recommended):  

mv 100_<stackname>_splunkcloud 100_<stackname>_splunkcloud_backup

      6. Install the latest credential package provided for the Splunk Cloud stack. 

- If installing via CLI: 
   

  $SPLUNK_HOME/bin/splunk install app <package_name>.spl -auth admin:<password>

- After the reinstalling deleting the mongod lock file .

1 . Ensure Splunk is stopped:

$SPLUNK_HOME/bin/splunk stop

2 .Navigate to the KV Store directory:

$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/

3. Locate the lock file:

mongod.lock

4. Delete the lock file:

rm -f mongod.lock

5.Only delete the lock file when Splunk is fully stopped.

-Start Splunk and Validate KV Store

Start Splunk:

$SPLUNK_HOME/bin/splunk start

Verify KV Store status:

$SPLUNK_HOME/bin/splunk show kvstore-status --verbose

Confirm the status shows:

status : ready
Expected Outcome

Related Links

KV store documentation 

Backup and restore details for KV store

Prerequisities of KV store 

Preparing custom certificates for use with KV store | Splunk Enterprise (last updated 2025-11-07T20:25:08.258Z)