KV store status failed after upgrade to Splunk 9.4.x and 10.0.x

Available Languages

Updated:September 29, 2026

Document ID:SPLK048482

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Cloud Splunk Enterprise Splunk DB Connect

Description

When upgrading Splunk to version 9.4 or 10.0 it is common to encountered errors regarding the KV store in failed status; this article will explain a common issue with this new version and the default certificates configured in cloud under the app: 100_<stackname>_splunkcloud.

The error in question is:

An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]

Investigation

When upgrading to Splunk 9.4 or 10.0, it's crucial to address Key-Value (KV) store compatibility and prerequisites to prevent future issues. Splunk Enterprise versions 9.4 and higher require KV store server version 7.0 for continued support and security. If your KV store is at version 4.2.x, it will automatically upgrade to 7.0 during the Splunk 9.4 upgrade. However, direct upgrades from Splunk versions before 9.3.x to 9.4 can lead to critical KV store issues.

Pre-Upgrade and Compatibility Checks

KV Store Version: Ensure your KV store is running version 4.2 or higher before upgrading to Splunk 9.4.
Prerequisites: Verify that your deployment meets all prerequisites for KV store server version 7.0. These include:
- Supported Architecture: Intel x86_64 (Sandy Bridge or higher Core processor with SSE4.2, AVX, and AES-NI) or AMD x86_64 (Bulldozer or higher processor with AVX).
- AVX Support: Confirm that your system's CPU settings have Advanced Vector Extensions (AVX) support enabled.
- Disk Space: Ensure more than 50% of your disk space is available on the partition where KV store is present.
- Custom Certificates/IPv6: If using custom certificates or IPv6, review the documentation on "Preparing custom certificates for use with KV store" for compatibility with KV store server version 7.0. Note that custom certificates cannot be used while migrating to MongoDB 7.0 in some cases, and IPv6 might also cause issues during migration.
Backup: Always back up all KV store databases before upgrading Splunk Enterprise.
App Compatibility: Not all Splunk apps and add-ons are compatible with Splunk Enterprise 9.4. Check the Splunk products version compatibility matrix and Splunkbase to confirm compatibility before upgrading.

Checking KV Store Status

To check the health and version of your KV store, use the following command:

$SPLUNK_HOME/bin/splunk show kvstore-status --verbose

Sample Error Output:

If the KV store is experiencing issues, you might see output similar to this:

This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : 1ABC12-D123-4A01-1AB2-B1CDEF1234
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

This error indicates that the KV store process failed to start or connect, often due to underlying issues like certificate problems or an unhealthy MongoDB instance.

Checking MongoDB Status

The KV store relies on MongoDB. It's important to verify its version and ensure it's running correctly.

To check the MongoDB version:

splunk cmd mongod --version

Sample Output of a Healthy MongoDB Service:

db version v7.0.14
Build Info: {
"version": "7.0.14",
"gitVersion": "ce59cfc6a3c5e5c067dca0d30697edd68d4f5188",
"openSSLVersion": "OpenSSL 1.0.2zk-fips 3 Sep 2024",
"modules": [
"enterprise"
],
"allocator": "tcmalloc",
"environment": {
"distmod": "rhel70",
"distarch": "x86_64",
"target_arch": "x86_64"

Checking Certificate Expiry

Expired certificates are a common cause of KV store failures. You can check the expiration date of your server.pem certificate with this command:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/<.pem files>

If the output date is before the current date, the certificate is expired.

Common Log Errors

Troubleshooting often involves reviewing splunkd.log and mongod.log for specific error messages.

splunkd.log Example:

WARN MongoClient [1234 KVStoreUpgradeStartupThread] - Disabling TLS hostname validation for localhost
ERROR KVStorageProvider [1234 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

mongod.log Example:

SSL peer certificate validation failed: self signed certificate in certificate chain

These errors frequently point to issues with SSL certificate validation or connectivity to the MongoDB instance. Specifically, MongoDB requires that SSL certificates include both serverAuth and clientAuth within the extendedKeyUsage field.

Root Cause: Certificate Override

A common issue identified during investigation is that an application, such as 100_<stackname>_splunkcloud, might override the sslRootCAPath attribute in server.conf. This prevents the KV store from reading the correct sslCAFile, leading to certificate validation failures.

More specifically, the problematic line can be found in:

$SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/server.conf
sslRootCAPath = $SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_cacert.pem

This configuration can cause the KV store to incorrectly use certificates, especially if the Splunk Cloud Credentials Package contained incorrect settings for [sslConfig] in server.conf in older releases.

Splunk 10.0 Migration Note

When migrating to Splunk 10.0 from versions 9.2 or 9.3, you might encounter errors related to the MongoDB version. It's important to note that you generally do not need to manually update the MongoDB version. The resolution steps for the certificate issues will often automatically update the MongoDB version as well.

Windows-Specific Troubleshooting

If you are using a Windows or Windows Server environment, you may need to perform additional steps. After creating combined certificates, you might need to delete existing SSL certificates from the Microsoft Management Console (MMC). The specific steps for this process would typically be detailed in the resolution section of the troubleshooting guide.

Resolution

UPDATE: An older release of the Splunk Cloud Credentials Package contained incorrect certificate settings for [sslConfig] in server.conf. These resulted in the certificates being used globally causing the KVstore to incorrectly use them. The KVstore will not work with these certificates. The latest Splunk Cloud Credentials Package has been updated to remove these server.conf entries and move them to the correct location in outputs.conf which are needed for securing the splunktcp communications between the OnPrem HF to Cloud. This allows the KVstore to use either the default Splunk certificates or your own custom certificates. The details below should no longer be needed but are being left for reference purposes.

The long term fix, is waiting until a new fix version of the app splunkclouduf.spl is released due to the big issue is having a cacert that overrides entirely the Splunk configuration.

In the meantime, the next workaround can be applied to get the KV store to ready state:

1. Combine CA Certificates

Files to Combine:

$SPLUNK_HOME/etc/auth/cacert.pem
$SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_cacert.pem

- Combine the two certificate files.

Save the combined certificate as
/home/splunker/splunk/etc/auth/test/combined_cacert.pem

2. Update server.conf

File to Edit: $SPLUNK_HOME/etc/system/local/server.conf

Under the [sslConfig] stanza

- Configuration Change: Add or update the following attribute:

sslRootCAPath = /home/splunker/splunk/etc/auth/test/combined_cacert.pem

3. Adjust KVStore SSL Settings

- Copy the sslRootCAPath config from $SPLUNK_HOME/etc/apps/100_<stack_name>/default/server.conf.

- Removal: Remove the sslRootCAPath entry from the global configuration file located at: $SPLUNK_HOME/etc/apps/100_<stack_name>/default/server.conf

- Copy the [tcpout:splunkcloud_…] stanza from $SPLUNK_HOME/etc/apps/100_<stack_name>/default/outputs.conf and paste it in $SPLUNK_HOME/etc/apps/100_<stack_name>/local/outputs.conf.

Next, add the sslRootCAPath config copied in the previous steps below this stanza.

Below is an example of how it should be: $SPLUNK_HOME/etc/apps/100_<stack_name>/local/outputs.conf

[tcpout:splunkcloud_…]
sslRootCAPath = <root_cert>

4. Removing certificates from Windows MMC (optional):

Click the Windows icon and search the below, and select the Application that shows up:
- ```
MMC
```
Click on the File button in the MMC, in the top left corner and in the drop down menu click on Add/Remove Snap-in.
In the left column, click on Certificates and select Add to move it to the right column.
Select OK to move on to the next steps.
Click on the Computer account from the Certificates Snap-in window to view certificates
Hit Next and click on Local computer on the next window.
To close the snap-in manager screen, click on Finish and then OK.
You will now see a list of certificates on your device in the left column of MMC.
The next step is to locate the certificate you wish to remove using the folder list on the left. Open the Third-Party Root Certification Authorities folder and click on Certificates.
Locate and right-click on the certificate you wish to remove.
- The certificates should be the old certificates which existed before the upgrade to v9.4
Click on Properties and then in the General tab, click on Disable all purposes for this certificate in the Certificate purposes section. Hit Apply and restart your server to complete the removal process.

5. Final Checks

Remove any comments or outdated settings related to sslRootCAPath from previous configurations.

6. Restart

Save all changes and restart Splunk to ensure that the KVStore starts as expected.