KVStore Fails to Start After Upgrading to Splunk Enterprise 9.4.x

Available Languages

Updated:September 30, 2026
Document ID:SPLK051092

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Cloud (HF) Splunk Enterprise

Description

This article outlines a scenario where the KVStore fails to start after upgrading Splunk Enterprise to version 9.4.x.

The issue often occurs when custom certificates are used instead of default certificates, KVStore may remain stuck on an unsupported version (e.g., 4.2), while Splunk Enterprise 9.4.x requires KVStore version 7.0 or higher.


Investigation

Follow the below troubleshooting steps:

  1. KVStore Status Check
    The KVStore status was checked using the command, validate the status of kvstore and serverVersion.
    ./splunk show kvstore-status --verbose

     

  2. Validate if the below logs are observed in splund.log and mongod.log.
    splunkd.log:
    [KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

    mongod.log    :
    2025-02-27T10:31:55.111Z E STORAGE [initandlisten] WiredTiger error (-31802) [1740652315:111379][47501:0x7f8dbaa4ab40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1740652315:111379][47501:0x7f8dbaa4ab40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error 2025-02-27T10:31:55.115Z W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version. 
2025-02-27T10:31:55.115Z F STORAGE [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error 
2025-02-27T10:31:55.115Z F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 928 
2025-02-27T10:31:55.115Z F - [initandlisten] \n\n***aborting after fassert()


  3. As the issue with compatibility of the mojngo version with Splunk version, validate if all the pre-requisites for mongo 7.0 are met. 
    A. Minimum 50% of disk space on partition were kvstore is present.
B. Use of Splunk default certs, custom certificates cannot be used while migrating to mongo 7.0
C. ipv6 cannot be used while migration to mongo 7.0.
D. The hardware should have AVX flag enabled.

     

  4. To fix the issue with compatibility follow the below steps:
    A. Created a backup of the mongo directory:
cp -r /opt/splunk/var/lib/splunk/kvstore/mongo /home/
B. Stop Splunk on the server
C. Performed a KVStore cleanup:
/opt/splunk/bin/splunk clean kvstore --localD. Start Splunk on the server

    WARNING: running clean kvstore --local will erase the whole kvstore database. Do this ONLY if you cannot recover the database due to corruption or a stale CLUSTERED KV store member!!

  5. If the above does not resolve the issue, configure server.conf to explicitly point to Splunk default certificates.
    [sslConfig] 
serverCert = $SPLUNK_HOME/etc/auth/server.pem 
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslPassword = password

    Restart Splunk and now validate the status of kvstore. 
  6. Given Splunk is configured to use default certificates, kvstore will work but due to this now Splunk default certificates will take precedence and the SSL certificates under cloud credential package will not be taken in memory and because of this data flow to cloud will break.


Resolution

UPDATE: An older release of the Splunk Cloud Credentials Package contained incorrect certificate settings for [sslConfig] in server.conf. These resulted in the certificates being used globally causing the KVstore to incorrectly use them. The KVstore will not work with these certificates. The latest Splunk Cloud Credentials Package has been updated to remove these server.conf entries and move them to the correct location in outputs.conf which are needed for securing the splunktcp communications between the OnPrem HF to Cloud. This allows the KVstore to use either the default Splunk certificates or your own custom certificates. The details below should no longer be needed but are being left for reference purposes.


The resolution involves using certificates from forwarder credential package, reconfiguring SSL settings, and upgrading the KVStore to version 7.0 if it remains stuck on an older version. Follow these steps:

Step 1: Configuring certificates from forwarder credentrial package

  1. Remove the custom certificates previously configured for KVStore.

  2. Navigate to $SPLUNK_HOME/etc/apps/100_cloudstack_splunkcloud/local/output.conf and copy the sslPassword.

  3. Decrypt the sslPassword using:
    ./splunk show-decrypted --value '<encrypted_value>'
  4. Note the plaintext password for the next step.

  5. Edit $SPLUNK_HOME/etc/system/local/server.conf and add the following under [sslConfig]: 
    [sslConfig] 
serverCert = $SPLUNK_HOME/etc/apps/100_cloudstack_splunkcloud/default/cloudstack_server.pem 
sslRootCAPath = $SPLUNK_HOME/etc/apps/100_cloudstack_splunkcloud/default/cloudstack_cacert.pem 
sslPassword = <decrypted_password_from_step_3>
  6. Restart the Splunk server.

  7. Verify KVStore status:
    ./splunk show kvstore-status
    Expected result: KVStore status shows "Ready."

Step 2: Address KVStore Version Below 7.0
If the KVStore remains in a "Failed" state after the above steps due to being stuck on a version lower than 7.0, proceed as follows:

  1. Verify KVStore Version
    • Check the KVStore version in the logs:
      cat /opt/splunk/var/log/splunk/mongod.log | grep -i "version"
  2. Remove Mongo Backup
    • Navigate to the KVStore directory:
      cd /opt/splunk/var/lib/splunk/kvstore/
    • Remove the mongo_backup directory:
      rm -rf mongo_backup


  3. Create a Backup
    • Back up the mongo directory:
      cp -r /opt/splunk/var/lib/splunk/kvstore/mongo /home/

  4. Stop Splunk Service

  5. Clean KVStore
    • Perform a local KVStore cleanup:
      /opt/splunk/bin/splunk clean kvstore --local
      WARNING: running clean kvstore --local will erase the whole kvstore database. Do this ONLY if you cannot recover the database due to corruption or a stale CLUSTERED KV store member!!
  6. Start Splunk Service

  7. Verify KVStore Status
    • Check the status:
      ./splunk show kvstore-status --verbose
      Expected result: KVStore status shows "Ready."

  8. Upgrade KVStore to Version 7.0
    • Upgrade the KVStore:
      /opt/splunk/bin/splunk start-standalone-upgrade kvstore -version 7.0
    • Monitor the upgrade process in the logs.

9. Once completed, verify the status again:

./splunk show kvstore-status --verbose
  • Expected result: KVStore is running on version 7.0 and in a "Ready" state.

Upgrade the KV store server version

KV store troubleshooting tools

About the app key value store

server.conf

Revision History

Revision Publish Date Comments
1.0
30-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products

Related Articles