KVStore fails to start after 9.1.x upgrade

Available Languages

Updated:September 29, 2026
Document ID:SPLK046966

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk enterprise version 9.1+

Description

This article describes a specific situation where Splunk KVStore fails to start after Splunk version upgrade, by following the steps.

  1. SSL cert pem file was deleted, does not exist in the file system.
  2. Splunk enterrise version upgrade happens
  3. Splunk starts which will, in turn, renew the default server cert that was deleted in step #1.
  4. KVStore mongod fails to start and crashes

Investigation

To confirm this hits a known issue 
  • Check there's no mongod running by entering a linux command, "ps -ef |grep mongo"
  • Check the migration failed in migration log, i.e) migration914.log for version 9.1.4.  
    <SNIP>
Started standalone KVStore update, start_time="2024-06-03 23:53:35".
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600seconds.
WARN: [App Key Value Store migration] Service(40) terminated before the service availability check could complete. Exit code 14, waited for 1 seconds.
App Key Value Store migration failed, check the migration log for details. After you have addressed the cause of the service failure, run the migration again, otherwise App Key Value Store won’t function.
<SNIP>
  • Find there's a file "VersionFile36" just created, such as $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/VersionFile36.
  • Find the error messages in mongod.log, due to the file, VersionFile36, mongod starts the db version v3.6.17-linux-splunk-v4 which is not compatible with WiredTiger. Below is the mongod log messages until it crashes. 
    2024-06-03T23:55:04.576Z I COMMAND [conn16] {msg: "Profiler settings changed", from:{ level: 0, slowms: 1000, sampleRate: 1.0 }, to:{ level: 0, slowms: 1000, sampleRate: 1.0 }}
2024-06-04T03:53:55.264Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] MongoDB starting : pid=1696 port=9491 dbpath=/home/splunk/splunk/var/lib/splunk/kvstore/mongo 64-bit host=xxx.xxx.xxx.xxx
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] db version v3.6.17-linux-splunk-v4
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] git version: 226949cc252af265483afbf859b446590b09b098
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2zj-fips 30 Jan 2024
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] allocator: tcmalloc
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] modules: none
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] build environment:
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] distarch: x86_64
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] target_arch: x86_64
2024-06-04T03:53:55.287Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 9491, ssl: { PEMKeyFile: "/home/splunk/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "7C618FDF-32BC-4D7C-832B-1B46079DD9A1" }, security: { javascriptEnabled: false, keyFile: "/home/splunk/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "/home/splunk/splunk/var/lib/splunk/kvstore/mongo", engine: "wiredTiger", wiredTiger: { engineConfig: { cacheSizeGB: 1.05 } } }, systemLog: { timeStampFormat: "iso8601-utc" } }
2024-06-04T03:53:55.287Z W - [initandlisten] Detected unclean shutdown - /home/splunk/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty.
2024-06-04T03:53:55.290Z W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2024-06-04T03:53:55.290Z I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1075M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-06-04T03:53:55.899Z E STORAGE [initandlisten] WiredTiger error (-31802) [1717473235:899878][1696:0x7f1165624a80], connection: __log_open_verify, 1028: Version incompatibility detected: unsupported WiredTiger file version: this build requires a maximum version of 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error Raw: [1717473235:899878][1696:0x7f1165624a80], connection: __log_open_verify, 1028: Version incompatibility detected: unsupported WiredTiger file version: this build requires a maximum version of 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error
2024-06-04T03:53:55.899Z E STORAGE [initandlisten] An unsupported journal format detected - If you are trying to rollback from version 4.0 to 3.6, please re-start a 4.0 binary and cleanly shut it down so that the journal format will be downgraded.
2024-06-04T03:53:55.904Z E - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 488
2024-06-04T03:53:55.904Z I STORAGE [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2024-06-04T03:53:55.904Z F - [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272 2024-06-04T03:53:55.904Z F - [initandlisten] ***aborting after invariant() failure
2024-06-04T03:53:55.917Z F - [initandlisten] Got signal: 6 (Aborted). 0x55a646fab181 0x55a646faa399 0x55a646faa87d 0x7f11646aa630 0x7f1164303387 0x7f1164304a78 0x55a645664440 0x55a645911e88 0x55a646e4fb91 0x55a646e4bc77 0x55a6456dbaac 0x55a646fa6535 0x55a645665623 0x55a6456e5460 0x55a6456e348a 0x55a645666269 0x7f11642ef555 0x55a6456ca7d5
 
 
 

Resolution

Below are confirmed as workarounds and  bring up the mongod process successfully.

  •  renaming

    $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/VersionFile36
    to
    $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/VersionFile42 

  • Or, delete the incorrect version file, VersionFile36 

Then, restart Splunk.

 


https://splunk.atlassian.net/browse/SPL-256889

Revision History

Revision Publish Date Comments
1.0
29-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products

Related Articles