KV Store stopped working after upgrade to v9.x

Applicability

Splunk Enterprise

Description

Splunk Enterprise versions 9.0 and higher require the WiredTiger storage engine to function and therefore, it can not start/run if the storage engine is set incorrectly.
Migrating the storage engine before or during upgrade to Splunk Enterprise 9.0 or higher is a best practice. However, if the KV store migration failed during/after an upgrade due to any unfortunate reason this article should help with the investigation and fix.

Investigation

• Review the below error in the kvstore-status output:

featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [connection closed calling ismaster on '127.0.0.1:8191']

• migration.log shows:

[App Key Value Store migration] Starting migrate-kvstore.
Created version file path=C:\Program Files\Splunk\var\run\splunk\kvstore_upgrade\versionFile36
[App Key Value Store migration] NOTE: Lock file is not empty. If migration fails, stop the service (if it is running), delete the lock file, and run the migration again.
Started standalone KVStore update, start_time="2023-04-19 23:54:01".
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600seconds.
[App Key Value Store migration] Migration is not required.
Created version file path=C:\Program Files\Splunk\var\run\splunk\kvstore_upgrade\versionFile40
Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
..Failed to migrate to storage engine wiredTiger, reason=
[App Key Value Store migration] Starting migrate-kvstore.
[App Key Value Store migration] Storage Engine hasn't been migrated to wireTiger. Cannot upgrade to service(42)
[DFS] Performing migration.
[DFS] Finished migration.
[Peer-apps] Performing migration.
[Peer-apps] Finished migration.

• In mongod.log review the "mmapv1" being considered:

2023-04-19T14:54:10.723Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { allowConnectionsWithoutCertificates: true, allowInvalidCertificates: true, allowInvalidHostnames: true, certificateSelector: "subject=SplunkServerDefaultCert", disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200 }, security: { javascriptEnabled: false, keyFile: "D:\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "D:\var\lib\splunk\kvstore\mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }

also:

2023-04-19T14:54:10.725Z I STORAGE [initandlisten] ** WARNING: Support for MMAPV1 storage engine has been deprecated and will be
2023-04-19T14:54:10.725Z I STORAGE [initandlisten] ** removed in version 4.2. Please plan to migrate to the wiredTiger
2023-04-19T14:54:10.725Z I STORAGE [initandlisten] ** storage engine.
2023-04-19T14:54:10.725Z I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/deprecated-mmapv1

splunkd.log shows below warnings that state mongod exited abruptly:

04-19-2023 23:55:05.840 +0900 ERROR MongodRunner [17252 MainThread] - MongtoolRunner exited with nonzero status=1
04-19-2023 23:57:02.294 +0900 ERROR MongodRunner [14776 MongodLogThread] - mongod exited abnormally (exit code 1, status: exited with code 1) - look at mongod.log to investigate.

Noted the storageEngine value under [kvstore] stanza in $SPLUNK_HOME\etc\system\local\server.conf 

[kvstore]
storageEngine = mmapv1

Noted the storageEngine value under [kvstore] stanza in $SPLUNK_HOME\etc\system\default\server.conf

[kvstore]
storageEngine = wiredTiger

Resolution

1. Remove the following line from the kvstore stanza in  $SPLUNK_HOME\etc\system\local\server.conf

[kvstore]
storageEngine = mmapv1 << remove this line

2. clean up the kvstore and restart 

splunk clean kvstore --local

splunk restart

3. note and ensure that the correct storageEngine value is set with server version 4.2

splunk show kvstore-status

Extra Note:

If the backups were taken previously while on mmapv1 or if the kvstore data is present in mmapv1 format the the data can be restored or converted from mmapv1 files to wiredTiger. (this is to be only considered if the kvstore data is present in mmapv1 format while the current storage is wiredTiger)

1. Check mmapv1 format data exists under $SPLUNK_HOME/var/lib/splunk/kvstore/old_db and find the directories with timestamp matches the splunk upgrade time.

2. Stop splunk

3. Backup existing mongo directory:

mv $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/* $SPLUNK_HOME/var/lib/splunk/kvstore/mongo_backup

4. Copy the mmapv1 format data to mongo directory:

cp -r $SPLUNK_HOME/var/lib/splunk/kvstore/old_db/<splunk_upgrade_time>/* $SPLUNK_HOME/var/lib/splunk/kvstore/mongo

5. Run wiredtiger migration to new dump directory:

splunk migrate kvstore-storage-engine --target-engine wiredTiger --output asolute_path/to/new_dump_dir

6. Run migrate kvstore

splunk migrate migrate-kvstore

Related Links

About upgrading to 9.0 READ THIS FIRST
Migrate the KV store storage engine
KV store troubleshooting tools

 

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release