Failed to migrate KV Store storage engine

Available Languages

Updated:September 29, 2026
Document ID:SPLK045789

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise

Description

 After upgrading Splunk Enterprise to 9.0, a message in the console indicates a failed KV Store migration: 
"Failed to migrate KV Store storage engine to WiredTiger. This prevented running the latest version of KV Store. Migrate KV Store storage engine to WiredTiger and then upgrade KV Store. The Mmapv1 engine is deprecated"

 

Investigation

1. Tried a manual migration on the command line, but the following command returned the following output:

sudo /opt/splunk/bin/splunk migrate kvstore-storage-engine --enable-compression --target-engine wiredTiger

Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:

ERROR: Failed to migrate to storage engine wiredTiger, reason=KVStore service will not start because kvstore process terminated

 

2. In order to resolve this, the KV Store storage engine needs to be migrated to WiredTiger

Single-instance deployments of Splunk Enterprise 9.0 and higher are automatically migrated to the WiredTiger storage engine and the latest version of MongoDB, server version 4.2, during the upgrade.

1. Complete any prompts during the Splunk Enterprise upgrade.

2. Verify that the latest version of the storage engine is present after the upgrade with the following command:
splunk show kvstore-status --verbose

3. Verify that the serverVersion and storageEngine fields indicate the latest versions:

serverVersion : 4.2.17

[...]

storageEngine : wiredTiger

Migrate the KV store in a clustered deployment :

The migration must be run manually during the upgrade to Splunk Enterprise 9.0 and higher if it has not been done prior to it.

Prepare the deployment, migrate to WiredTiger, and once the migration is completed, upgrade the MongoDB version to 4.2.

 

3. Server version is 4.0.24, which is an interim version and should be upgraded to 4.2. It is probable that the upgrade fails due to using an unsupported OS version.

 

4. Note that it is needed to have MongoDB version 3.0 or greater in order to use the WiredTiger storage engine. If a previous version is being used, it must be upgraded first.

 

5. From the mongod.log file 
I NETWORK [conn278965] received client metadata from 127.0.0.1:52344 conn278965: { driver: { name: "mongoc", version: "1.16.2" }, os: { type: "Linux", name: "Ubuntu", version: "16.04", architecture: "x86_64" }, platform: "cfg=0x00001620c9 posix=200112 stdc=201710 CC=GCC 9.1.0 CFLAGS="-g -fstack-protector-strong -static-libgcc -L/opt/splunk-home/lib/static-libstdc" LDFLA..." }
I ACCESS [conn278965] Successfully authenticated as principal __system on local from client 127.0.0.1:52344
2022-07-25T12:26:55.282Z I NETWORK [conn278965] end connection 127.0.0.1:52344 (14 connections now open)

 

6. Please run the ./splunk clean kvstore -local command and verify if the issue persists after restarting the Splunk service.

 

If the upgrade is not happening, run the following command to check if the configurations are preventing the storageEngine upgrade:

splunk btool server list --debug | grep storageEngine

Found the following stanza in server.conf, removing the line allowed the upgrade to be completed:

[kvstore]
storageEngine = mmapv1

Resolution

You can use the below command to migrate KV-store:

./splunk migrate migrate-kvstore

 If it is a stand alone server, and the manual upgrade does not work and kvstore is still in failed state, you can try the following:

  • Backup KVStore:
./splunk backup kvstore

This will save the back up in $SPLUNK_DB/kvstorebackup directory.

 

  • Then stop splunk and clean kvstore.
splunk stop
splunk clean kvstore --local

 

  • Before starting Splunk, ensure that only versionFile42 exists in the  $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/ directory.
  • Now start Splunk:
    splunk start

     

  • Restore from backup:
    splunk restore kvstore



Reference - MigrateKVstore

Revision History

Revision Publish Date Comments
1.0
29-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products

Related Articles