KV Store status is showing "starting"

Applicability

Splunk Enterprise 9.x upgraded from 8.x

Description

An environment with Search Head Cluster with 3 nodes. KV Store fails after migrating to KV Store WiredTiger with Splunk version 9 upgrade in Search Head Cluster. The KV Store status always shows "starting" when the following command is run on Search Heads

$SPLUNK_HOME/bin/splunk show kvstore-status

Investigation

Problem Analysis:
MongoD logs in $SPLUNK_HOME/var/log/splunk/mongod.log indicates MongoD storage engine is still in MMAPv1 version while KV Store (MongoD) had been upgraded to 4.2, and this results in KV Store not being able to start properly on Search Head Cluster members. For more information, please refer to "Storage Engine Page from MongoDB" in Related Links section below.

Related logs:

ERROR KVStoreBulletinBoardManager [20719 MainThread] - Detected KV Store storage engine Mmapv1. Migrate the KV Store storage engine to WiredTiger and then upgrade KV Store. The Mmapv1 engine is deprecated. See the Splunk platform documentation for more information.

ERROR SHCMasterKVStoreMigrationState [10084 SHPoolMasterKVStoreUpgradeThread] - Compatibility check failed for peer=https://1.2.3.4:8089. StorageEngine=mmapv1. Version=4.0.24-linux-splunk-v1. Failing KV Store upgrade version=4.2

How does Splunk upgrade work with KV Store upgrade?

Splunk Search Head clustering configures MongoD replica set with one MongoD node running on each Search Head Cluster member. Search Head Cluster upgrade refers to upgrading all members to new Splunk version. In Search Head Cluster, KV Store upgrade refers to upgrading all nodes to replica set from MMAP to WiredTiger.

MongoD provides mechanism to upgrade replica set to WiredTiger in rolling manner. A selected MongoD node is restarted with WiredTiger option and empty data directory. It then performs initial sync of data from primary replica set. This is repeated until all nodes are running WiredTiger. During replica set upgrade, MongoD client connects to other nodes of replica set to perform read and write operations.

Following ways to upgrade Search Head Cluster:

1. offline upgrade: stop all members and upgrade them all at the same time.
2. member-by-member upgrade: upgrade one member, make it captain, then upgrade remaining members one at a time.
3. rolling upgrade: Put member in manual detention, upgrade it (it is made captain as part of upgrade), repeat the same for all members.

Finalize the rolling upgrade by running the following CLI command on Search Head captain: 

$SPLUNK_HOME/bin/splunk upgrade-finalize shcluster-members

For reference, please see SHCrollingupgrade in Related Links section below.

Resolution

Search Head Cluster KV Store migration:

Enable feature flag by setting the following in $SPLUNK_HOME/etc/system/local/server.conf for all Search Head Cluster members

[kvstore]
storageEngineMigration = true

* If set to true, perform KV Store WiredTiger upgrade during Splunk upgrade.
* If set to false, no upgrade will be performed.
* Default: true

and then restart each Search Head Cluster member by running splunk restart command.

$SPLUNK_HOME/bin/splunk restart

Clean the KV Store cluster:

$SPLUNK_HOME/bin/splunk clean kvstore --cluster

Note** "--cluster" : This deletes only cluster information (Replica Set config, oplog etc.) - i.e. it removes local.* files from $SPLUNK_DB/kvstore/mongo folder
Used to prepare backup to be restored in a new cluster.

Verify the KV Store status:
Very useful to get information about KV Store cluster members, their status, oplog and up times.

$SPLUNK_HOME/bin/splunk show kvstore-status

Verify Search Head Cluster status:
To check the overall status of your search head cluster, run the following command on any member:

$SPLUNK_HOME/bin/splunk show shcluster-status -auth <username>:<password>

Related Links

Storage Engine Page from MongoDB
SHCrollingupgrade
TroubleshootKVstore
BackupKVstore
ResyncKVstore

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release