KVStore fails to start after 9.3.x upgrade

Applicability

Description

This article describes a specific situation where Splunk KVStore fails to start after Splunk version upgrade  to 9.3  and the $SPLUNK_HOME/var/log/splunk.mongod.log shows the following error

2025-06-10T15:58:04.772Z E REPL   [replexec-0] Locally stored replica set configuration is invalid; See http://www.mongodb.org/dochub/core/recover-replica-set-from-invalid-config
for information on how to recover from this. Got "BadValue: Support for replication protocol version 0 was removed in MongoDB 4.0. Downgrade to MongoDB version 3.6 and upgrade your protocol version to 1 before upgrading your MongoDB version" while validating { id: "21FBDB99-7307-41EB-AB8C-F16BB2DED2D1", version: 1, protocolVersion: 0, writeConcernMajorityJournalDefault: true, members: [ { id: 0, host: "127.0.0.1:8191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "21FBDB99-7307-41EB-AB8C-F16BB2DED2D1", all: "all" }, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }

Investigation

This is a standalone Search Head and the kvstore status is as not in 'ready' state as  shown below

 

In addition the mongod process is not running by entering a linux command, "ps -ef |grep mongo". 
The $SPLUNK_HOME/var/log/splubnk/mongod.log  has the below error

Resolution

This was a standalone Search Head and following steps were followed to resolve the kvstore issue.

1) Stop Splunk Service
2) rename $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/versionFile42 TO $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/versionFile42_back
3) Create file $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/versionFile36
4) Run migration by running $SPLUNK_HOME/bin/splunk migrate migrate-kvstore  , this converted mongod from version 3.6 to 4.0
5) Run migration for  wiretiger using $SPLUNK_HOME/bin/splunk migrate kvstore-storage-engine --target-engine wiredTiger
6) In the last run migrate mongod to mongod 4.2 using $SPLUNK_HOME/bin/splunk migrate migrate-kvstore
7) restarted the splunk and kvstore started fine with Mongod version 4.2.17.