KVStore fails to start due to version incompatibility

Applicability

Description

This article is to document the behaviour for Mongod/KVStore startup behaviour for different mongod version binaries.

If something affects your kvstore (not part of this article) then you may find logs indicating the kvstore is not starting due to version incompatibility. For example, in mongod.log you would see:

INITSYNC [replication-0] Initial sync attempt failed -- attempts left: 0 cause: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6" }. See https://docs.mongodb.com/manual/release-notes/4.2-compatibility/#feature-compatibility.

For example, Splunk 9.0 kvstore requires Mongod/ServerVersion 4.2 by default along with WiredTiger storage engine (not the older mmapv1).

The previous issues with migration to 8.x to 9.x has been from cluster members running 3.6 and 4.2 binaries which are not feature compatible. This is also the case with 9.4 migrations where the KVStore mongod version is upgraded to 7.0

Ref Links:

MongoDB WiredTiger Manual
Migrate the KV store storage engine

Investigation

If something affects your kvstore (not part of this article) then you may find logs indicating the kvstore is not starting due to version incompatibility.

1) Mongod binaries are located in your splunk/bin directory. i.e. /opt/splunk/bin

a) The latest current binary will be named mongod.
b) Previous mongod binaries are named: mongod-3.6, mongod-4.0. etc.

2) The startup behaviour for mongod/kvstore can be affected via a versionFile.
versionFiles will be located in:

/opt/splunk/var/run/splunk/kvstore_upgrade/

3) By default, Splunk will start the mongod binary with a versionFile created. Otherwise Splunk will iterate though to the latest version if migration is required if standalone, or manual upgrade is required if a cluster.

If you do see this error or an error referencing similar, check the below directory for a versionFile.

versionFile36 = starts mongod 3.6 binary
versionFile40 = starts mongod 4.0 binary
versionFile42 = starts mongod 4.2 binary

If there is a current kvstore created in for example, 4.2 but a versionFile36 file exists causing mongod 3.6 to launch, you will have a failed kv start.

Standalone:

If there is no current kvstore created that mongod finds on start, Splunk migrates the kvstore to the latest version defined in the code. That is unless a versonFile defining an older version is present, in which case, it will start that version of the mongod binary if present.

Clustered:
A manual upgrade is required to be initiated. However, if the kvstore has a versionFile older than the migrated version, it will attempt to start that and cause sync issues.

Resolution

1) Removing an older version of the versionFile if present.

2) You may need to backup, clean and sync if the kvstore is stale.

Resync the KV store

Back up and restore KV store

Troubleshoot KVstore

KV store server version (9.4 server version 7.0)

3) Please note, as per the documentation, you can remove the older mongod binaries if desired as this will prevent them from running the previous versions:

Migrate the KV store storage engine
"After completing your migration to WiredTiger and server version 4.2, you can optionally remove the unsupported binary files for previous versions of MongoDB."

Related Links

MongoDB WiredTiger Manual

Resync the KV store

Back up and restore KV store

Troubleshoot KVstore

KV store server version (9.4 server version 7.0)

Troubleshoot KVstore

KV store server version (9.4 server version 7.0)