Kvstore Failed To Start When Unclean Shutdown Occurs

Applicability

Description

Failed to start Kvstore after upgrade or when unclean shutdown occurs

Investigation

Check mongod.log, splunkd.log and mongod_upgrade.log, if an unclean shutdown is identified. Then Kvstore may fail to start. This can occur during Splunk upgrade or manual Kvstore upgrade or normal operations.

For the Splunk 9.4.0 upgrade, the kvstore/mongod upgrade sequence is: 4.2 > 4.4 >5.0 > 6.0 > 7.0. This process takes some time to complete. If there is any interruption during this process. The Kvstore may be impacted causing it to fail to start.

For example, when kvstore/mongod upgrades to 5.0 then going to continue the upgrade to 6.0 then 7.0. See below logs:

2025-01-19T23:37:27.400Z INFO [mongod_upgrade] All upgraded to 4.4, proceeding.2025-01-19T23:38:24.945Z INFO [mongod_upgrade] All upgraded to 5.0, proceeding.{"t":{"$date":"2025-01-19T23:38:24.950Z"},"s":"I", "c":"REPL", "id":20459, "ctx":"conn20","msg":"Setting featureCompatibilityVersion","attr":{"currentVersion":"upgrading from 4.4 to 5.0","newVersion":"5.0"}}2025-01-19T23:38:24.946Z INFO [mongod_upgrade] Setting new FCV Version: 5.0

If at this time, the Splunkd service restarts. This could cause an unclean shutdown. See below logs as an example:

01-19-2025 23:38:03.640 +0000 INFO AuditLogger - Audit:[timestamp=01-19-2025 23:38:03.641, user=n/a, action=splunkShuttingDown, info=n/a]{"t":{"$date":"2025-01-19T23:38:24.934Z"},"s":"I", "c":"-", "id":20883, "ctx":"conn9","msg":"Interrupted operation as its client disconnected","attr":{"opId":40}}01-19-2025 23:38:30.909 +0000 INFO IndexProcessor [25908 indexerPipe] - request state change from=SHUTDOWN_IN_PROGRESS to=SHUTDOWN_COMPLETE01-19-2025 23:38:39.918 +0000 INFO ServerConfig [0 MainThread] - My GUID is AD8C0E9A-5C12-4C08-9120-0F685643209801-19-2025 23:38:39.966 +0000 INFO loader [5105 MainThread] - Splunkd starting (build 6b4ebe426ca6).
2025-01-19T23:38:58.675Z W STORAGE [initandlisten] Detected unclean shutdown - kvstore/mongo/mongod.lock is not empty.

Then it will cause FCV(featureCompatibilityVersion) version mismatch. See this log:

{"t":{"$date":"2025-01-20T02:21:04.466Z"},"s":"F", "c":"CONTROL", "id":20573, "ctx":"initandlisten","msg":"Wrong mongod version","attr":{"error":"UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"4.4\" }. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value '4.4'; expected '6.0' or '6.3' or '7.0'. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility.). If the current featureCompatibilityVersion is below 6.0, see the documentation on upgrading at https://docs.mongodb.com/master/release-notes/6.0/#upgrade-procedures."}}

In some scenario, when showing kvstore status finds the status "failed", it is best to check the mongod.log to understand the reason of the status. If kvstore is in the middle of upgrade and can not bring up temporarily. It needs to leave for the process to finish. Take Splunk 9.4.0 upgrade as an example, even small size of Kvstore/mongo directory can take at least 10 minutes and more. So for the environment with large mongo directory. It will take longer to convert the mongo data file format(mongod upgrade requirement).

Resolution

When this log is found:

2025-01-19T23:38:58.675Z W  STORAGE  [initandlisten] Detected unclean shutdown - /mnt/data3/splunkdb/kvstore/mongo/mongod.lock is not empty.

The file mongod.lock should be deleted and restart Splunk to try bring up kvstore.

When the latest and highest FCV version is identified in the mongod logs, like this:

 2025-01-19T23:38:24.945Z INFO  [mongod_upgrade] All upgraded to 5.0, proceeding.
 2025-01-19T23:38:24.946Z INFO  [mongod_upgrade] Setting new FCV Version: 5.0

It means Kvstore/mongod is upgraded to version 5.0. Then actions can be tried:

1. Touch versionFile50 under directory: $SPLUNK_HOME/var/run/splunk/kvstore_upgrade
2. Run command for members resync: $SPLUNK_HOME/bin/splunk resync kvstore -featureCompatibilityVersion 5.0

Then continue the kvstore migration process until the expected Kvstore/mongod version is migrated.

If this is still not restoring the Kvstore, then clean command should be considered:

$SPLUNK_HOME/bin/splunk clean kvstore --local 

 Perform backup and restore for the directory of $SPLUNK_DB/kvstore/mongo 

For cluster environment, one problematic Kvstore member can be resync'ed by the cluster captain node.

The best practice for upgrading Splunk and Kvstore should be to monitor the mongod.log and/or splunkd.log after the Splunk upgrade to understand the behaviour in the background. That hopefully gives more understanding what the process is up to and/or any issue occurring. Then more time should be waited for the correct process to finish or to address any issue accordingly if there is any.

More documents and articles can be found in the section "Related Links"

Related Links

Resync the KV store
Mongod failed to upgrade to 7.0 during Splunk migration from 9.x to 9.4
Troubleshoot KV Store data missing after running clean kvstore --local command