Mongo migration failure post upgrade to Splunk Enterprise 9.4.0.

Applicability

Description

Upgrading Splunk to 9.4.0 should also automatically upgrade mongo version to 9.4.0 but the mongo migration to 7.0 fails and KVstore gets either in failed status or gets stuck at starting.

Investigation

1. Validate the status of KVstore with the below command.

./splunk show kvstore-status --verbose

A. Check the status of KVstore and also the server Version.
B. Run the below command to check the version of mongo.

splunk cmd mongod -version

If the status of KVstore is failed or stuck at starting proceed further.

2. Validate if the SSL certs are valid as there KVstore has SSL dependency.

/opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem
 Where: /opt/splunk/etc/auth/server.pem is the path of the SSL certificate. This can be different if a 3rd party self signed certificates are being used.

If the SSL certs are valid, proceed further.

3. Take the backup of KVstore before getting to any further steps. To take the backup run the below command,

./splunk backup kvstore

If the above command does not help, then use the below command to take mongo backup.

cp -R $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/ /tmp/

4. Verify in the mondod.log if Splunk tried to migrate mongo to 7.0 or not. Look for logs like the one below,

0x-0x-xxxx xx:0x:xx.xxx +0530 INFO  MongodRunner [19744 KVStoreConfigurationThread] - Starting mongod with executable name=mongod.exe version=kvstore version 7.0

Validate the same for mongo version 4.4, 5.0 and 6.0 as the migration involves automated migration to these mongo versions.

5. Validate if you find binaries for mongo-4.4, mongo-5.0, mongo-6.0 and mongo-7.0 under Splunk bin directory.

6. Verify if the prerequisites to migrate KVstore to mongo 7.0 on 9.4.0 are met.
A. Do not use IPv6.
B. At least 50% disk space on the partition where KVstore is present.
C. Use Splunk default certs while migration and not any 3rd party SSL certs.

7. If all the above are met, then go to 

$SPLUNK_HOME/var/run/splunk/kvstore_upgrade/

And verify the versionFile present there. If versionFile7.0 is present then that's fine, if not, then move any other versionFile out of the location and restart Splunk service. Validate the status of KVstore.

8. If the KVstore is still not in a ready state, proceed further.

Resolution

1. Go to  $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/  and rename VersionFile7.0 to VersionFile4.4 so that mongo4.4 binary will be used.

2. Stop Splunk on the server.

3. Clean KVstore using the below command,

./splunk clean kvstore --local

4. Start the service again and check the status of KVstore now, which should be ready. If it's failed or ready state.

Also check the serverVersion, check if that has changed. Monitor it using the watch command

watch /opt/splunk/bin/splunk show kvstore-status --verbose

5. If serverVersion is not changing, then stop the Splunk service, then configure the below so that there wont be any automatic upgrade of mongo to 7.0 on startup.

./server.conf
[kvstore]
 kvstoreUpgradeOnStartupEnabled = false
Restart Splunk.

6. Manually upgrade the mongo to 7.0.

./splunk start-standalone-upgrade kvstore -version 7.0

Related Links

MigrateKVstore

BackupKVstore