Upgrade failure for Mongo after upgrading Splunk to 9.4

Applicability

Description

After upgrading Splunk Enterprise to 9.4, Kvstore initiated in failure status and diagnosed that the issue was impacting searches due to the following error message when doing a simple search in a stand-alone instance.

Could not load lookup=LOOKUP-exc_host
Could not load lookup=LOOKUP-splunk_security_essentials

Investigation

The following can be used to help diagnose Mongo Upgrade failure:

1. Check logs in $SPLUNK_HOME/etc/var/log/splunk/mongod.log:
   

   E  STORAGE  [initandlisten] WiredTiger error (-31802) [1739632215:112475][2157746:0x7f6315a43b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error Raw: [1739632215:112475][2157746:0x7f6315a43b40], connection: __log_open_verify, 925: unsupported WiredTiger file version: this build only supports versions up to 4, and the file is version 5: WT_ERROR: non-specific WiredTiger error
   W  STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
   F  STORAGE  [initandlisten] Reason: -31802: WT_ERROR: non-specific WiredTiger error
   F  -        [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 928
   F  -        [initandlisten] \n\n***aborting after fassert() failure\n\n

2. Check log located in $SPLUNK_HOME/etc/var/log/splunk/mongo_upgrade.log 
   

   INFO  [mongod_upgrade::commands] UPGRADE_TO_4.4_SUCCESSFUL
   INFO  [mongod_upgrade::commands] Attempting to update status
   INFO  [mongod_upgrade::commands] Status updated successfully
   INFO  [mongod_upgrade] Waiting for other nodes in replica set to upgrade
   DEBUG [mongod_upgrade::conditions] Upgraded count: 1
   INFO  [mongod_upgrade] All upgraded to 4.4, proceeding.
   INFO  [mongod_upgrade] Setting new FCV Version: 4.4
   INFO  [mongod_upgrade] FCV change successful: ()
   INFO  [mongod_upgrade] Received signal 15 - preparing to shut down
   

   In the previous scenario, somehow Splunk decided to stop the upgrade after reaching 4.4 version. NOTE: a successful upgrade log should look like this:
   

   INFO  [mongod_upgrade::commands] UPGRADE_TO_7.0_SUCCESSFUL
   INFO  [mongod_upgrade::commands] Attempting to update status
   INFO  [mongod_upgrade::commands] Status updated successfully
   INFO  [mongod_upgrade] Waiting for other nodes in replica set to upgrade
   DEBUG [mongod_upgrade::conditions] Upgraded count: 1
   INFO  [mongod_upgrade] All upgraded to 7.0, proceeding.
   INFO  [mongod_upgrade] Setting new FCV Version: 7.0
   INFO  [mongod_upgrade] FCV change successful: ()
   INFO  [mongod_upgrade] Upgrades completed
   INFO  [mongod_upgrade] Waiting for completion
   DEBUG [mongod_upgrade::conditions] Completed count: 1
   DEBUG [mongod_upgrade::conditions] Hostname count: 1
   INFO  [mongod_upgrade] All completed
   INFO  [mongod_upgrade] Dropped migration metadata database.
   INFO  [mongod_upgrade::commands] Shutting down the database
   WARN  [mongod_upgrade] mongod failed to shut down before exiting: Kind: I/O error: unexpected end of file, labels: {}

   
   
   
3. Verify in $SPLUNK_HOME/var/run/splunk/kvstore_upgrade. A file will be located if the file is named "versionFile42", which means the upgrade cannot be completed.
   A successful upgrade should store a file named "versionFile70"

Resolution

To fix the error causes by the incomplete upgrade, perform the next steps:

1. Go to $SPLUNK_HOME/etc/system/local/server.conf. Edit with following attribute and value
   

   [kvstore]
   kvstoreUpgradeOnStartupEnabled = false

2. Move the current Kvstore directory
   

   mv $SPLUNK_HOME/var/lib/splunk/kvstore/mongo $SPLUNK_HOME/var/lib/splunk/kvstore/mongo_bak

3. Restore kvstore backup (Backup created by Splunk before upgrading to 9.4)
   

   mv $SPLUNK_HOME/var/lib/splunk/kvstore/mongo_backup/mongo $SPLUNK_HOME/var/lib/splunk/kvstore/
   

4. Restart Splunk
5. Check instance is ready to migrate
   

   $SPLUNK_HOME/bin/splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true
   

6. If checks are passed, continue with the upgrade
   
   

   $SPLUNK_HOME/bin/splunk start-standalone-upgrade kvstore -version 7.0

7. Upgrade will be run in the background, to monitor its status use this command
   

   $SPLUNK_HOME/bin/splunk show standalone-kvupgrade-status

8. Once the upgrade finishes, verify Kvstore status and version
   

   $SPLUNK_HOME/bin/splunk show kvstore-status --verbose

   
   

   $SPLUNK_HOME/bin/splunk cmd mongod -version

Search Head Cluster Troubleshooting

In case steps above does not work by upgrading one by one in a Search Head Cluster Environment, follow these steps, in these ones is a must to take a backup of kvstore since next step is going to clean kvstore to force the upgrade in a fresh kvstore directory: Kvstore has stuck at starting stage for all the search heads in cluster.

Related Links

Upgrade the KV store server version

Kvstore has stuck at starting stage for all the search heads in cluster.