Failed to migrate KV Store storage engine to WiredTiger.

Applicability

Description

KV Store upgrade to wiredTiger failed on the on-prem Splunk instance.
Steps  taken:
1. ran kvstore-backup in MMAP
2. stopped Splunk
3. prepared server.conf for wiredTiger
4. moved the mongo directory
5. started Splunk
6. Tried to restored the kvstore and failed.

The following error message is bein noticed:

ERROR: Failed to migrate engine wiredTiger, reason=Failed to receive response from KVstore error=, service not ready after waiting for timeout=30031ms

Investigation

Verify the certs, use:

openssl x509 -enddate -noout -in ./server.pem

If the message received is like this: server is not due until 2026, then the certificate will not be the issue 

Next, try the following: 
Generate a new GUI for Splunk by:
Go to /opt/splunk/etc
Rename the instance.cfg to “instance.cfg.backup” and restart Splunk.
Once restarted Splunk will generate a new GUID  and after prechecks, Splunk should be working normally.
A new Instance.cfg will be generated, force the resync by:

./splunk resync kvstore -source <GUID of Node>
./splunk resync kvstore -source <number you obtain from the new “instance.cfg”>

After this process has been completed, use the following command to verify synchronization

./splunk show kvstore-status

Resolution

The best solution will be to find if it is feasible to reinstall Splunk.  Due to the new Splunk version (hopefully the latest available) coming with the latest version of Mongo, it will not get any issues when it comes to KV Store.

To follow the proper steps to uninstall/upgrade Splunk here you have the following pages:
Uninstall Splunk Enterprise
How to upgrade Splunk Enterprise= here you can find full details about the process 


 

Related Links