Search Head Cluster Member Stuck in KV Store Recovering.

Applicability

Description

KV Store replication stuck on initial sync or recovery status.

Errors found on $SPLUNK_HOME/var/log/splunk/splunkd.log of the instance affected.
 

ERROR KVStoreBulletinBoardManager - Local KV Store has replication issues. See introspection data and mongod.log for details. Local instance has state Recovering..
ERROR KVStoreBulletinBoardManager - Local KV Store has replication issues. See introspection data and mongod.log for details. Local instance has state Initial Sync..
The member is too stale to use <cluster member host name> as a sync source. 

Investigation

How to identify it:
1. Some of the search heads are stuck in recovering status for KV store.

2. following the Resync the KB store in the Splunk documentation didn’t fix the issue:

• Determine which node is currently the search head cluster captain using the CLI command splunk show shcluster-status.
• Log into the shell on the search head cluster captain node.
• Run the command splunk resync kvstore [-source sourceId]. -source is an optional parameter, if a member other than the search head cluster captain needs to be used as the source. SourceId refers to the GUID of the search head member that will be used to synchronize.
•  Enter the admin login credentials.
•  Wait for a confirmation message on the command line.
•  Use the splunk show kvstore-status command to verify that the cluster is resynced.

Resolution

How to fix it:
1. Run 'splunk resync kvstore' from SH captain, KV store captain will come up as ready.

2. On the other nodes (the ones stuck in recovering status), stop Splunk process and perform a "splunk clean kvstore --local", then restart Splunk.

The nodes will come up in 'initial sync' on KV store until completed. Monitor the 'initial sync' by checking the kvstore files are transfer with that size of $SPLUNK_HOME/var/lib/splunk/kvstore is growing.

Sample command:

 du -sh $SPLUNK_HOME/var/lib/splunk/kvstore 

Then they will be in ready status with non-captain KV store member.

If the solution above doesn't fix the issue follow the next steps: 

1. Remove the member from the cluster, run the below command from the instance that needs to be removed:

./splunk remove shcluster-member

To run the splunk remove command from another member, use this version of the command:

./splunk remove shcluster-member -mgmt_uri <URI>:<management_port>

2. Stop Splunk
3. To confirm that this instance has been removed from the KV store, run the following command on any remaining cluster member.

./splunk show kvstore-status 

4. Before starting Splunk again run:

./splunk clean all

NOTE: The clean all command will remove the local admin account. A file called user-seed.conf can be used to reconfigure an admin user after the passwd file is moved or removed from $SPLUNK_HOME/etc. Please refer to the spec file for more information: User-seed.conf 

5. Start Splunk. 
6. Add the instance to the cluster: 
When running the Splunk add command on the new member itself, use this version of the command:

./splunk add shcluster-member -current_member_uri <URI>:<management_port>

When running the Splunk add command from a current cluster member, use this version of the command:

./splunk add shcluster-member -new_member_uri <URI>:<management_port>

7. Allow time for the instance to complete the sync process.

Related Links

Splunk Documentation

Resync the KV store

Remove a cluster member

Add a member to the cluster