KV Store is stuck at starting status in a Search Head Cluster (SHC)

Applicability

Description

All Search Head Cluster members are stuck in a "starting..." state.

Note:
If the SHC is running healthy, but only one or two KV store member show "starting", then first attempt to resync the kvstore following these steps from Splunk documentation: Resync stale KV store members

Investigation

Splunk internal logs were checked and the next was found:

From the SHC Captain:
splunkd.log:

03-25-2021 17:25:12.247 -0400 WARN SHPConfReplicationBookmarkFromKVStore - writeSHCStateToStateStore:
failed to write SHC state to KVstore:03-25-2021 17:24:42.246 -0400 WARN SHPConfReplicationBookmarkFromKVStore - service: failed to read 
SHC state from KVstore: read status: -96

mongod.log:

2021-03-23T20:18:17.337Z I REPL [initandlisten] Did not find local replica set configuration 
document at startup; NoMatchingDocument: Did not find replica set configuration document in 
local.system.replset2021-03-23T20:18:17.338Z I NETWORK [initandlisten] listening via socket bound to 0.0.0.02021-03-23T20:18:17.338Z I NETWORK [initandlisten] waiting for connections on port 8191 ssl2021-03-23T20:18:17.338Z I CONTROL [LogicalSessionCacheReap] Sessions collection is not set up; 
waiting until next sessions reap interval: config.system.sessions does not exist

These are indications of a race condition with KV store captaincy change during the previous attempt, which leads to all KV store members being stuck at "starting" status. Based on the logs above, the KV store is out of sync and it could also be experiencing some network issues when attempting to connect to other servers in the KV store port 8191.

Resolution

Proposed Resolution:
==================
Follow below procedure, often referred to as a "destructive resync" to restore KV Store cluster without Downtime on the SHC.

Proposed Steps
==================

1. Preparation
   1. Ensure each SHC member can establish a successful connection on port 8191. For example (does not require https://):
      

      ./splunk cmd openssl s_client -connect <FQDN>:8191

   2. Find which SH is the current SHC Captain by running the following command from any member:
      

      /opt/splunk/bin/splunk show shcluster-status --verbose

   3. Identify a known good member that is NOT the SHC captain, which will be used for backing up the KV Store.
2. Clear and disable KV Store on each member:
   1. ONLY on your one good member (Not the SHC captain):
      1. Stop splunk on this member only.
      2. Backup the kvstore by copying the full directory and its contents. Because the kvstore is not in a ready state, the kvstore backup command will not work. (please make sure the kvstore-backup folder below does not already exist on this system from a previous restore attempt)
         For example: : 

         cp -R $SPLUNK_DB/kvstore $SPLUNK_DB/kvstore-backup

         OR:
         

         cp -R /opt/splunk/var/lib/splunk/kvstore /opt/splunk/var/lib/splunk/kvstore-backup

      3. Clean up KV Store with this command (this will EMPTY the kvstore database itself):

         splunk clean kvstore --local

      4. With splunk still down, disable KV Store in server.conf (see: Server.conf)

         [kvstore]
         disabled = true

      5. Now start Splunk on this member.
   2. Follow this same operation, one by one, with the remaining members EXCEPT the captain, but omit the backup steps):
      1. Stop splunk on the current non-captain member.
      2. Clean up KV Store with this command (this will EMPTY the kvstore itself):

         splunk clean kvstore --local

      3. With splunk still down, disable KV Store in server.conf 

         [kvstore]
         disabled = true

      4. Start Splunk on this member.
   3. Lastly, follow the same steps with the Captain (again no backup required).
3. At this point, you should have a working SHC with KV Store disabled on all members. Run the following command to confirm the search head cluster is in good shape:

   ./splunk show shcluster-status --verbose
   

4. Note again which member is captain, as captaincy will have changed.
5. Re-enable KV Store on each member, starting with the good member.
   1. On the one known good member: 
      1. Stop Splunk.
      2. Enable kv store in server.conf:

         [kvstore]
         disabled = false

      3. Copy the kvstore backup folder back to $SPLUNK_DB/kvstore. For example:

      4. mv $SPLUNK_DB/kvstore $SPLUNK_DB/kvstore-backup2
         cp -R $SPLUNK_DB/kvstore-backup $SPLUNK_DB/kvstore

      5. Run the following command to clear out cluster configuration info. 
         (There should be no $SPLUNK_DB/kvstore/mongo/local.* files left after this step)
         (Splunkd does not need to be running to use this command.)

         splunk clean kvstore --cluster

      6. Start Splunk.
   2. Now, on each additional member EXCEPT the captain, again enable kvstore and clean the cluster (backup not required):
      1. Stop Splunk.
      2. Enable kv store in server.conf

         [kvstore]
         disabled = false

      3. Run the following command to clear out cluster configuration info. 
         (There should be no $SPLUNK_DB/kvstore/mongo/local.* files left after this step)
         (Splunkd does not need to be running to use this command.)

         splunk clean kvstore --cluster

      4. Start Splunk.
   3. Once again, follow the same steps with the Captain (again no backup to restore).
6. Verify KV Store status is ready on each member, and confirm no issues in splunkd.log on the SHC Captain.
   Example for verifying KVstore is ready (run from CLI on each member): 
   

   curl -s -k https://localhost:8089/services/server/info | grep kvStoreStatuscurl -s -k https://localhost:8089/services/server/info | grep kvStoreStatus

Cluster members will resynchronize all data from the first member. Give them some time to do this (could take a bit if your kvstore is quite large).

Related Links

May require scheduling a maintenance window to perform this operation.

• Resync stale KV store members
• Kvstore has stuck at starting stage for all the search heads in cluster
• Server.conf.spec