Troubleshooting KVStore Failures and Search Head Clustering Issues in Splunk Enterprise

Applicability

Description

Common Symptoms:

• KVStore status shows "failed" or "recovering"
• Splunk Web UI displays "KV Store is down"
• Errors in mongod.log and splunkd.log
• Upgrade hangs or fails due to FCV mismatch or invalid storage paths
• SHC nodes cannot elect a captain or show "missing common baseline with the captain"
• KVStore stuck in "starting" status after service restart

Investigation

Initial Troubleshooting Steps

• Run KVStore status checks:
  $SPLUNK_HOME/bin/splunk show kvstore-status --verbose
• curl -k -u user:pass https://<host>:<mPort>/services/kvstore/status
• Review logs:
  • $SPLUNK_HOME/var/log/splunk/mongod.log
  • $SPLUNK_HOME/var/log/splunk/splunkd.log
  • $SPLUNK_HOME/var/log/splunk/mongod_upgrade.log
• Look for errors such as:
  • Detected unclean shutdown - mongod.lock is not empty
  • Invalid feature compatibility version value
  • KVStore process terminated abnormally (exit code 14)
  • Missing common baseline with the captain
  • Node seems to have already joined another cluster
• Confirm that TCP connection between hosts is possible in the management port (8191) 

Resolution

1. Resolution: Unclean Shutdown

• If log shows unclean shutdown:
  Detected unclean shutdown - mongod.lock is not empty
  • Action:
    rm -f $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.lock
  • $SPLUNK_HOME/bin/splunk restart

2. Resolution: FCV Upgrade Stuck

• If interrupted during FCV upgrade (e.g., 4.4 > 5.0 > 6.0 > 7.0):
  • Action:
    touch $SPLUNK_HOME/var/run/splunk/kvstore_upgrade/versionFile50
  • $SPLUNK_HOME/bin/splunk resync kvstore -featureCompatibilityVersion 5.0
  • Continue process until FCV reaches 7.0

3. Resolution: New SHC Member Fails to Join

• If replicationStatus is Down and lastHeartbeatRecvSec = 0:
  • Check for SSL compression mismatch:
    $SPLUNK_HOME/bin/splunk btool server list sslConfig --debug
  • Align allowSslCompression across SHC members:
    [sslConfig]
  • allowSslCompression = false
  • Restart the affected SH node.

4. SHC Rebuild and KVStore Sync Post-Patching

Symptoms:

• No captain in SHC
• Nodes stuck due to raft metadata
• KVStore in "starting" or failed state

Actions:

• Stop Splunk on all SH nodes:
  

systemctl stop Splunkd

• Clean raft metadata on each SH:
  

splunk clean raft

• Fix KVStore issues:
  

chmod 600 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.keychown splunk:splunk $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.keyrm -f $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.lock

• (Optional) Clean KVStore if corruption suspected:
  

  splunk clean kvstore --localsplunk clean kvstore --cluster

   

• Restart all SH nodes:
  

  systemctl start Splunkd

   

• Bootstrap captain from one healthy SH node:
  

  splunk bootstrap shcluster-captain -servers_list "https://sh1:8089,https://sh2:8089,https://sh3:8089" -auth admin:<password>

• Push fresh bundle from deployer:
  

  splunk apply shcluster-bundle -target https://<captain>:8089 -auth admin:<password>

• Validate final state:
  

  splunk show shcluster-statussplunk show kvstore-status

Related Links

TroubleshootKVstore

ResyncKVstore

MigrateKVstore

Troubleshoot_a_failed_upgrade_to_server_version_7.0

Handleraftissues