Kvstore is stuck in starting status in search head cluster and resync command not working

Applicability

Description

In the search head cluster, kvstore continued to fail even after executing

splunk resync kvstore

. Timeout occurred during the resynchronization. It was tried for a couple of times but failed every time. 

Then

splunk clean kvstore --local

 was run on each search head, this command completed and swept kvstore data however the kvstore status did not become ready after restarting the instance.

Investigation

First, boot-strap a search head cluster with only one instance and wait for kvstore to be ready, then rejoin others.

To rebuild the cluster, we run necessary commands and results were as follows

• run 

  splunk clean kvstore --cluster

  • This did not finish after running for a while.
  • The process had to be killed to cancel the job.

•  run 

  splunk clean kvstore --local

  • Completed and cleared the kvstore data locally
  • However "splunk start" and "splunk show kvstore-status" showed starting status for extended period and the issue persisted. 

Resolution

The cause of the issue was kv store port was not in LISTEN status and but blocked by firewall.

• It was noticed that kvstore port was blocked due to nftables ruleset which was added recently.

  • ss -antup

     showed that tcp SYN_SENT between kvstore 8191 port and local: 127.0.0.1:xxxxx (splunkd adhoc ports).
  • This was because TCP connections were blocked by firewall on the local machine OS (on SHC members server).

• nft list ruleset

   showed there was a ruleset to block local tcp connection for security
• To solve this issue the setting below was added to nftables
  • "ip saddr 127.0.0.1 tcp dport 8191 accept"

kvstore will start working normally.