Error pulling configurations from the search head cluster captain; consider performing a destructive configuration resync on this search head cluster member

Applicability

Description

The Search Head Cluster can fall out of sync with what the captain has listed as configuration updates for various reasons: unsuspected outages, network issues, and large size for replication are just a few of the reasons why this issue occurs.
What is occurring is the member attempting to apply the set of intervening replicated changes from the captain and if the set exceeds the purge limits and the member and captain no longer share a common commit, a banner message appears on the member's UI, with text similar to the following:

Error pulling configurations from the search head cluster captain; consider performing a destructive configuration resync on this search head cluster member.

To resolve this, manually resync the cluster members with the captain.

Investigation

Reference knowledge article Reason behind Search head members going out of sync for reasons why a search head member goes out of sync.

Error pulling configurations from the search head cluster captain; consider performing a destructive configuration resync on this search head cluster member.

This log can exist in splunkd.log as well, and it can identify which hosts need to have a destructive resync:

ERROR ConfReplicationThread [nnnnnnn ConfReplicationThread] - Error pulling configurations from captain=https://X.X.X.X:8089, consecutiveErrors=nnn msg="Error in fetchFrom, at=: Non-200 status-code=500: refuse request without valid baseline; snapshot exists at op_id=xxxxxxxxxxxxxxxxxxxxxxxx for repo=https://X.X.X.X:8089": Search head cluster member (https://X.X.X.X:8089) is having problems pulling configurations from the search head cluster captain (https://X.X.X.X:8089). Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member.

The address mentioned after "Search head cluster member" is the host with an outdated op ID. 

Resolution

For Splunk Cloud deployments these steps will be performed by internal teams.

Confirm which node is the current captain by running the following command from the Splunk bin directory:

./splunk show shcluster-status

Find out which node is the captain, and run the following command on every stale member (excluding the captain) from the Splunk bin directory:

splunk resync shcluster-replicated-config

Important Note: If Splunk needs to be resynced frequently, there may be a replication issue in the environment causing it to go out of sync. This will require investigation to find out the cause. It could be network bandwidth, frequently updating lookups, the need to increase replication timeouts, etc. 

Related Links

Perform a manual resync
Why do I have this search head cluster synchronization problem?

Reason behind Search head members going out of sync.