Search head cluster members are out of sync

Applicability

search head clustering

Description

In a search head cluster (SHC) environment the configuration is replicated from the SHC captain to all SHC peers.  This replication ensures that when searches are delegated to peers from the captain, they all have the same baseline configuration thus ensuring that the search we produce the same results regardless of which SH is delegated to run it.  If there is an issue with replicating the configuration, the SH becomes out of sync and cannot perform searches.  
 

Investigation

The following error message was found in splunkd.log showing issues with replicating configuration:
 

ERROR ConfReplicationThread - Error pulling configurations from captain=https://splunk.domain.com:8089, consecutiveErrors=676 msg="Error in fetchFrom, at=: Non-200 status_code=500: refuse request
without valid baseline; snapshot exists at op_id=2391d4b0a120046bf2ff9abe2e93ea9390cd7d18 for repo=https://splunk.domain.com:8089": Search head clustering:
Search head cluster member (https://splunk.domain.com:8089) is having problems pulling configurations from the search head cluster captain (https://splunk.domain.com:8089).
Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a
destructive configuration resync on this search head cluster member.

Check the last_conf_replication time by running the shcluster-status command:

splunk@sh1:/opt/splunk/bin/splunk show shcluster-status
  
Captain:
               dynamic_captain : 1
               elected_captain : Mon Oct 31 18:48:16 2022
                            id : E09B2CEC-C392-47AE-B70D-641B164D059B
              initialized_flag : 1
                         label : sh1
                      mgmt_uri : https://sh1:8089
         min_peers_joined_flag : 1
          rolling_restart_flag : 0
            service_ready_flag : 1
Members:
...<snip>...
 
sh2
                         label : sh4
         last_conf_replication : Sat Oct 29 15:46:51 2022
                      mgmt_uri : https://sh4:8089
                mgmt_uri_alias : https://x.x.x.x:8089
                        status : Up

In this case, member sh4 has been out of sync for 2 days.

Resolution

A destructive resync needed to be run on the SHC member that was out of sync.  This forced the member to resync with the captain. On sh4, the following was run:

splunk@sh4:/opt/splunk/bin/splunk resync shcluster-replicated-config

Once complete, we verified all peers were in sync by running the shcluster-status command again:
 

splunk@sh1:/opt/splunk/bin/splunk show shcluster-status
  
Captain:
               dynamic_captain : 1
               elected_captain : Tues Nov 1 10:48:16 2022
                            id : E09B2CEC-C392-47AE-B70D-641B164D059B
              initialized_flag : 1
                         label : sh1
                      mgmt_uri : https://sh1:8089
         min_peers_joined_flag : 1
          rolling_restart_flag : 0
            service_ready_flag : 1
Members:
...<snip>...
 
sh2
                         label : sh4
         last_conf_replication : Tues Nov 1 10:48:18 2022
                      mgmt_uri : https://sh4:8089
                mgmt_uri_alias : https://x.x.x.x:8089
                        status : Up

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release