Search Head is Not able to join Search Head Cluster (static captain)

Available Languages

Updated:September 30, 2026

Document ID:SPLK050999

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise

Description

After performing SH migration, Cu found one a SH is not able to join back to SHC.

There are 3 SHs,
SH11 -> It is in SHC
SH12. -> not able to join back to SHC
SH13 -> It is in SHC

static captain is used in this SH env

1. On SH12,
run "splunk show shcluster-status --verbose"

 Captain:
        decommission_search_jobs_wait_secs : 180
                       dynamic_captain : 0
                       elected_captain : Thu Jul 13 14:22:57 2023
                                    id : BC993918-F9C6-4058-A27A-************
                      initialized_flag : 1
            kvstore_maintenance_status : disabled
                                 label : se-sinkc1-a3112.*******.com
         max_failures_to_keep_majority : 18446744073709551615
                              mgmt_uri : https://10.168.***.***:8089
                 min_peers_joined_flag : 1
                       rolling_restart : restart
                  rolling_restart_flag : 0
                  rolling_upgrade_flag : 0
                    service_ready_flag : 1
                        stable_captain : 0 Cluster Manager(s):
    https://10.168.161.12:8089        splunk_version: 9.0.2 Members:

2. On SH11,
run "splunk show shcluster-status --verbose"

 Captain:
		decommission_search_jobs_wait_secs : 180
		               dynamic_captain : 0
		               elected_captain : Wed Jul 12 15:56:14 2023
		                            id : BC993918-F9C6-4058-A27A-*********
		              initialized_flag : 1
		    kvstore_maintenance_status : disabled
		                         label : se-sinkc1-a3113.*********
		 max_failures_to_keep_majority : 1
		                      mgmt_uri : https://10.168.*********:8089
		         min_peers_joined_flag : 1
		               rolling_restart : restart
		          rolling_restart_flag : 0
		          rolling_upgrade_flag : 0
		            service_ready_flag : 1
		                stable_captain : 1 Cluster Manager(s):
	https://10.168.161.12:8089		splunk_version: 9.0.2 Members: 
	se-sinkc1-a3111.singnet.com.sg
		                kvstore_status : ready
		                         label : se-sinkc1-a3111.*********
		         last_conf_replication : Fri Jul 14 10:31:56 2023
		              manual_detention : off
		                      mgmt_uri : https://10.168.*********:8089
		                mgmt_uri_alias : https://10.168.*********:8089
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 9.0.2
		                        status : Up
	se-sinkc1-a3112.singnet.com.sg
		                kvstore_status : starting
		                         label : se-sinkc1-a3112.*********
		         last_conf_replication : Wed Jul 12 15:56:45 2023
		              manual_detention : off
		                      mgmt_uri : https://10.168.*********
		                mgmt_uri_alias : https://10.168.*********
		              out_of_sync_node : 1
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 9.0.2
		                        status : Up
	se-sinkc1-a3113.singnet.com.sg
		                kvstore_status : ready
		                         label : se-sinkc1-a3113*********
		              manual_detention : off
		                      mgmt_uri : https://10.168.*********
		                mgmt_uri_alias : https://10.168.*********
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 9.0.2
		                        status : Up

3. On SH13,
run "splunk show shcluster-status --verbose"

 Captain:
		decommission_search_jobs_wait_secs : 180
		               dynamic_captain : 0
		               elected_captain : Wed Jul 12 15:56:14 2023
		                            id : BC993918-F9C6-4058-A27A-******
		              initialized_flag : 1
		    kvstore_maintenance_status : disabled
		                         label : se-sinkc1-a3113.******
		 max_failures_to_keep_majority : 1
		                      mgmt_uri : https://10.168.******
		         min_peers_joined_flag : 1
		               rolling_restart : restart
		          rolling_restart_flag : 0
		          rolling_upgrade_flag : 0
		            service_ready_flag : 1
		                stable_captain : 1 Cluster Manager(s):
	https://10.168.161.12:8089		splunk_version: 9.0.2 Members: 
	se-sinkc1-a3111.singnet.com.sg
		                kvstore_status : ready
		                         label : se-sinkc1-a3111.******
		         last_conf_replication : Fri Jul 14 10:34:56 2023
		              manual_detention : off
		                      mgmt_uri : https://10.168.******
		                mgmt_uri_alias : https://10.168.******
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 9.0.2
		                        status : Up
	se-sinkc1-a3112.singnet.com.sg
		                kvstore_status : starting
		                         label : se-sinkc1-a3112.******
		         last_conf_replication : Wed Jul 12 15:56:45 2023
		              manual_detention : off
		                      mgmt_uri : https://10.168.******
		                mgmt_uri_alias : https://10.168.******
		              out_of_sync_node : 1
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 9.0.2
		                        status : Up
	se-sinkc1-a3113.singnet.com.sg
		                kvstore_status : ready
		                         label : se-sinkc1-a3113.******
		              manual_detention : off
		                      mgmt_uri : https://10.168.******
		                mgmt_uri_alias : https://10.168.******
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 9.0.2
		                        status : Up

Investigation

[+] SH13

etc/system/local/server.conf
[shclustering]
election = 0
mode = captain

[+] SH12

etc/system/local/server.conf
[shclustering]
election = 0
mode = captain <------ we have two captain in this SHC :) we can have only one captain in static election

[+] SH11

etc/system/local/server.conf
[shclustering]
election = 0
mode = member

Resolution

If we needs SH-13 be the captain, then make SH-12 as a member.

On SH-12, run this CLI command:

splunk edit shcluster-config -mode member -captain_uri <URI>:<management_port> -election false

Use cases for static captain
https://docs.splunk.com/Documentation/Splunk/9.0.2/DistSearch/Staticcaptain#Use_cases_for_static_captain

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

This Document Applies to These Products

Splunk Enterprise