ERROR: "Search peer has the following message: Failed to register with cluster master reason: failed"

Applicability

Description

Search peer <indexer_name> has the following message: Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?
output_mode=json master=<cluster_master_node>:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx 
status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=<indexer_ip> mgmtport=8089 (reason: non-zero pending job count=8, 
guid=<indexer_GUID). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=40544E13F5C497C4493557620EC60123 add_type=Clear-Masks-And-ReAdd 
base_generation_id=10009 batch_serialno=1 batch_size=45 forwarderdata_rcv_port=9996 forwarderdata_use_ssl=1 last_complete_generation_id=0 latest_bundle_id=
40544E13F5C497C4493557620EC60123 mgmt_port=8089 name=<indexer_GUID> register_forwarder_address= register_replication_address= register_search_address= 
replication_port=9880 replication_use_ssl=0 replications=indexname~bucket#~<indexer_GUID>::<indexer_GUID>::<indexer_GUID> server_name=<indexer_name> 
site=default splunk_version=x.x.x splunkd_build_number=<build_number> status=Up } ].

Investigation

The above example shows that the peer's latest cluster bundle ID (latest_bundle_id) was 40544E13F5C497C4493557620EC60123. The output of $SPLUNK_HOME/bin/splunk show cluster-bundle-status command on the cluster manager, however, showed that the ID had been updated.

Cluster manager's splunkd.log confirmed that the bundle update was successful across all indexers:

01-20-2020 19:34:47.042 +1000 INFO CMBundleMgr - setting active bundle=40544E13F5C497C4493557620EC60123 to latest bundle=3BA36284AC1048488AED4426319C5555
.
01-20-2020 19:34:49.495 +1000 INFO CMMaster - All peers have reloaded the bundle without a restart.
01-20-2020 19:34:49.495 +1000 INFO CMMaster - Triggered fetching of remote storage indexes info as part of event="bundle has been pushed successfully to slaves"
01-20-2020 19:34:49.495 +1000 INFO CMBundleMgr - apply bundle status transitioning from='Bundle reload is in progress. Waiting for all peers to return the status.', to='None'

Furthermore, the peer's splunkd.log confirmed that its bundle had been updated successfully (notice the new ID):

01-20-2020 19:34:49.495 +1000 INFO CMBundleMgr - setting active bundle=40544E13F5C497C4493557620EC60123 to latest bundle=3BA36284AC1048488AED4426319C5555
01-20-2020 19:34:49.495 +1000 INFO CMBundleMgr - Remove old bundles.
01-20-2020 19:34:49.495 +1000 INFO BundleJob - No restart required at reload - updated the active bundle id=3BA36284AC1048488AED4426319C5555
01-20-2020 19:34:49.497 +1000 INFO BundleJob - Informed bundle reload status=4 to the master for bundle id=3BA36284AC1048488AED4426319C5555

Resolution

In the above example, both the cluster manager's splunkd.log and the peer's splunkd.log confirmed that the cluster bundle had been updated successfully, therefore the error message was benign. The message should clear by itself, but if it does not, follow the following recommendations: 

1. Perform name resolution tests to confirm there is no DNS issue. Slow DNS is known to delay Splunk cluster communications.

2. Restart Splunk services on the cluster manager to force refresh the cluster communication. For Splunk Cloud environment, please open a support ticket to request for a restart of the Cluster Master.