Search Head Cluster failed to add SH members due to large payload after large bundle push

Applicability

Description

After a recent bundle push from deployer to search head cluster (SHC) members running Splunk Enterprise (observed on version 7.2.4), SHC entered a broken state with missing member information:

[splunk@SH1 bin]$ ./splunk show shcluster-status 
 
 Captain: 
 dynamic_captain : 1 
 elected_captain : Wed Feb 20 19:02:42 2019 
 id : [redacted] 
 initialized_flag : 0 
 label : SH1
 mgmt_uri : https://SH1:8089 
 min_peers_joined_flag : 0 
 rolling_restart_flag : 0 
 service_ready_flag : 0 
 
 Members: 
 
 [splunk@SH2 bin]$ ./splunk show shcluster-status 
 
 Captain: 
 dynamic_captain : 1 
 elected_captain : Wed Feb 20 19:02:42 2019 
 id : [redacted]
 initialized_flag : 0 
 label : SH1
 mgmt_uri : https://SH1:8089 
 min_peers_joined_flag : 0 
 rolling_restart_flag : 0 
 service_ready_flag : 0 
 
 Members:  [splunk@SH3 bin]$ ./splunk show shcluster-status 
 
 Captain: 
 dynamic_captain : 1 
 elected_captain : Wed Feb 20 19:02:42 2019 
 id : [redacted]
 initialized_flag : 0 
 label : SH1 
 mgmt_uri : https://SH1:8089 
 min_peers_joined_flag : 0 
 rolling_restart_flag : 0 
 service_ready_flag : 0 
 
 Members:

Investigation

It appears the election was successfully done with all members voted SH1 to be the captain, but member information just could not get updated.

From SHC captain SH1's splunkd.log:

02-20-2019 19:02:53.796 -0600 ERROR SHCRaftConsensus - failed appendEntriesRequest err: uri=https://SH3:8089/services/shcluster/member/consensus/pseudoid/raft_append_entries?output_mode=json, 
socket_error=Connection refused to https://SH3:8089

Action Taken:

1. Tried below procedure to clean up RAFT then bootstrap a static captain but the same result after doing so:

   Fix the entire cluster 

2. Confirmed all members have their serverName defined properly to its own name.

3. curl -s -k https://hostname:8089/services/server/info

4. Also tried increasing the thread through the below setting and restarted Splunk on all members.
   server.conf

   [httpServer] 
   maxSockets =1000000
   maxThreads= 50000

The issue remains the same. None of the SHC members are listed under "show shcluster-status" and the SHC remains broken along with kvstore cluster not established.

Resolution

This issue is due to dispatch directory on each of the SHC members was very large through large bundle push and leads to large payload hence SHC is failing to add the SH members.

You may check splunkd_access.log to look for any 413 PAYLOAD TOO LARGE error:

e.g.
 x.x.x.x - - [20/Feb/2019:19:32:29.471 -0600] "POST /services/shcluster/captain/members HTTP/1.1" 413 180 - - - 0ms
 x.x.x.x - - [20/Feb/2019:19:32:25.024 -0600] "POST /services/shcluster/captain/members HTTP/1.1" 413 180 - - - 0ms

Root Cause: The bundle is too large and hit 2GB limit of max_content_length.
To resolve it try setting the following in server.conf on all the SH members and restart splunk to apply the setting:

[httpServer]
max_content_length=21474836480

Reference:

max_content_length = 
Maximum content length, in bytes.
HTTP requests over the size specified are rejected.
This setting exists to avoid allocating an unreasonable amount 
of memory from web requests.
In environments where indexers have enormous amounts of RAM, this number
can be reasonably increased to handle
large quantities of bundle data.
* Default: 2147483648 (2GB)

The above solution will increase the maximum allowable size for http requests. If increasing the above setting does not resolve the issue, clearing the dispatch may have to be done to remove some of the queued jobs. This will reduce the size of the http request and allow the member to join the cluster. If the cluster member cannot join after restoring KV Store from a failed upgrade (and encountering this error), cleaning dispatch may be required to get it to join the cluster. 

Related Links

Splunk Documentation
Splunkd HTTP server configuration

Splunk Answers
In my Search Head Cluster, why does "show shcluster-status" show captain, but not the members' information?

External Links
HTTP Status Codes Glossary (WebFX)
List of HTTP status codes (Wikipedia)