Cluster Master crash with CMAddPeerWorker thread when peers trying to join the cluster

Applicability

Description

Cluster master continiously crashing with below crash thread and because of this Indexer Cluster is very unstable.

[build a414fc70250e] 2024-10-30 20:41:12
Received fatal signal 6 (Aborted) on PID 8940.
Cause:
Signal sent by PID 8940 running under UID 1001.
Crashing thread: CMAddPeerWorker-0

Investigation

Below are the investigation steps that can be followed to troubleshoot Indexer Cluster when Cluster Master is crashing upon peer trying to join the cluster:

1. Start from checking the crash thread on the Cluster Master, as every Splunk crash generates a crash thread. If the crash thread matches the once below, proceed further. 

[build a414fc70250e] 2024-10-30 20:41:12
Received fatal signal 6 (Aborted) on PID 8940.
Cause:
Signal sent by PID 8940 running under UID 1001.
Crashing thread: CMAddPeerWorker-0

2. If the above crash thread matches, verify how many peers are trying to join the cluster at once. This can be verified by checking the below parameter on Cluster Master.

server.conf
[clustering]
max_concurrent_peers_joining = <>

Use the btool command to verify,

splunk btool server list --debug clustering | grep -i "max_concurrent_peers_joining"

3. Cluster Master crahses with CMAddPeerWorker thead when there are too many peers joining the cluster at a time and CM is resource constrained or the idnexer trying to join the cluster is having replication queue full/replication issues.

To verify CM resource usage,

Settings-->Monitoring Console-->Resource Usage-->Resource Uage:Instance-->Verify the load average and resource usage

To verify if there are replication queue issues,

index=_internal host=<all indexers> "replication queue" "full" sourcetype=splunkd | timechart span=1m count by peer useother=f

4. If all of the above is fine, then proceed to decrease "max_concurrent_peers_joining" on Cluster Master under clustering stanza server.conf.

./server.conf
[clustering]
max_concurrent_peers_joining = <>

Resolution

If none of the above helps, then configure the below on Cluster Master, under clusteritng stanza server.conf.

./server.conf
[clusteting]
enable_parallel_add_peer = false

Restart Cluster Master

Additionally, once the Indexer joins the cluster, there are possibility of standalone buckets or many fixup tasks. Use the below to increase the speed of fixup tasks given that the Cluster has enough resource.

Run on Cluster Master,

splunk edit cluster-config -max_peer_build_load <>
splunk edit cluster-config -max_peer_rep_load <>

Restart of CM is not required.