Single indexer cluster crashing multiple times a day

Applicability

Description

On-prem single side indexer cluster.

Investigation

This issue was reported as a crash after an upgrade. GUI messages display the below error multiple times:

TCPOutAutoLB-0Root Cause(s):
More than 20% of forwarding destinations have failed. Ensure your hosts and ports in outputs.conf 
are correct. Also ensure that the indexers are all running, and that any SSL certificates being used
for forwarding are correct.

Check the cluster status on the CM CLI:

splunk show cluster-status --verbose

All running fine at the time.

On indexers run:
Splunk display listen:
- Receiving on 9997 port.

Netstat -tulpn | grep 9997

- 9997 enabled, listening.   --> unable to gather output from cx.

Check crash logs on indexers:

[build 63079c59e632] 2021-04-09 16:08:32
Received fatal signal 11 (Segmentation fault).
Cause:
  Signal sent by kernel.
Crashing thread: BucketSummaryActorThread

Other crash logs point to '- No memory mapped at address [0x00000000000002B0]' as the crashing thread.

KVstore memory usage increases considerably when an application accesses large lookups with several accelerated fields using lookup commands. The crash logs show the information below on some of the crashes which points to the kvstore memory usage causing the issue:

using CLOCK_MONOTONIC
Preforked process=0/692: process_runtime_msec=302944, search=0/1977, search_runtime_msec=302937, new_user=N, export_search=Y, args_size=369, completed_searches=0, user_changes=0, cache_rotations=0

Thread: "BucketSummaryActorThread", did_join=0, ready_to_run=Y, main_thread=N, token=139928453248768
MutexByte: MutexByte-waiting={none}

BucketSummaryActorThread:
    verify=N
    done=N
    created=N
    terminate=N
    bsi={summaryId=FED0326D-5B6D-497C-B5A0-DA275A1DF525_DM_Splunk_SA_CIM_Intrusion_Detection summaryPath=/app/splunk/var/lib/splunk/zscaler/datamodel_summary/682_F7093F61-8FF6-4CC7-8849-A342725E428C/FED0326D-5B6D-497C-B5A0-DA275A1DF525/DM_Splunk_SA_CIM_Intrusion_Detection bucketId=zscaler~682~F7093F61-8FF6-4CC7-8849-A342725E428C tstats=1}

===============Search Processor Information===============
Search Processor: "litsearch"
type="SP_STREAM"
search_string="((index=* OR index=_*) (index=symantec_wss OR index=cisco_firepower OR index=paloalto OR index=cisco_asa OR index=meraki OR index=cloudflare OR index=zscaler) ((sourcetype="zscalernss-web" threatname!="None") OR (sourcetype=bit9:carbonblack:json (is_alert=* OR alert_type=*)) OR ((sourcetype="cisco:asa" OR sourcetype=cisco_asa OR sourcetype="cisco:pix" OR sourcetype=cisco_pix) Cisco_ASA_message_id="4000*") OR (sourcetype=cisco:sourcefire:appliance:syslog host=fmc) OR (sourcetype=cisco:sourcefire impact>0) OR (sourcetype=cisco:sourcefire (rec_type_simple=EVENT OR rec_type_simple="IPS EVENT")) OR (index=cloudflare sourcetype="cloudflare:json" WAFAction=drop) OR ((sourcetype=eStreamer OR sourcetype=estreamer) archive_timestamp priority) OR (sourcetype=meraki meraki_app="events-airmarshal") OR (sourcetype=meraki meraki_app="ids-alerts") OR ((sourcetype=pan_threat OR sourcetype=pan:threat OR sourcetype=pan_threat) (((sourcetype="pan:aperture" OR sourcetype=pan_aperture) log_type=spyware) OR log_subtype=spyware)) OR ...

===============Search Processor Information===============
Search Processor: "lookup"
type="SP_STREAM"
search_string=""
normalized_search_string=""
litsearch=""
raw_search_string_set=0 raw_search_string=""
args=
input_count=5824
output_count=0
directive_args=
 

In the logs you can also find problems pointing to buckets in the summaryPath:

Errors point to bucket :bsi={summaryId=FED0326D-5B6D-497C-B5A0-DA275A1DF525_DM_Splunk_SA_CIM_Endpoint
.Filesystem summaryPath=/app/splunk/var/lib/splunk/carbonblack/datamodel_summary/1329_C4EB53A7-1686-
4A43-9D61-673421EE9C41/FED0326D-5B6D-497C-B5A0-DA275A1DF525/DM_Splunk_SA_CIM_Endpoint.Filesystem 
bucketId=carbonblack~1329~C4EB53A7-1686-4A43-9D61-673421EE9C41 tstats=1}

Resolution

Implementing changes to increase performance has to be done per install (on each indexer in the cluster). Below is the configuration required in limits.conf to boost performance on the system which would in turn stop the crashes:

[lookup]
enable_splunkd_kv_lookup_indexing = true

Limits.conf Description of the attribute:

enable_splunkd_kv_lookup_indexing = <boolean>
* This setting determines whether KV Store lookup indexing is performed  
 during bundle replication.
* When set to true, KVStore lookup 
indexing occurs on the main splunkd process,  
 asynchronous to searches.
* When set to false, KV Store lookup indexing is triggered by the search  
 process, potentially slowing search performance.
* NOTE: Do not change this setting unless instructed to do so by Splunk Support.
* Default: false

Further information about this particular issue can be found in the next JIRA:

Crashing thread: BucketSummaryActorThread; Signal: Segmentation fault when lookups fail

In there, multiple JIRA's listed in the comments are discussed pointing to the very same issue on different environments. Other errors might be present in the environments that are not discussed in this article. Hence, working issue per case which is discussed in the JIRA above as well.