Splunk search head crashed

Applicability

Description

Environment

Indexer and a Search Head Clustering environment:
1 x Cluster Master
3 x Indexers 
3 x Search Head 
1 x Deployment Server 
1 x Deployer
 

Investigation

The logs were reviewed and the next was found:

a. ulimit configuration was not the recommended:

-open files (-n) 1024 (too low should be 64000 minimum)
-max user processes (-u) 1024 (too low should be 16000 minimum)

b. The next searches were skipping:

Scheduled search name app user Skip count

_ACCELERATE_A1C6DBE7-4E0F46F47141_splunk_app_aws_nobody_0a9e0cdf2c63ea09_ACCELERATE_ splunk_app_aws nobody 3486
_ACCELERATE_A1C6DBE7-4E0F46F47141_splunk_app_aws_nobody_63b0f00b79375849_ACCELERATE_ splunk_app_aws nobody 3486
_ACCELERATE_A1C6DBE7-4E0F46F47141_splunk_app_aws_nobody_ed1c132904831158_ACCELERATE_ splunk_app_aws nobody 3486
_ACCELERATE_A1C6DBE7-4E0F46F47141_splunk_app_stream_nobody_615f3533e7_ACCELERATE_ splunk_app_stream nobody 3486
_ACCELERATE_DM_splunk_app_aws_CloudFront_Access_Log_ACCELERATE_ splunk_app_aws nobody 5910
_ACCELERATE_DM_splunk_app_aws_Detailed_Billing_ACCELERATE_ splunk_app_aws nobody 5910
_ACCELERATE_DM_splunk_app_aws_Instance_Hour_ACCELERATE_ splunk_app_aws nobody 5910
_ACCELERATE_DM_splunk_app_aws_S3_Access_Log_ACCELERATE_ splunk_app_aws nobody 5910
_ACCELERATE_DM_splunk_app_db_connect_DB_Health_ACCELERATE_ splunk_app_db_connect nobody 5910

c. THP was enabled.
 

Resolution

1. The first thing to increase is the open files and max user processes.

open files (-n) 1024 (too low should be 64000 minimum)
max user processes (-u) 1024 (too low should be 16000 minimum)

System requirements


2. The Health check shows:

THP Enabled Medium system Fail.

Splunk and THP

THP should be disabled. This is a prime cause of crashes:

28/09/2020
07:17:56.250
09-28-2020 00:17:56.250 -0700 WARN ulimit - This configuration of transparent hugepages is 
known to cause serious runtime problems with Splunk. Typical symptoms include generally 
reduced performance and catastrophic breakdown in system responsiveness under high memory 
pressure. Please fix by setting the values for transparent huge pages to "madvise" or 
preferably "never" via sysctl, kernel boot parameters, or other method recommended by 
your Linux distribution.

.