Splunk Search Head Is Crashing with an "AlertNotifierWorker-0" Error.

Applicability

Splunk Enterprise

Description

Description:
Splunk Search head crashed with an "AlertNotifierWorker-0" error.


 

Investigation

How to identify it:

1. Splunk search head is shutting down with the following crash threads:
[build a0c72a66db66] 2019-02-23 20:52:14
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 24174 running under UID 0.
Crashing thread: AlertNotifierWorker-0

2. In splunkd.log you can see this entry:

“02-23-2019 20:52:14.579 +0700 WARN Thread - AlertNotifierWorker-0: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 32118 threads active”

Resolution

How to fix it:
1. Increase the 'pid_max' value to 4194303 (this value can go up to 2^22 for 64-bit Linux)

• Steps on how to change it can be found here: https://www.cyberciti.biz/tips/howto-linux-increase-pid-limits.html

2.  Verify the following Linux OS settings and their impact with Splunk Enterprise:

• THP (Transparent Huge Pages)
• Linux kernel memory overcommitting
• Ulimit

Related Links

Splunk Documentation
Transparent huge memory pages and Splunk performance:
https://docs.splunk.com/Documentation/Splunk/7.1.2/ReleaseNotes/SplunkandTHP

Linux kernel memory overcommitting and Splunk crashes:
https://docs.splunk.com/Documentation/Splunk/7.1.2/ReleaseNotes/LinuxmemoryovercommittingandSplunkcrashes

What’s Your ulimit?
https://www.splunk.com/blog/2011/11/21/whats-your-ulimit.html

External Documentation
How to change ‘pid_max’
https://www.cyberciti.biz/tips/howto-linux-increase-pid-limits.html
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release