Search Head Is Crashing with "Resource temporarily unavailable" Error

Applicability

Splunk Enterprise

Description

These are the errors seen:
 

ERROR SearchProcessRunner - preforked process=0/4323 died on exception: Main Thread: 
about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 
4 threads activeWARN  Thread - DistributedPeerMonitorThread: about to throw a ThreadException: 
pthread_create: Resource temporarily unavailable; 101 threads active

Investigation

1. crash log is generated for each crash, however it is related to different threads

crash-2020-09-04-02:23:04.log: Crashing thread: searchOrchestrator
crash-2020-09-04-02:25:02.log: Crashing thread: searchOrchestrator
crash-2020-09-04-02:28:03.log: Crashing thread: searchOrchestrator
crash-2020-09-04-02:29:03.log: Crashing thread: searchOrchestrator
crash-2020-09-04-02:29:04.log: Crashing thread: ReplicationDataReceiverThread
crash-2020-09-04-20:18:57.log: Crashing thread: TcpChannelThread
crash-2020-09-05-04:39:09.log: Crashing thread: CallbackRunnerThread
crash-2020-09-05-20:03:21.log: Crashing thread: TcpChannelThread
crash-2020-09-06-05:49:33.log: Crashing thread: searchOrchestrator
crash-2020-09-06-05:49:53.log: Crashing thread: BundleReplicatorThread
crash-2020-09-06-20:08:05.log: Crashing thread: TcpChannelThread

Crash.log as shown below:

[build a6fe1ee8894b] 2022-12-05 01:20:08
Received fatal signal 6 (Aborted) on PID 3290445.
 Cause:
   Signal sent by PID 3290445 running under UID 1020.
 Crashing thread: ReplicationDataReceiverThread
 Registers:
    RIP:  [0x00007FE53DBD170F] gsignal + 271 (libc.so.6 + 0x3770F)
    RDI:  [0x0000000000000002]
    RSI:  [0x00007FE5147FE0A0]
    RBP:  [0x0000000000000000]
    RSP:  [0x00007FE5147FE0A0]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x00007FE53DBD170F]
    RDX:  [0x0000000000000000]
    R8:  [0x0000000000000000]
    R9:  [0x00007FE5147FE0A0]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000246]
    R12:  [0x00007FE515634900]
    R13:  [0xFFFFFFFFFFFFFFFF]
    R14:  [0x00007FE515655280]
    R15:  [0x0000000000000000]
    EFL:  [0x0000000000000246]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x002B000000000033]
    OLDMASK:  [0x0000000000000000] OS: Linux
 Arch: x86-64 Backtrace (PIC build):
  [0x00007FE53DBD170F] gsignal + 271 (libc.so.6 + 0x3770F)
  [0x00007FE53DBBBB25] abort + 295 (libc.so.6 + 0x21B25)
  [0x0000559D2D5291B4] ? (splunkd + 0x15751B4)
  [0x0000559D2F909D56] _ZN10__cxxabiv111__terminateEPFvvE + 6 (splunkd + 0x3955D56)
  [0x0000559D2F99BE29] ? (splunkd + 0x39E7E29)
  [0x0000559D2F909764] __gxx_personality_v0 + 612 (splunkd + 0x3955764)
  [0x0000559D2F9ADE2B] ? (splunkd + 0x39F9E2B)
  [0x0000559D2F9AE7E6] ? (splunkd + 0x39FA7E6)
  [0x0000559D2D2B7A63] ? (splunkd + 0x1303A63)
  [0x0000559D2D97EFE7] _ZN23S2SArtifactFileReceiver16connectionClosedEb + 55 (splunkd + 0x19CAFE7)
  [0x0000559D2D95D912] _ZN22ReplicationDataChannel18cleanUpS2SReceiverEv + 338 (splunkd + 0x19A9912)
  [0x0000559D2D96E414] _ZN22ReplicationDataChannelD1Ev + 36 (splunkd + 0x19BA414)
  [0x0000559D2D96E479] _ZN22ReplicationDataChannelD0Ev + 9 (splunkd + 0x19BA479)
  [0x0000559D2E65E0A0] _ZN10TcpChannel9after_eofE18PollableDescriptor + 96 (splunkd + 0x26AA0A0)
  [0x0000559D2E5B074B] _ZN9EventLoop3runEv + 715 (splunkd + 0x25FC74B)
  [0x0000559D2E65E7FC] _ZN19Base_TcpChannelLoop7_do_runEv + 28 (splunkd + 0x26AA7FC)
  [0x0000559D2E65E8FE] _ZN25SubordinateTcpChannelLoop3runEv + 222 (splunkd + 0x26AA8FE)
  [0x0000559D2E66B637] _ZN6Thread8callMainEPv + 135 (splunkd + 0x26B7637)
  [0x00007FE53DF642DE] ? (libpthread.so.0 + 0x82DE)
  [0x00007FE53DC95E83] clone + 67 (libc.so.6 + 0xFBE83)
 Linux / MYUCBRSPSIAPP01 / 4.18.0-348.20.1.el8_5.x86_64 / #1 SMP Tue Mar 8 12:56:54 EST 2022 / x86_64
 C++ exception:
  exception_addr=0x7fe515634900
  typeinfo=0x559d307987e0, name=15ThreadException
  what(): ReplicationDataReceiverThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 206 threads active
 /etc/redhat-release: Red Hat Enterprise Linux release 8.2 (Ootpa)
 glibc version: 2.28
 glibc release: stable

2. Before crashing, there are multiple errors related to system resource
    a.)  From "Resource temporarily unavailable", it is lack of system resources 
    b.)  From "self.pid = os.fork()" and "pthread_create", it is failed to create thread
 

ERROR SearchProcessRunner - preforked process=0/7002 died on exception: Main Thread: about 
to throw a ThreadException: pthread_create: Resource temporarily unavailable; 
3 threads activeERROR ExecProcessor - message from "/opt/splunk/etc/apps/Splunk_TA_nix/bin/bandwidth.sh" 
/opt/splunk/etc/apps/Splunk_TA_nix/bin/common.sh: fork: retry: Resource temporarily 
unavailableERROR ExecProcessor - message from 
"python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_backup_restore.py"     self.pid = os.fork()ERROR SearchProcessRunner - preforked process=0/4323 died on exception: Main Thread: about to
throw a ThreadException: pthread_create: Resource temporarily unavailable; 4 threads activeWARN  Thread - DistributedPeerMonitorThread: about to throw a ThreadException: 
pthread_create: Resource temporarily unavailable; 101 threads active

Resolution

There are multiple possible resolutions for system tuning:

1. Usually it is caused by too low ulimit, check ulimit settings through splunkd.log
    max user processes (ulimit –u)

2. It may be also caused by reaching max PID number, increase the 'pid_max' value to 4194303 (https://www.cyberciti.biz/tips/howto-linux-increase-pid-limits.html)
    /proc/sys/kernel/pid_max

3. Set TasksMax=infinity in /etc/systemd/system/Splunkd.service

Set limits using the /etc/systemd configuration files  
https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ulimitErrors#Set_limits_using_the_.2Fetc.2Fsystemd_configuration_files
 

Related Links

What’s Your ulimit?

How to change ‘pid_max’

TasksMax=infinity

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release