1. From splunkd.log, ulimit is set to 64K
Current ulimit
08-24-2021 19:54:58.627 -0700 INFO ulimit [20572 MainThread] - Limit: open files: 64000 files 08-24-2021 19:54:58.627 -0700 INFO ulimit [20572 MainThread] - Limit: user processes: 16000 processes
2.However, from job inspector, search.log is complaining about 'Too many open file'
08-24-2021 02:00:20.491 -0700 ERROR SearchResultsWriter [25269 TcpChannelThread] - Unable to open output file: path=/opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX2FwcF9zdHJlYW0__RMD5bbe0d1bf70bdee86_at_1629795600_76358_2D4BB268-DE7C-4DDA-97B3-56656611DCA8/info.csv.93a7f4d2af34acb10319a770a2425cd8.tmp error=Too many open files in system
3. From metrics.log, usage is just 100+ only.
4. Search still complaining about no enough file descriptor
08-24-2021 02:00:20.491 -0700 ERROR SearchResultsWriter [25269 TcpChannelThread] - Unable to open output file: path=/opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX2FwcF9zdHJlYW0__RMD5bbe0d1bf70bdee86_at_1629795600_76358_2D4BB268-DE7C-4DDA-97B3-56656611DCA8/info.csv.93a7f4d2af34acb10319a770a2425cd8.tmp error=Too many open files in system08-24-2021 02:00:20.491 -0700 ERROR SearchResultsCSVSerializer [25269 TcpChannelThread] - Failed to open file for writing. 08-24-2021 02:00:20.491 -0700 ERROR DispatchProcess [25269 TcpChannelThread] - Failed to write the info file to /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX2FwcF9zdHJlYW0__RMD5bbe0d1bf70bdee86_at_1629795600_76358_2D4BB268-DE7C-4DDA-97B3-56656611DCA8/info.csv.08-24-2021 02:00:20.491 -0700 ERROR DispatchProcess [25269 TcpChannelThread] - sid=scheduler__nobody_c3BsdW5rX2FwcF9zdHJlYW0__RMD5bbe0d1bf70bdee86_at_1629795600_76358_2D4BB268-DE7C-4DDA-97B3-56656611DCA8 user=splunk-system-user failed to start due to err="Error in 'DispatchProcess': Failed to write the info file to /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX2FwcF9zdHJlYW0__RMD5bbe0d1bf70bdee86_at_1629795600_76358_2D4BB268-DE7C-4DDA-97B3-56656611DCA8/info.csv.", releasing quota08-24-2021 02:00:20.491 -0700 ERROR DispatchProcess [25269 TcpChannelThread] - could not create file='/opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX2FwcF9zdHJlYW0__RMD5bbe0d1bf70bdee86_at_1629795600_76358_2D4BB268-DE7C-4DDA-97B3-56656611DCA8/failedtostart' to expedite dispatch directory cleanup (Too many open files in system)
Libc abort message: splunkd: /opt/splunk/src/util/EventLoop.cpp:861: void EventLoop::add(PolledFd*, PollableDescriptor, events_mask_t): Assertion `fd.valid()' failed.As Splunk got 64K file descriptor and search still failed, it may be the kernel level setting which cause this.
ile-max is the maximum File Descriptors (FD) enforced on a kernel level, which cannot be surpassed by all processes without increasing.ulimit is enforced on a process level, which can be less than the file-max.cat /proc/sys/fs/file-max
$ sysctl -w fs.file-max=12586070
/etc/sysctl.conf fs.file-max=12586070
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |