“I have set my ulimits in: /etc/security/limits.conf for the Splunk User but there is a Warning about ulimits being set to 4096 for User Processes….”
The following can sometimes also apply for Linux ulimit settings:
“I am having a problem with Splunk Crashing every XXX minutes/hours.”
There are other ways ulimit related issues are addressed and it would be considered "Best Practice" to eliminate Linux ulimits right off hand. The following provides the steps.
Step-by-step guide - Investigating Ulimits
1. First find out is what's Linux Distro being used for this Splunk installation.
uname -a
This command will output version information. Example
CENTOS6 Linux centos-idx-1.localhost.sv.splunk.com 2.6.32-754.2.1.el6.x86_64 #1 SMP Fri Jul 13 12:50:12 UTC 2018 x86_64 x86_64 x86_64 GNU/LinuxCENTOS7 Linux localhost.sv.splunk.com 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
From the output, check the ELX output (RHEL) to find the version. The above example shows el6 or el7 output.
2. With the version information, and if using SysV or SystemD, these will determine our next steps.
** NOTE: It is more likely to find SystemD, therefore, start the test with systemctl call **
systemctl status splunk
Example:
[myuser@localhost ~]$ sudo systemctl status splunk.service [sudo] password for myuser: ? splunk.service - Splunk indexer service Loaded: loaded (/etc/systemd/system/splunk.service; disabled; vendor preset: disabled) Active: active (running) since Wed 2018-08-22 14:14:24 CDT; 1 day 18h ago Process: 7990 ExecStop=/opt/splunk/bin/splunk stop (code=exited, status=0/SUCCESS) Process: 8709 ExecStart=/opt/splunk/bin/splunk start (code=exited, status=0/SUCCESS) Main PID: 8752 (splunkd) CGroup: /system.slice/splunk.service ?? 8752 splunkd -p 8089 start ?? 8757 [splunkd pid=8752] splunkd -p 8089 start [process-runner] ?? 8893 python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_consumer.py ?? 8975 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=1... ?? 9523 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore ??20733 python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_backup_restore.py ??22857 python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_actions_queue_consumer.py ??22859 python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_actions_queue_consumer.py ??22891 python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_actions_queue_consumer.py ??25562 python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_user_access_init.py ??26355 [splunkd pid=8752] [search-launcher] ??26356 [splunkd pid=8752] [search-launcher] [process-runner] ??26567 [splunkd pid=8752] [search-launcher] ??26569 [splunkd pid=8752] [search-launcher] [process-runner] ??26575 [splunkd pid=8752] [search-launcher] ??26594 [splunkd pid=8752] [search-launcher] [process-runner] ??27006 [splunkd pid=8752] [search-launcher] ??27011 [splunkd pid=8752] [search-launcher] [process-runner]Aug 22 14:14:12 jthompsonlin02.sv.splunk.com splunk[8709]: All installed files intact. Aug 22 14:14:12 jthompsonlin02.sv.splunk.com splunk[8709]: Done Aug 22 14:14:12 jthompsonlin02.sv.splunk.com splunk[8709]: All preliminary checks passed. Aug 22 14:14:12 jthompsonlin02.sv.splunk.com splunk[8709]: Starting splunk server daemon (splunkd)... Aug 22 14:14:12 jthompsonlin02.sv.splunk.com splunk[8709]: Done Aug 22 14:14:24 jthompsonlin02.sv.splunk.com splunk[8709]: Waiting for web server at http://127.0.0.1:8000 to be available....... Done Aug 22 14:14:24 jthompsonlin02.sv.splunk.com splunk[8709]: If you get stuck, we're here to help. Aug 22 14:14:24 jthompsonlin02.sv.splunk.com splunk[8709]: Look for answers here: http://docs.splunk.com Aug 22 14:14:24 jthompsonlin02.sv.splunk.com splunk[8709]: The Splunk web interface is at http://jthompsonlin02.sv.splunk.com:8000 Aug 22 14:14:24 jthompsonlin02.sv.splunk.com systemd[1]: Started Splunk indexer service. Hint: Some lines were ellipsized, use -l to show in full. [myuser@localhost ~]$
If the first command fails then more than likely it is a SysV. Test with the "service command"
service splunk status
If this is a SysV (System5) Server then you will be working with an init.d Script located: /etc/init.d/splunk
If this is a SystemD Server then identify whether the server uses a SysV Startup Script (init.d/splunk) or a SystemD unit file .
To identify this run the following command:
systemctl status splunk | grep -i loaded
This will provide you with the location of the SystemD unit file that was used. If this lists: /etc/init.d/splunk —> You are working with a SystemD Server that has the Old Service Call available and as such able to use the Older SysV init.d Script. If this lists: /etc/systemd/system/splunk.service OR /usr/lib/systemd/system/splunk.service —> you are working with a SystemD unit file.
3. Next identify the ulimit issue that was reported and update the configuration to make the ulimit changes work
Identifying the parent process:
ps auxf | grep splunk | grep start | grep -v "pid=" | awk '{print$2}'
Example output
[myuser@localhost ~]# ps auxf | grep splunk | grep start | grep -v "pid=" | awk '{print$2}'
1854
4. Now that you have the PID, output the actual ulimits that Splunk is using.
** Replace <PID> with the PID number given from step 3. **
cat /proc/<PID>/limits
Example:
[myuser@localhost ~]# cat /proc/1854/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15211 15211 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15211 15211 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us [root@centos-idx-1 ~]#
In this example output, review the ulimits Splunk is starting with, and check to check if limited.
Compare the output from the above step, with the ulimits from the limits.conf file by running
cat /etc/security/limits.conf
Example
cat /etc/security/limits.conf:#<domain> <type> <item> <value> # #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4# Setting Splunk User ulimits: splunk hard nproc 64000 splunk hard nofile 64000# End of file
5. If there is a discrepancy, Lets check nproc.conf file:
Location: /etc/security/limits.d/
Filenames:
RHEL/CentOS 6: 90-nproc.conf
RHEL/CentOS 7: 20-nproc.conf
cat /etc/security/limits.d/<NPROC FILE NAME>
Example
[myuser@localhost ~]# cat /etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning.* soft nproc 4096 root soft nproc unlimited
** It is not recommended to change the nproc.conf file for security reasons. Please follow the steps below **
From this example, the soft file open ulimit is 4096, and this is the root cause in our example of why the limits are different between the limits.conf file and the process ulimit.
Step-by-step guide - Updating Ulimits
To fix the difference, modify the Splunk startup process to set the needed ulimits using hard coded limits in either the init.d Script or the Unit File.
** AWS Linux AMI edit the /etc/init.d/splunk unit file **
cp /etc/init.d/splunk ~/splunk-init.d-BAK
ulimit -Hn 64000 ulimit -Hu 64000These will both get added to the 'start' & 'restart' functions inside of the init.d Script. First, open the splunk init.d script using your preferred editor.
splunk_start() {
echo Starting Splunk...
"/opt/splunk/bin/splunk" start --no-prompt --answer-yes
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_start() {
echo starting Splunk...
ulimit -Hn 64000
ulimit -Hu 16000
"/opt/splunk/bin/splunk" start
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_restart() {
echo Restarting Splunk...
ulimit -Hn 64000
ulimit -Hu 16000
"/opt/splunk/bin/splunk" restart
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
cp ~/splunk-init.d-BAK /etc/init.d/splunk
systemctl daemon-reload
'systemctl restart splunk' (for RHEL/CentOS 7) or 'service restart splunk' (for RHEL/CentOS 6/7) or '/etc/init.d/splunk start' or '/etc/init.d/splunk restart'
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |