C++ exception: exception_addr=0x7f196743ad40 typeinfo=0x5573bbda60f8, name=18EventLoopException what(): CMHeartbeatThread: about to throw an EventLoopException: error from epoll_create: Too many open files /etc/redhat-release: CentOS Linux release 7.5.1804 (Core)
Confirm that all instances where using same CentOS 7.X.
Based on the description is pointing that the open files descriptor ULIMIT was too low but the limits.conf file is set to 65535
It was discovered that RHEL/CentOS 7.X version do not honor open files ulimit with Splunk
First identify how the Splunk Process is being started on your RHEL/CentOS 7 Server. The following are the steps to identify how the Splunk Service is being started (init.d or systemctl):
Identify if systemctl or service is in use:
service splunk status or systemctl status splunk.service
Based on the Printout provided by these commands depends on what Script or Configuration file needs to be modified to resolve this type of issue. The following is a printout from a CentOS 7 Server that is using systemctl and the "OId" init.d Script:
]$ systemctl status splunk.service
● splunk.service - SYSV: Splunk indexer service
Loaded: loaded (/etc/rc.d/init.d/splunk; bad; vendor preset: disabled)
Active: active (exited) since Thu 2017-10-26 07:46:59 CDT; 2 weeks 6 days ago
Docs: man:systemd-sysv-generator(8)
As this printout is showing the use of the init.d Splunk Script the modification needed would be to: /etc/init.d/splunk and the modification would be to add the specific ulimits to the Startup Script. The following would be adding the NoFile (Number of File Descriptors) ulimit to the Splunk Startup Script. These would be placed inside of the "Starting" Function of the script:
ulimit -Hn 65000 ulimit -Sn 65000
The above ulimit values would need to be added to the Start Function like so:
splunk_start() {
echo Starting Splunk...
ulimit -Hn 65000
ulimit -Sn 6500
This will force the Linux Kernel to apply the ulimit settings for the process as its starting.
Now if you are running RHEL/CentOS 7 and using a systemD unit file the modification is very similar. To identify where your systemD unit file is located for the Splunk Process you would issue the following command set:
systemctl status splunk.service
This will provide a printout like the one posted previously in this Post. It will provide the location of the unit file being used to load the Splunk Process. This is usually located in: /etc/systemd/system -- but could be located in a couple of different locations based on your installation and configuration of the OS. The following is an example of the unit file for Splunk on a systemD Server:
]$ systemctl status splunk.service ● splunk.service - SYSV: Splunk indexer service Loaded: loaded (/etc/systemd/system/splunk.service; bad; vendor preset: disabled) Active: active (exited) since Thu 2017-10-26 07:46:59 CDT; 2 weeks 6 days ago
The following is an example of a systemD unit file prior to the ulimit modifications:
[Unit] Description=Splunk indexer service Wants=network.target After=network.target Requires=thp-disable.service[Service] Type=forking Restart=on-failure User=splunk Group=splunk ExecStart=/opt/splunk/bin/splunk start ExecStop=/opt/splunk/bin/splunk stop ExecReload=/opt/splunk/bin/splunk restart StandardOutput=syslog TimeoutSec=300 PIDFile=/opt/splunk/var/run/splunk/splunkd.pid[Install] WantedBy=multi-user.target To correct the issue with ulimits not being honored for a Server that is using systemD you would want to add the following configuration options to the systemD unit file:LimitNOFILE="NUMBER" LimitNPROC="NUMBER" LimitDATA="NUMBER" LimitFSIZE="NUMBER"
After making the needed modification to allow ulimits to be honored you would have a file that looks like the following:
[Unit] Description=Splunk indexer service Wants=network.target After=network.target Requires=thp-disable.service[Service] Type=forking Restart=on-failure User=splunk Group=splunk ExecStart=/opt/splunk/bin/splunk start ExecStop=/opt/splunk/bin/splunk stop ExecReload=/opt/splunk/bin/splunk restart StandardOutput=syslog LimitNOFILE=65535 LimitNPROC=16384 LimitDATA=85256585 LimitFSIZE=632524157485 TimeoutSec=300 PIDFile=/opt/splunk/var/run/splunk/splunkd.pid[Install] WantedBy=multi-user.target
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
30-Sep-2026
|
Initial Release |