Splunk v. 7.x cannot be started with default systemd config in Centos/RedHat 8 environment, with systemctl(enable boot-start -systemd-managed 1).
Error:
Job for SplunkForwarder.service failed because the control process exited with error code.
The error is not present in older RHEL versions, RHEL 7, or when Splunk v. 8.x is in use.
Run the following commands:
For Splunk Enterprise
sudo systemctl status Splunkd.service sudo journalctl -xe
For Universal Forwarder
sudo systemctl status SplunkForwarder.service () sudo journalctl -xe
Sample outputs:
sudo systemctl status SplunkForwarder.service
● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2021-04-23 08:05:47 UTC; 5min ago Process: 1614 ExecStartPost=/bin/bash -c chown -R 2024:2024 /sys/fs/cgroup/cpu/init.scope/system.slice/SplunkForwarder.service (code=exited, status=1/FAILURE) Process: 1613 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=killed, signal=TERM) Main PID: 1613 (code=killed, signal=TERM)Apr 23 08:05:46 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. Apr 23 08:05:46 ip-10-202-39-12.ec2.splunkit.io systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Service RestartSec=100ms expired, scheduling restart. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Scheduled restart job, restart counter is at 5. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Start request repeated too quickly. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'.
sudo journalctl -xe
– Unit SplunkForwarder.service has finished shutting down. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Start request repeated too quickly. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: SplunkForwarder.service: Failed with result 'exit-code'. – Subject: Unit failed – Defined-By: systemd – Support: https://access.redhat.com/support – – The unit SplunkForwarder.service has entered the 'failed' state with result 'exit-code'. Apr 23 08:05:47 ip-10-202-39-12.ec2.splunkit.io systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'. – Subject: Unit SplunkForwarder.service has failed – Defined-By: systemd – Support: https://access.redhat.com/support – – Unit SplunkForwarder.service has failed. – – The result is failed. Apr 23 08:10:58 ip-10-202-39-12.ec2.splunkit.io sudo[1625]: splunker : TTY=pts/0 ; PWD=/opt ; USER=root ; COMMAND=/bin/systemctl status SplunkForwarder.service Apr 23 08:10:58 ip-10-202-39-12.ec2.splunkit.io sudo[1625]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice Apr 23 08:10:58 ip-10-202-39-12.ec2.splunkit.io sudo[1625]: pam_unix(sudo:session): session opened for user root by splunker(uid=0) Apr 23 08:10:58 ip-10-202-39-12.ec2.splunkit.io sudo[1625]: pam_unix(sudo:session): session closed for user root Apr 23 08:11:32 ip-10-202-39-12.ec2.splunkit.io systemd[1]: Starting dnf makecache... – Subject: Unit dnf-makecache.service has begun start-up – Defined-By: systemd – Support: https://access.redhat.com/support – – Unit dnf-makecache.service has begun starting up. Apr 23 08:11:32 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: Failed determining last makecache time. Apr 23 08:11:32 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: CentOS Linux 8 - AppStream 27 kB/s | 4.3 kB 00:00 Apr 23 08:11:33 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: CentOS Linux 8 - AppStream 48 MB/s | 6.3 MB 00:00 Apr 23 08:11:34 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: CentOS Linux 8 - BaseOS 69 kB/s | 3.9 kB 00:00 Apr 23 08:11:34 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: CentOS Linux 8 - BaseOS 20 MB/s | 2.3 MB 00:00 Apr 23 08:11:35 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: CentOS Linux 8 - Extras 28 kB/s | 1.5 kB 00:00 Apr 23 08:11:35 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: CentOS Linux 8 - Extras 136 kB/s | 9.6 kB 00:00 Apr 23 08:11:35 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: Extra Packages for Enterprise Linux Modular 8 - 44 kB/s | 16 kB 00:00 Apr 23 08:11:36 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: Extra Packages for Enterprise Linux Modular 8 - 800 kB/s | 557 kB 00:00 Apr 23 08:11:36 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: Extra Packages for Enterprise Linux 8 - x86_64 31 kB/s | 8.5 kB 00:00 Apr 23 08:11:37 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: Extra Packages for Enterprise Linux 8 - x86_64 9.7 MB/s | 9.3 MB 00:00 Apr 23 08:11:41 ip-10-202-39-12.ec2.splunkit.io dnf[1629]: Metadata cache created. Apr 23 08:11:41 ip-10-202-39-12.ec2.splunkit.io systemd[1]: dnf-makecache.service: Succeeded. – Subject: Unit succeeded – Defined-By: systemd – Support: https://access.redhat.com/support – – The unit dnf-makecache.service has successfully entered the 'dead' state. Apr 23 08:11:41 ip-10-202-39-12.ec2.splunkit.io systemd[1]: Started dnf makecache. – Subject: Unit dnf-makecache.service has finished start-up – Defined-By: systemd – Support: https://access.redhat.com/support – – Unit dnf-makecache.service has finished starting up. – – The start-up result is done. Apr 23 08:15:26 ip-10-202-39-12.ec2.splunkit.io sudo[1670]: splunker : TTY=pts/0 ; PWD=/opt ; USER=root ; COMMAND=/bin/journalctl -xe Apr 23 08:15:26 ip-10-202-39-12.ec2.splunkit.io sudo[1670]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice Apr 23 08:15:26 ip-10-202-39-12.ec2.splunkit.io sudo[1670]: pam_unix(sudo:session): session opened for user root by splunker(uid=0)
Steps to reproduce the problem (bash commands):
1. sudo groupadd splunk_nologin 2. sudo useradd l00_splunk_nologin -g splunk_nologin 3. cd /opt 4. sudo wget -O splunkforwarder-x.x.x.x-651b7244cf2-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.3.1.1 &product=universalforwarder&filename=splunkforwarder-7.3.1.1-7651b7244cf2-linux-2.6-x86_64.rpm&wget=true' 5. (set the terminal shell for ‘l00_splunk_nologin’ to /sbin/nologin in /etc/passwd) sudo vi /etc/passwd l00_splunk_nologin:x:2024:2024::/home/l00_splunk_nologin:/sbin/nologin 6. sudo rpm -i splunkforwarder-x.x.x.x-651b7244cf2-linux-2.6-x86_64.rpm 7. sudo chown -R l00_splunk_nologin:splunk /opt/splunkforwarder 8. sudo /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1 -user 'l00_splunk_nologin' --accept-license --answer-yes --no-prompt 9. sudo systemctl start SplunkForwarder.service 10. sudo systemctl status SplunkForwarder.service
/sys/fs/cgroup/cpu/system.slice/%n /sys/fs/cgroup/memory/system.slice/%nFrom:
/sys/fs/cgroup/cpu/init.scope/system.slice/%n /sys/fs/cgroup/memory/init.scope/system.slice/%n </pre>Example ExecStartPost settings in [service] stanza, after applying the change: ExecStartPost=/bin/bash -c "chown -R 2024:2024 /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R 2024:2024 /sys/fs/cgroup/memory/system.slice/%n"
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |