Splunk Universal Forwarder unable to start on Oracle Linux 7.9 with systemd.

Applicability

Description

After upgrading the Splunk Universal Forwarder hosted on Oracle Linux server 7.9 to forwarder version 9.0.4 and migrating the service from init.d managed service to systemd managed service, there it was observed that the boot start to enable by using systemd is not working and It shows the following message:

systemd[1]: Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'
Failed at step CAPABILITIES spawning /opt/splunkforwarder/bin/splunk: Invalid argument.

Note: Splunk service failed to start when managed by systemd on Oracle Enterprise Linux whereas same is working fine with initd

systemd version:

systemctl --version]# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

System Details :

Operating system : Oracle Linux Server 7.9
Kernel Version : 4.1.12-124.53.5.2.el7uek.x86_64Upgraded from 8.2.1 to 9.0.4

Investigation

The Universal Forwarder boot-start was enabled with init.d, and after the upgrade, as per requirement,  moved to enable boot-start with systemd managed.

1. Ran the below-given command to enable boot-start using systemd

./splunk enable boot-start -systemd-managed 1 -user <user name>

2. Checked the systemctl status SplunkForwarder
> Here are the following message mentioning the status as disabled.

SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Wed 2023-10-18 06:12:13 CDT; 3s ago
Process: 113366 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 113363 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=0/SUCCESS)
Process: 113362 ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=exited, status=218/CAPABILITIES)
Process: 113360 ExecStartPre=/bin/bash -c chown -R splunk:splunk /opt/splunkforwarder (code=exited, status=0/SUCCESS)
Main PID: 113362 (code=exited, status=218/CAPABILITIES)

3)  Confirm that init.d configuration is removed from the UF.

# ls -l /etc/init.d/splunk
ls: cannot access /etc/init.d/splunk: No such file or directory

4)  Run the below command, to confirm the unit file that is being used.

./bin/splunk display boot-start

# ./bin/splunk display boot-start
Checking if init.d script exists:
File is not installed (checked: /etc/init.d/splunk).
Init script is not configured to run at boot.Checking if systemd unit file exists:
Systemd unit file installed by user at /etc/systemd/system/SplunkForwarder.service.
Polkit rules are not configured.
Configured as systemd managed service.

Resolution

Compatibility between the Splunk unit file's AmbientCapabilities, systemd, and kernel versions. The Linux capabilities document mentions the presence of Ambient since Linux 4.3. In contrast, the kernel version used in the above example is kernel:4.1.12-124.66.3.el7uek.x86_64,  does not include this feature.

A systemd version of 229 or above is required (as the AmbientCapabilities= setting was introduced in version 229)

"A new service setting AmbientCapabilities= has been added. It allows configuration of additional Linux process capabilities that are passed to the activated processes. This is only available on very recent kernels."

Steps to resolve the issue:

1) Open the Splunk unit file under etc/systemd/system/SplunkForwarder.service , and replace with the following:

AmbientCapabilities=CAP_DAC_READ_SEARCH to AmbientCapabilities=

2) Configuration after the changes under etc/systemd/system/SplunkForwarder.service

[Service]
AmbientCapabilities=

3. Run the following command before restarting the Universal Forwarder:

systemctl daemon-reload

4) Restart the Splunk Universal Forwarder.

5) Check the status by running the following command:

 systemctl status SplunkForwarder

After the above steps enable boot-start was successfully enabled with systemd managed

Related Links

Refrences:

Splunk Docs - Install a Universal Forwarder

Linux Man Page capabilities

SystemD Github - Line 12224 [AmbientCapabilities=]

Forwarder Least Privileged User - SystemD Unit File AmbientCapabilities Reference