Universal Forwarder is able to ingest files that it does not have permission to read

Applicability

Description

Installing Splunk Universal Forwarder version 9+ automatically creates a least-priviledged user. The least-privileged user possesses the CAP_DAC_READ_SEARCH capability which is applied through the systemd unit file AmbientCapabilities.

The capability CAP_DAC_READ_SEARCH allows the service to bypass file read and directory search permission checks. Essentially this enables the service to read files or search directories regardless of permission settings.

Investigation

As the capability (CAP_DAC_READ_SEARCH) is granted only to the process created by systemd when the service starts, it does not change the permissions of the Splunk user account outside of the process started by systemd.

For example when using "sudo -u splunkfwd tail /var/log/path/log_name.log" in the CLI the capability CAP_DAC_READ_SEARCH does not apply and if the splunkfwd user does not have permission to read the file the following error will be displayed:

tail: cannot open 'log_name.log' for reading: Permission denied

To check the systemd unit file, use the cat command to read the file which is located at /etc/systemd/system/SplunkForwarder.service by default.

In the example below the systemd unit file contains AmbientCapabilities=CAP_DAC_READ_SEARCH

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network-online.target
Wants=network-online.target[Service]
Type=simple
Restart=always
ExecStart=/opt/splunkforwarder/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
LimitRTPRIO=99
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunkfwd
Group=splunkfwd
NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH
ExecStartPre=-/bin/bash -c "chown -R splunkfwd:splunkfwd /opt/splunkforwarder"
Delegate=true
CPUShares=1024
MemoryLimit=<memory limit>
PermissionsStartOnly=true
ExecStartPost=-/bin/bash -c "chown -R splunkfwd:splunkfwd /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=-/bin/bash -c "chown -R splunkfwd:splunkfwd /sys/fs/cgroup/memory/system.slice/%n"[Install]
WantedBy=multi-user.target

Resolution

It is expected that the splunkfwd user has access to read all files on the system to allow file ingestion, if however the permission needs to be revoked you can override the AmbientCapabilities from the systemd unit file.

The unit file override can created by completing the following steps:

1. Stop Splunk

$SPLUNK_HOME/bin/splunk stop

2. Enable boot-start with systemd (if not done already)

 $SPLUNK_HOME/bin/splunk enable boot-start -user root -systemd-managed 1

3. Create the systemd unit file override file

sudo systemctl edit SplunkForwarder.service

4. When you run the above it create the override file and open it in the text editor vi.
5. In the vi editor add the following lines to the file, then write and quit the file 

[Service]
AmbientCapabilities=

6. Reload the daemon

sudo systemctl daemon-reload

7. Start Splunk

sudo systemctl start SplunkForwarder.service

8. Confirm Splunk is running

sudo systemctl status SplunkForwarder.service

By running the systemctl edit command it will create the override file in /etc/systemd/system/SplunkForwarder.service.d/override.conf which can be edited further as required. If any changes are made reload the daemon as per the above.

Related Links

Manage a Linux least-privileged user