Splunk service cannot start after upgrade

Applicability

Investigation

After Splunk upgrade, the Splunkd.service cannot startup. The output is like:

#systemctl status SplunkForwarder.service● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
   Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/SplunkForwarder.service.d
           └─20-controls.conf
   Active: failed (Result: start-limit) since Fri 2024-08-30 04:24:01 EDT; 33s ago
  Process: 28846 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=1/FAILURE)
  Process: 28844 ExecStart=/usr/local/gov/splunk/splunkforwarder/bin/splunk _internal_launch_under_systemd (code=killed, signal=INT)
  Process: 28842 ExecStartPre=/bin/chown -R splovzels:splovzels /sys/fs/cgroup/memory/agent.slice/%n (code=exited, status=0/SUCCESS)
  Process: 28840 ExecStartPre=/bin/chown -R splovzels:splovzels /sys/fs/cgroup/cpu/agent.slice/%n (code=exited, status=0/SUCCESS)

Comparing these two lines:

  Process: 28846 ExecStartPost=/bin/bash -c chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n (code=exited, status=1/FAILURE)
  Process: 28840 ExecStartPre=/bin/chown -R splovzels:splovzels /sys/fs/cgroup/cpu/agent.slice/%n (code=exited, status=0/SUCCESS)

It seems the failure is due to the cgroup path. The one with agent.slice is OK. the one with system.slice failed.

Further check the previous systemctl status, the output was

systemctl status SplunkForwarder.service● SplunkForwarder.service - Systemd service file for Splunk, generated by 'splunk enable boot-start'
   Loaded: loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/SplunkForwarder.service.d
           └─20-controls.conf
   Active: active (running) since Fri 2024-08-16 22:55:14 EDT; 1 weeks 6 days ago
Main PID: 23788 (splunkd)
   CGroup: /agent.slice/SplunkForwarder.service
           ├─23788 splunkd --under-systemd --systemd-delegate=yes -p 48089 _internal_launch_under_systemd
           └─23878 [splunkd pid=23788] splunkd --under-systemd --systemd-delegate=yes -p 48089 _internal_launch_under_systemd [process-runner]

 This confirms the service was using agent.slice as cgroup path. When the Splunkd.service file includes these 2 lines:

ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

The Splunkd service will fail to startup.

When Splunk upgrade from 8.0 to 8.1 or higher, if Splunk Enterprise version 8.0.x was configured to run as a systemd service, upon upgrade to version 8.1 or higher, Splunk Enterprise adds the following properties to the Splunkd.service unit file:

User
Group
ExecStartPost

When Splunk Enterprise adds these unit file properties, it creates a new unit file that replaces the existing unit file Splunkd.service. It also renames the old unit file Splunkd.service_<timestamp>, which it saves for backup purposes only.

If the above mentioned 3 properties with the incorrect cgroup path are manually put into the Splunkd.service, the service will fail to startup too.

Resolution

Use command systemctl status Splunk.service to check what cgroup path it use. And ensure ExecStartPost will use the according cgroup path. The default settings should be if the service use system.slice:

ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

However, if the service is using agent.slice cgroup path, then the settings should look like this:

ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/agent.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/agent.slice/%n"

After updating the settings in Splunk.service, run this command

systemctl daemon-reloadsystemctl restart SplunkForwarder.service

The service should be able to start.

Related Links

Upgrade considerations for systemd