Splunk Not Auto-Starting After OS Upgrade to RHEL 8 or 9

Applicability

Description

The issue is due to upgrading the operating system from CentOS 7 to RHEL 8 and above, the Splunk not automatically starting after system reboot. This occurs even when the init.d scripts are present and boot-start is enabled for Splunk. The problem stems from changes in how services are managed in newer versions of RHEL, which prefer systemd over the traditional init.d system.

Investigation

1. Verify that Splunk init.d scripts were present in /etc/init.d/ and boot-start was enabled using the command

$SPLUNK_HOME/bin/splunk display boot-startChecking if init.d script exists:
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.Checking if systemd unit file exists:
File is not installed (checked: /etc/systemd/system/SplunkForwarder.service and /usr/lib/systemd/system/SplunkForwarder.service).
Polkit rules are not configured.
Not configured as systemd managed service.

2. Check system logs for any errors related to Splunk startup

3. Investigated the OS upgrade path, for example:

CentOS 7 -> RHEL 7 -> RHEL 8 -> RHEL 9.2

4. Examined the output of commands like the below command to confirm systemd usage 

"ps 1" and "file /sbin/init"

If the system use the systemd will show the output as below:
$ ps 1
PID TTY STAT TIME COMMAND
1 ? Ss 40:46 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
$ file /sbin/init
/sbin/init: symbolic link to `../lib/systemd/systemd'

Resolution

1. The RHEL 9 uses systemd as the preferred method for managing services, and init.d scripts are no longer the recommended approach.

2. To resolve the issue, reconfigure Splunk to use systemd: 

a. Stop Splunk if it's running:

$SPLUNK_HOME/bin/splunk stop

b. Disable the current boot-start configuration:

$SPLUNK_HOME/bin/splunk disable boot-start

c. Enable boot-start using systemd: 

sudo $SPLUNK_HOME/bin/splunk enable boot-start -user splunk -systemd-managed 1

d. Start Splunk:

sudo systemctl start Splunkd

3. Verify that Splunk starts automatically after a system reboot.

sudo systemctl status Splunkd

4. Update any relevant documentation or procedures to reflect the use of systemd for managing Splunk on RHEL 8 and 9 systems.

Note: It's recommended to test this change in a non-production environment before applying it to critical systems. Always ensure you have a backup of your Splunk installation and configuration before making changes.

As per the Splunk docs “The init.d boot-start script is not compatible with RHEL 8 and higher.”

Related Links

Additional_options_for_enable_boot-start v9.1.2
Additional_options_for_enable_boot-start_latest