Splunk Universal is not automatically restarting, getting splunkd is not running. [FAILED] error

Available Languages

Updated:September 29, 2026
Document ID:SPLK046381

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Universal Forwarder Splunk Enterprise Linux

Description

Upgrading the Splunk Univeral Forwarder (UF) agent client in Linux systems from 8.1.1 to 9.0.2, followed the steps detailed by the UF upgrade article

$SPLUNK_HOME/bin/splunk stop

The UF package was directly installed over the existing deployment, then the service was started:

$SPLUNK_HOME/bin/splunk start

The forwarder displays the following: This appears to be an upgrade of Splunk.

$SPLUNK_HOME/bin/splunk start --accept-license --answer-yes --no-prompt --seed-passwd --gen-and-print-passwd

The process should finish and Splunk service should be running. Instead, the following message appeared.

[root@sc6ingl6515 ~]# /opt/splunkforwarder/bin/splunk restart
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R svc-splunk /opt/splunkforwarder"
splunkd is not running. [FAILED]
splunkd.pid doesn't exist...Splunk> Needle. Haystack. Found.

 

Investigation

Splunk Enterprise 7.2.2 and later adds broad support for Linux systemd, with an updated enable boot-start command that lets you automatically configure systemd to manage splunkd as a service.
Among other things, systemd starts splunkd at boot. and monitor and manage splunkd service during runtime.

In the /etc/systemd/system directory systemd creates a unit file (text file) using naming syntaxis  <SPLUNK_SERVER_NAME>.service, such as Splunkd.service.  
This unit file replaces the traditional start-up script for systems configuration, and is used when enabling boot-start for Splunk on systemd-based Linux distributions.

Configure Systemd

The issue was caused by some configuration on the particular instance and was not related to the upgrade process.  While replicating the issue, noticed during the restart the following:

loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: enabled
on the failed restart it showed:
loaded (/etc/systemd/system/SplunkForwarder.service; enabled; vendor preset: disabled


The documentation indicates:
Do not use the following properties on the unit file (/etc/systemd/system/Splunkd.service) . These properties can cause splunkd to fail on restart.
Properties:

RemainAfterExit=yes
ExecStop


 


Resolution

Reviewing the file /etc/systemd/system/Splunkd.service it showed the mentioned settings.
The following lines were removed from the unit file (/etc/systemd/system/Splunkd.service)  
RemainAfterExit=yes
ExecStop
The issue was solved.
Universal Forwarder upgrade article
Configure Systemd

Revision History

Revision Publish Date Comments
1.0
29-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products

Related Articles