Splunk Application server is not running

Applicability

Description

Start Splunk after the new certificate installation but observed the PID error.
As this error was observed after certificate installation, rebooted the server which helps the server to go up and run Splunk. But, that was a temporary solution.
The issue continued. Splunk was down and not running.

Investigation

Check for AntiVirus.

Check if THP (Transparent Huge Pages) is enabled. 

Resolution

The issue was resolved after following the below troubleshooting steps provided by Splunk support. They are:-

• Disable and stop Antivirus software for time being (at least 15 days) from that oracle 6 Linux instance.
• Reboot the server.
• Check whether the antivirus software is running or not. It should not be running this time.

Check Splunk service status.

$SPLUNK_HOME/bin/splunk status

If running then stop it.

$SPLUNK_HOME/bin/splunk stop

Change the permission to Splunk user (Splunk user should have a sudo access permission)

chown -R splunk:splunk /opt/splunk

Execute the below commands one by one:

$SPLUNK_HOME/bin/splunk disable boot-start (if the command throws any error please ignore this step)
$SPLUNK_HOME/bin/splunk enable boot-start -user splunk

Switch to Splunk user and follow the below command.

sudo su - splunk
$SPLUNK_HOME/bin/splunk start
$SPLUNK_HOME/bin/splunk status

Run the below command

sudo netstat -tunlp | grep -i "splunk"

 It should show the output result something like below:

tcp 0 0 x.x.x.x:8089 x.x.x.x:* LISTEN 741/splunkd
tcp 0 0 x.x.x.x:8000 x.x.x.x:* LISTEN 741/splunkd

Run the command

top -c

here the splunk process will be running from the "splunk" user itself as per below format:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
741 splunk 20 0 462148 100128 23276 S 0.7 1.3 0:08.53 splunkd

Check for minimum required Linux kernel version so please run the below command and validate the linux kernel version. Run this from the root user.

uname -a

 The document Workloads Requirements for reference

 
THP needs to be disabled. 

WARN  ulimit - This configuration of transparent hugepages is known to cause serious runtime problems with Splunk. 
Typical symptoms include generally reduced performance and catastrophic breakdown in system responsiveness under high memory pressure. 
Please fix by setting the values for transparent huge pages to "madvise" or preferably 
"never" via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.

 See the Disable Transparent Huge Pages  document link for reference:

 
Linux Ulimit is very less "4096" on this system. As per Splunk recommendation it needs at least 65534 so please set this value in linux accordingly.
how-can-i-increase-open-files-limit-for-all-processes
Considerations_regarding_system-wide_resource_limits_on_.2Anix_systems



 

Related Links

Workloads/Requirements
How-do-I-disable-Transparent-Huge-Pages
how-can-i-increase-open-files-limit-for-all-processes
Systemrequirements
https://splunkbase.splunk.com/app/4856/