Search Head WEB UI is Down Due to Unknown directory in /opt/splunk/bin/ called Needle

Applicability

Splunk Enterprise

Description

Search head Web UI is not accessible but the CLI is working fine. No changes recently happened to the search head and no application has been installed.

The Search head is not accessible with both HTTP and HTTPS. 

Investigation

While checking the search head diag, found the below error as follows:

W - [initandlisten] Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty.

As per the logs, there is an unclean shutdown happened. Suspect, the splunkd.lock and mongod.lock could be empty.

So follow the below steps to check if the file exists or corrupted. 

1- stop Splunk service
2- find /opt/splunk/var/run/splunk -name *.lock
3- find /opt/splunk/var/run/splunk -name *.pid

If the files exist,  create a new folder in $SPLUNK_HOME named "tmp" and move the .lock and .pid files to the path directory.

$SPLUNK_HOME/tmp

If files is not present, please continue with the next steps.

4- start Splunk service.

Starting the service will create new splunkd.lock and mongod.lock files.

Also, While checking the splunkd_ui_access.log, found many 401 and 301 codes for this particular app
"tmm_security_app". So for the time being, disabled the app "tmm_security_app".

Disable if any antivirus software (AV) is installed on this SH. And also disable SELinux on the Linux server.

Furthermore, check the permission of the Splunk user for the Splunk directory with the below command.

find /opt/splunk \! -user splunk -print

The above command will show the permissions of users with the Splunk directory apart from the user "Splunk". So if you find the permission of the directory is set for a different user, change the permission to the Splunk user and restart the SH for the changes to impact.

 

Resolution

Done with all the above-mentioned workarounds still the UI is not accessible. 

While checking the search head instance,  found there is a directory present over in /opt/splunk/bin/ named "Needle" which should not be there.

So, stopped the Splunk service and moved the Needle directory to $SPLUNK_HOME/tmp, and started Splunk again. After this, the WEB UI is accessible without any issues.

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release