Splunkweb UI not reachable after restart

Applicability

Description

Linux admins had recently patched servers. A restart of each server was completed on impacted Search Head cluster members. Upon restart, Splunkd started on all members. While conducting validation testing, 2 members of the Search Head Cluster were not reachable from the UI. 

Investigation

Troubleshooting:

1. Validate Splunk service is running.

# ./splunk status
splunkd is running (PID: 13775).Splunkd.log shows:08-16-

2. Start Splunk if it's not running or restart if the service is running.

./splunk start
or
./splunk restart

3. If Search Head clustering is in use, check that the Search Head cluster status is in a healthy state.  Usage of the CLI to view information about a search head cluster here.

./splunk show shcluster-status

4. Check port configured for Splunkweb is open and available on the server

nestat -tulpn

In the below example output it's shown several standard ports in use by Splunk such as port 8000, 8089, 8088, and 8065 are LISTENING.

[rootxxxx bin]# netstat -tulpn | grep splunkdtcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      13775/splunkd       tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      13775/splunkd       tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      13775/splunkd       [root@xxxx bin]# netstat -tulpn | grep 8065tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      14358/python3.7  

5. Review splunkd.log for following errors in it.

Splunkd.log shows:
202322:28:12.093+0200ERRORUiHttpListener[5819WebuiStartup] -Noappserverisrunning, stopinitializinghttpserver
08-16-202322:28:12.093+0200ERRORUiPythonFallback[5819WebuiStartup] -Couldn'tstartanyappserverprocesses, UIwillprobablynotfunctioncorrectly!
08-16-202322:28:12.093+0200ERRORUiPythonFallback[5819WebuiStartup] -Couldn'tstartappserverprocessonport8065:Appserverathttp://127.0.0.1:8065neverstartedup.Set`appServerProcessLogStderr`to"true" under[settings] inweb.conf.Restart, trytheoperationagain, andreviewsplunkd.logforanymessagesthatcontain"UiAppServer-Fromappserver".

The above ERROR  is an indicator that appserver never started up on port 8065. Port 8065 shows listed as python3.7 in above screenshot from an Instance with default values where Splunk UI is working as expected. There is a dependency if port 8065 is not avaiable. Splunkweb UI will not start.
 

Resolution

Research into the above ERRORS found similar cases where Splunk App For Stream was binding to port 8065 causing Splunkweb to not start. To resolve this Splunk app from Stream was disabled on all impacted SHC members.

For this case, to accomplish this is needed to set below on all (3) members.

splunk/etc/apps/splunk_app_stream/default/app.conf
[install]
state = disabled

Splunk was restarted. Was validated Splunkweb started and was reachable.