Unable to Access Enterprise Security (ES) Search Head Web UI Due to Unexpected Connection Closure

Applicability

Description

Users experienced an issue where accessing the ES search head Web UI resulted in the connection being unexpectedly closed. Attempts to access the stack via the Web UI also failed. Backend access to the ES search head was confirmed to be operational, and the Splunk daemon (splunkd) was running normally. The issue was isolated to Web UI access only, with no initial indication of broader system failure

Investigation

The investigation involved:

• Confirming backend accessibility and splunkd service status to isolate the problem to the Web UI layer.
• Monitoring log forwarding from Universal Forwarders (UFs) and Heavy Forwarders (HFs) to Splunk Cloud, as well as API integrations, to check for ingestion impact.
• Reviewing scheduled search performance and skipped search ratios during the outage period.
• Analyzing resource usage on the ES search head, particularly memory consumption and out-of-memory (OOM) events.

Sample SPL query to check skipped searches ratio:

index=_internal =scheduler component=Scheduler skipped=1 | stats count by savedsearch_name  

Findings included:

• No impact on log forwarding or API-based data collection.
• A temporary increase in skipped scheduled searches to approximately 15% during the incident, which resolved without intervention.
• High memory consumption on the ES search head instance around the time of the issue, with evidence of OOM events affecting some processes.

Resolution

The root cause was identified as high memory usage leading to OOM conditions on the ES search head, which caused the Web UI connection to close unexpectedly. No direct cause for the memory spike was determined during the investigation.

Recommended actions:

• Monitor memory usage on the ES search head continuously.
• Set up alerts for high memory usage via the Cluster Manager Console (CMC) or equivalent monitoring dashboards. For example, create an alert that triggers when memory usage exceeds a defined threshold and sends email notifications.
• If a memory spike recurs, escalate immediately by contacting Splunk Support and requesting priority handling (P1) for urgent investigation, including potential heap dump collection for deeper analysis

Related Links

https://help.splunk.com/en/splunk-cloud-platform/administer/admin-manual/9.3.2411/monitor-your-splunk-cloud-platform-deployment/use-the-alerts-dashboard#:~:text=a%20telemetry%20endpoint-,Create,-custom%20alerts