HEC Endpoint Requests Fail with Status 404 "The requested URL was not found on this server."

Applicability

Splunk Enterprise

Description

The HTTP Event Collector (HEC) is a method of sending data to a Splunk environment for indexing. A client system which does not require any knowledge of Splunk other than a URL and unique identifier (HEC Token) can send a properly formatted HTTP Request and have it indexed (or parsed and indexed) by the Splunk environment.

Splunk docs provides a troubleshooting guide with information on what HEC logs and what error codes it might return, but does not include HTTP 404 (aka "Page Not Found") as a possible error code.

This article covers a scenario where a client sending data to Splunk over a previously operational HEC connection starts receiving 404 errors. 
 

Investigation

The troubleshooting guide noted in the description provides information on summary metrics that can be used to diagnose a HEC connection. One of these metrics is "num_of_requests_to_incorrect_url". Use this metric to find information on when a Splunk host recorded this kind of failure:

index=_introspection sourcetype=http_event_collector_metrics component=HttpEventCollector 
| timechart sum(data.num_of_requests_to_incorrect_url) by host

Additionally, the client may also be logging the 404 occurrences, which could be referenced directly to determine when the errors occur.

Correlate that with instances of splunkd being stopped or restarted:

index=_internal sourcetype=splunkd "shutting down splunkd" OR "Splunkd starting"

Correlating the two data sets, if you can see that splunkd was shutting down at the time of the 404's, the issue described in this article is present.
 

Resolution

This behaviour is expected in the current design of Splunk. When a shutdown is issued to Splunk, it stops accepting new connections and HEC is locked down to just processing the currently accepted requests. During the period that Splunk is still up but not accepting the connections, the Splunk HTTP server recieves the request and returns a generic 404 error. When the Splunk process is fully stopped, the connection would then timeout until it came back online.

Typically this behaviour would only be observed for clients sending requests very frequently, increasing the chances of hitting the small window between shutdown initiated and completed. This can be mitigated somewhat in Clustered environment with a Load Balancer to manage the traffic, but may still occur as the Load Balancer tries to send a request before the host is considered unhealthy.

There is an Enhancement Request under development to enable Splunk to support Connection Draining. Once implemented, this will allow Splunk to be configured to respond to HEC requests during this window appropriately, so the Load Balancer can redirect the in-flight request, or in the case of a direct connection the client would receive a more appropriate response.

Until then however, the client needs to have logic and retry mechanisms setup to process the 404 response. The specifics of how this might be achieved are outside the scope of this article, but existing logic used to handle the existing known HEC error code HTTP 503 "Server is busy" message could be utilized.
 

Related Links

Splunk Documentation:
Troubleshoot HTTP Event Collector.
https://docs.splunk.com/Documentation/Splunk/latest/Data/TroubleshootHTTPEventCollector
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release