Unable to index data via HEC, getting error code 400, reply = 5.

Applicability

Description

Data is not being ingested via HEC. The source application is receiving error code 400 and no data is being ingested into the expected index.

This was already reported with Pindrop Application and Google Cloud Platform.
 

Investigation

Verify the Splunk internal logs, and the HttpInputDataHandler component to verify what errors are being returned for a specific token:

index=_internal host=<indexers> component=HttpInputDataHandler "<token_name>"

Messages like the one below might be returned:

ERROR HttpInputDataHandler [3929397 HttpDedicatedIoThread-1] - Failed processing http input, token name=<token_name>, channel=n/a, source_IP=<Source_IP>, reply=5, status_message="No data", status=400, events_processed=0, http_input_body_size=123, parsing_err="No data"

As per the Troubleshoot HTTP Event Collector > Possible error codes, reply = 5, error code 400, is returned for Bad Request - No data.

The '“No data' error could be tricky to interpret. It could be related to data-sending mechanisms from the source application.

An example, the application could send empty packets to the Splunk HEC endpoint before sending the actual data in order to validate a healthy connection.  Not necessarily it is an issue to get errors regarding 'No data' events. However, this is something that needs to be verified with the source application support to understand how it works.

When Splunk reports the 'No data' error, it means it received no data.

It could also mean the data was sent without the proper commands or format.

One example is when using the /services/collector/event endpoint for JavaScript Object Notation (JSON)-formatted events. If the "event" key is not specified in the command, an HTTP 400, 'No data', error will be returned. More information under Set up and use HTTP Event Collector in Splunk Web > JSON request and response.

The services/collector/raw endpoint can be used for raw events. This endpoint can also be used to validate that the source can send data via HEC, and the issue is on how the data is being sent in the command, or source application, while using the /services/collector/event endpoint. 

Resolution

Validate that the source application can reach the HEC endpoint, and that there are no firewall restrictions. More information regarding HEC Endpoints under Send data to HTTP Event Collector on Splunk Cloud Platform.

Note that the endpoint for Amazon Kinesis Firehose in Splunk Cloud is different and needs to be enabled, more information under Configure the Amazon Kinesis Firehose on a paid Splunk Cloud deployment.

Perform a test using the token in question - Example of sending data to HEC with an HTTP request. 

Try to ingest the same data using the raw endpoint,  services/collector/raw.

Validate that the format of the JSON event being sent is correct.

The following test would fail with an HTTP 400 error, reply = 5:

curl https://http-inputs.<SplunkCloud_stack>.splunkcloud.com/services/collector/event?token=<token value> -d '{"sourcetype": "test", "data": "testing data"}'

The following command successfully ingests the test data into Splunk, because the "event" key is specified in the command:

curl https://http-inputs.<SplunkCloud_stack>.splunkcloud.com/services/collector/event?token=<token_valye> -d '{"sourcetype": "test", "event": "testing data"}'

With that, verify how the event is formatted when sent to Splunk via HEC using the /services/collector/event endpoint.

Related Links

Troubleshoot HTTP Event Collector > Possible error codes

Send data to HTTP Event Collector on Splunk Cloud Platform

Example of sending data to HEC with an HTTP request