Data sent using HTTP Event Collector is missing or not indexed in Splunk Cloud.

Applicability

Description

When sending data to Splunk Cloud via HEC one of the below behaviors is seen:

• The Splunk returned status code indicates the action wasn't successful (not equal to zero {0} )
• The command hangs.
• An error message is received on the sending side.
• The related index in the token doesn't show the data as ingested.
• No related event in Splunk logs or the reply code indicates an error

Investigation

Relevant information to consider when troubleshooting HEC :

• HEC is enabled by default in Splunk Cloud
• Splunk Cloud HEC default port is 443
• Amazon Web Services (AWS) Kinesis Firehose HEC, isn't enabled by default.
• Indexer acknowledgment is only available for AWS Kinesis Firehose
• HEC uses a token-based authentication model. An active token is therefore necessary. 
• The token is related to an index that must also be enabled.

Original HEC URI used

curl -k https://<stack_name>.splunkcloud.com:8083/services/collector -H
'Authorization: Splunk de21618b-3c3f-4e0e-b127-b61dbd180f96' -d ''

Message Received on the sending side:

<!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><meta http-equiv="refresh" content="1;
url=https://<...>.splunkcloud.com/en-US/services/collector/event"><title>303 See Other</title></head><body><h1>See Other</h1><p>The resource has moved 
temporarily <a href="https://<...>.splunkcloud.com/en-US/services/collector/event">here</a>.</p></body></html>;

Event though it is evident there is an error there is no related event recorded on $HOME/splunk/var/log/splunk/splunkd.log , impling no communication between the sender and the stack. 
Tested using another cURL command and received the following return message:

curl -k https://http-inputs-<stack_name>.splunkcloud.com:443/services/collector/event -H "Authorization: Splunk 374c3d9c-6953-4340-b99d-ff0abe066da" -d '{"event":"Hello World Test"}'{"text":"Invalid token","code":4}

In this case, the original URI was incorrectly formatted, the correct endpoint enabled communication but still, the data was not indexed as there is an error code 4 (Invalid Token)

Resolution

Pre-Checks:

1. Check no firewall/proxy/gateway is blocking traffic.
2. The token and index are enabled.

Check HEC URI format:

Send data using a specific uniform resource identifier (URI) for HEC. The standard form for the HEC URI:

<protocol>://http-inputs-<host>.splunkcloud.com:<port>/<endpoint>

Test connection and response by using a  cURL command: HEC Endpoint and Test HEC  

Reviewed and cthe original orrected cURL command. Found the following: the required prefix before the stack name was missing, corrected port to cloud default, used the correct token.

curl -k https://http-inputs-<stack>.splunkcloud.com:443/services/collector/event -H "Authorization: Splunk de21618b-3c3f-4e0e-b127-b61dbd180f96 " 
-d '{"event":"Hello world!"}'{"text":"Success","code":0}

Reviewing $HOME/splunk/var/log/splunk/splunkd.log component HttpInputDataHandler. If the reply is not 0 there was an error processing the HEC data. Most reply codes will also indicate that the data was not indexed.
SPL: 

index=_internal sourcetype=splunkd component= HttpInputDataHandler

ERROR HttpInputDataHandler [1308 HttpDedicatedIoThread-0] - Failed processing http input, token name=n/a, channel=n/a, source_IP=xx.xx.xx.xx, reply=0, events_processed=1, http_input_body_size=28, parsing_err=""

Success Confirmation:

Once there is a successful reply HEC metrics data to the _introspection index.  To search the accumulated HEC metrics 

index="_introspection" token-name

Related Links

Use the HTTP Event Collector

 HEC Endpoint and Test HEC