Unable to Ingest Data via HEC Token Over Splunk Cloud

Applicability

Description

When encountering an issue where data is not being ingested via any HTTP Event Collector (HEC) token in Splunk Cloud.

The failure occurs despite using a valid token, preventing events from being sent successfully.  

This article outlines key investigation steps to identify the root cause and provides a resolution to restore HEC data ingestion.

Investigation

Step 1: Verify HEC Enablement
- Navigate to Settings > Data Inputs > HTTP Event Collector > Global Settings.  
- Ensure that HEC is enabled.

Step 2: Check HEC Token Configuration
- Navigate to Settings > Data Inputs > HTTP Event Collector.  
- Confirm that the required HEC token is created and active.

Step 3: Test HEC Token Connectivity
- Use the following CURL command to validate connectivity: 

curl https://hec.example.com:8088/services/collector/event -H "Authorization: Splunk <HEC token>" -d '{"event": "hello world"}'

- If the request fails, it indicates connectivity or authentication issues.  

Step 4: Review Internal Logs for Errors  
- Check for timeout errors using the following SPL query:  

index="_internal" level="ERROR" *timeout*

- If errors related to timeout or connection failures appear, further investigation into network settings is needed.  

Step 5: Check the IP Allow List 
- Navigate to Settings > Server Settings > IP Allow List 
- Ensure that the required Splunk server IPs ( Search Heads, Indexers, Universal Forwarders) are included.  
- If the IP allow list is restricted to a few IPs, HEC requests from unauthorized IPs will be rejected.  

From the above steps, able to pinpoint the issue below as follows:
- The Splunk server IPs were not included in the allow list, leading to HEC connection failures.  

Resolution

To resolve the issue, update the IP Allow List to include the required Splunk server IPs:

Steps to Update the IP Allow List:
1. Navigate to Settings > Server Settings > IP Allow List.  
2. Add the Splunk Search Head (SH), Indexer (IDX), and Universal Forwarder (UF) IPs to the list.  
3. Save the changes and restart the required Splunk services if necessary.  

Once the IP restrictions were lifted, HEC requests were successfully processed, and data ingestion resumed as expected. 

Related Links

JSON request and response of the curl command: here