Cannot Ingest Events in Splunk Cloud through HEC from AWS Kinesis Firehose.

Applicability

Description

"If you have a proxy (ELB or other) between Kinesis Firehose and the HEC node, you must enable sticky sessions and not provide any session expiration to support HEC ACKs"

Investigation

How to identify it:

1) You may get an error like the one below in AWS:

{
"deliveryStreamARN": "arn:aws:firehose:ap-southeast-2:406119630047:deliverystream/wagering-ivr-UATYarraBaseline-wivr-logs-kinesis-firehose",
"destination": "https://http-inputs-ack-xxxx.splunkcloud.com/services/collector/raw",
"deliveryStreamVersionId": 1,
"message": "If you have a proxy (ELB or other) between Kinesis Firehose and the HEC node, you must enable sticky sessions and not provide any session 
expiration to support HEC ACKs.",
"errorCode": "Splunk.ProxyWithoutStickySessions",
"processor": "arn:aws:lambda:ap-southeast-2:406119630047:function:paism-log-processor"
}

2) If you try to send a test event to the HEC token using the command below you receive the error

{"text":"Data channel is missing","code":10}

curl -k "https://http-inputs-<stack-id>.splunkcloud.com/services/collector" -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d 
'{"event": "Hello, world!", "sourcetype": "manual"}' 

Resolution

How to fix it:

To fix it disable indexer acknowledgements by going to Settings » Data Inputs » HTTP Event Collector, then click on "edit" on the token with the issue and uncheck "Enable indexer acknowledgement" and click on "Save"

If getting the error {"text":"Data channel is missing","code":10} and not sending the events from AWS Kinesis Firehose, but rather using the curl command and you want to keep the indexer acknowledgements enabled, then just add a channel to the command, for instance:

curl https://http-inputs-<stack-id>.splunkcloud.com/services/collector?channel=FE0ECFAD-13D5-401B-847D-77833BD77131 -H "Authorization: Splunk 
BD274822-96AA-4DA6-90EC-18940FB2414C" -d '{"event": "Hello, world!", "sourcetype": "manual"}' -v

Related Links