AWS Kinesis Firehose is not sending any logs to Splunk HTTP Event Collector (HEC)

Applicability

Description

The Kinesis Firehose service which is provided by Amazon Web Service (AWS) is configured to send AWS RDS DB logs to Splunk HTTP Event Collector (HEC). However, the logs are not getting pushed to Splunk but the other workloads having a similar configuration are working as expected.

Investigation

Upon investigating and checking the _internal logs, it was found that there are no Error or Warning messages in Splunk.

SPL:

index=_internal HttpInputDataHandler sourcetype=splunkd "*HEC_Token_Name*"

Therefore, checked the _introspection logs using the below SPL to determine if Kinesis Firehose was able to send any logs to Splunk and it was found that there were no metrics logs about data which is been sent by Kinesis Firehose.

SPL:

index=_introspection sourcetype=http_event_collector_metrics "data.token_name"="<HEC_Token_Name"

To isolate the issues on the AWS end, validated if there are any Destination error logs been generated. If Kinesis Firehose fails to send events to HEC, there is CloudWatch Log Group created with the associated Error logs.

To find these logs, navigate to the Log Group found in Firehose Stream and then towards "DestinationDelivery" Log Stream. Added a one of the example below:

Found the below ERROR multiple time in the AWS log_stream:

Indexer acknowledgement is disabled for the HEC token. Enable indexer acknowledgement and try again. For more info, see Enable indexer acknowledgement (http://dev.splunk.com/view/event-collector/SP-CAAAE8X#enable).

The above message indicates that indexer acknowledgment is currently disabled for the HTTP Event Collector (HEC) token in Splunk which was causing the issue.

Resolution

To resolve this issue, need to enable indexer acknowledgement for the associated HEC token form the Data Input page.

Path: Splunk GUI > Settings > Data inputs > HTTP Event Collector > Select HEC token > Select "Enable indexer acknowledgement"

Screenshot:

Related Links

Configuration overview for the Amazon Kinesis Firehose