Excessive Event Duplication with Fluent Bit HEC Ingestion on a Heavy Forwarder (HF)

Available Languages

Updated:October 1, 2026
Document ID:SPLK054903

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Cloud Splunk Enterprise Ingestion HEC Heavy Forwarder Fluent Bit

Description

The problem presents itself as a significant, unexpected volume spike of identical events within a target index. While the event content (_raw) and the original generation timestamps (_time) remain completely indistinguishable across copies, their indexing times (_indextime) vary drastically, sometimes arriving in clusters minutes or hours apart.
 
This symptom indicates that the upstream log collector is experiencing silent data transmission loops. When data is transmitted via the HTTP Event Collector (HEC) and Indexer Acknowledgement is disabled, Splunk writes incoming packets to disk instantly. However, if network latency or intermediate bottlenecks delay the "HTTP 200 OK" validation signal back to the client, the sender assumes a connection failure, retains the data block in its internal buffers, and schedules an automated transmission retry.
 

Investigation

To diagnose if the duplication stems from internal Splunk processing errors or distinct external network transactions, use the internal physical disk location identifier (_cd). Every incoming event block is committed to a unique structural position on the indexer disk. Run the following search to verify the disk footprint:

index=your_target_index
| stats count, values(_indextime) as Ingest_Times, values(splunk_server) as Indexers by _cd
| where count > 1

If this search returns zero results while the duplicate events physically exist in the index with differing _indextime markers, it confirms that Splunk did not double-process a single transmission. Instead, the indexing layer successfully received multiple distinct HTTP POST operations.
To map the exact timing behavior of the upstream scheduler and calculate the precise transmission gaps (in seconds) between each duplicate payload, run the following optimized transaction search. Ensure the time range picker is explicitly configured to include the historical timeframe of the affected logs:

index=your_target_index "your_exact_event_unique_string_or_timestamp"
| eval index_epoch=_indextime
| sort 0 index_epoch
| streamstats current=f last(index_epoch) as previous_index_epoch
| eval gap_seconds = index_epoch - previous_index_epoch
| eval index_time=strftime(index_epoch,"%Y-%m-%d %H:%M:%S")
| table index_time gap_seconds _raw

Analyzing the gap_seconds column provides the definitive diagnostic signature:
  • Sequential 1-Second Increments (57s, 58s, 59s...): Confirms a serialized queue timeout sequence where Fluent Bit attempts consecutive retries, with the processing clock ticking forward as it waits out consecutive network time limits.
  • Abrupt Multi-Minute Jumps (631s): Signals that the transmission failure threshold has triggered the internal backoff limits, pausing the agent pipeline before executing an automated flush of the backlog buffer.


Resolution

1. Client-Side Fluent Bit Modifications
Modify the output plugin configurations in the Fluent Bit infrastructure to extend transmission windows and handle network handshakes properly:
  • Increase Network Timeouts: Update net.io_timeout to 30 or 60 seconds (up from the default 10 seconds) within the output section to allow late-arriving HTTP confirmations to return before dropping the connection.
  • Impose Retry Limits: Define an explicit Retry_Limit value instead of allowing unlimited retries (False), preventing failed elements from looping infinitely.
  • Verify Storage Persistence: Review storage.type. If configured to disk, inspect tracking files for corruption or unexpected agent reboots, which can cause the backlog manager to re-read and replay historical data chunks into the pipeline.
2. Splunk-Side Architectural Alternative
Users can evaluate enabling HTTP Event Collector (HEC) Indexer Acknowledgment on the target HEC token. When active, Splunk issues a unique transaction ID (ackID) and delays confirming a successful transaction until the event is completely written and replicated across the indexing cluster.
 
Note: Splunk Cloud Platform inherently restricts standard client-driven HEC Indexer Acknowledgment to specific native cloud structures, such as AWS Kinesis Data Firehose pipelines. Users must review this architectural alternative internally with an Enterprise Architect to confirm if deploying a dedicated cloud gateway stream like AWS Firehose is a preferred long-term design choice, or if remediation should rely purely on fixing client-side timeouts.
 

Fluent Bit Manual: Networking Administration

Fluent Bit Manual: Scheduling and Retries Engine

Fluent Bit Manual: Buffering & Storage Mechanics

Revision History

Revision Publish Date Comments
1.0
01-Oct-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products