Intermittent missing logs caused by indexing queue saturation and timestamp parsing issues

Applicability

Description

This article explains a situation where log lines were intermittently missing when forwarding data from Universal Forwarders through Heavy Forwarders into Splunk Cloud. The behavior appeared random and was not related to filtering, pipeline limits, or queue size thresholds on the forwarders.

Investigation

After reviewing the environment, it was confirmed that the ingestion pipelines and forwarder configurations were properly set. Settings such as useACK and parallel ingestion pipelines were already enabled and working as expected.

A deeper review of the Splunk Cloud stack showed that the indexing queue was reaching 100 percent consistently over multiple days. This indicated that the environment was receiving more data than it could process in real time.

When the indexing queue stays full for long periods, Splunk starts applying backpressure. This slows down data ingestion and can lead to delays or even skipped events during indexing.

At the same time, multiple data quality issues were identified, especially in sourcetypes such as aws:cloudwatchlogs:dns_firewall and syslog. The main problem was incorrect timestamp parsing, which caused events to be delayed or not indexed correctly.

Additionally, several Universal Forwarders were running versions close to support expiration, which can contribute to inconsistent ingestion behavior.

Resolution

The issue was resolved by addressing the root causes related to ingestion pressure and data quality.

The first step was to reduce the load on the indexing tier by improving data quality. Fixing timestamp parsing ensured that events were processed correctly and avoided delays in indexing.

Guidance for resolving these issues can be found here:

Resolve Data Qualiy Issues

Next, it was confirmed that proper file tracking was in place on the Universal Forwarders to avoid skipped events during file rotation. This included validating settings such as crcSalt and initCrcLength.

More information about file monitoring behavior is available here:
Log file rotation

Finally, upgrading outdated Universal Forwarders was recommended to maintain stability and compatibility with the Splunk Cloud platform:

Upgrade Universal Forwarders

It is very important to check the Cloud Monitoring Console (CMC), please see the guide below:

CMC

Related Links

Resolve Data Qualiy Issues

Log file rotation

Upgrade Universal Forwarders

CMC