Applicability

Description

When Kubernetes rotates container logs, the active log file (e.g. 0.log) is renamed to a timestamped file (e.g. 0.log.20260302-001), and a new 0.log is created by the container runtime. Users configuring the filelog receiver with a strict pattern such as *.log may wonder whether:

• Rotated files are silently dropped, causing data loss.
• Rotated files are re-ingested from the beginning, causing duplicate log entries.

Two distinct scenarios arise depending on whether the Collector is running at the time of rotation.

Scenario 1 — Rotation whilst the Collector is running: The active log file is renamed mid-stream whilst the Collector is actively reading it.

Scenario 2 — Rotation whilst the Collector is stopped: The log file is rotated and the Collector is restarted after the fact, potentially missing lines written to the rotated file during downtime.

Investigation

To confirm whether a deployment is affected by either scenario, check the following:

Check the include pattern

Inspect the filelog receiver configuration. A pattern that only matches active log files will miss rotated files entirely:

include:
  - /var/log/pods/*/*/*.log    # restrictive — misses rotated files

A wildcard pattern that also matches rotated filenames is required:

include:
  - /var/log/pods/*/*/*.log*   # correct — matches 0.log and 0.log.<timestamp>

.

Check that file_storage is configured

The file_storage extension persists file offsets and fingerprints to disk. Without it, offsets are lost on restart, causing the Collector to re-read files from the beginning on startup.

extensions:
  file_storage:
    directory: /tmp/otelcol
    create_directory: truereceivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log*
    storage: file_storage

Verify log continuity at rotation boundaries

To check for gaps or duplicates around a rotation event, identify the last line in the rotated file and the first line in the new file, then search the Collector's output for those line numbers. Each should appear exactly once with no gap between them.

Resolution

The following configuration ensures no data loss and no duplicate ingestion across both rotation scenarios.

Required configuration

1. Use a wildcard include pattern so that rotated files (e.g. 0.log.20260302-001) are discovered alongside the active log file:

include:
  - /var/log/pods/*/*/*.log*

2. Enable file_storage and reference it from the filelog receiver:

extensions:
  file_storage:
    directory: /tmp/otelcol
    create_directory: truereceivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log*
    storage: file_storage
    start_at: beginning

How each scenario is handled

Scenario 1 — Rotation whilst running: The filelog receiver tracks files by fingerprint (the first N bytes of the file, defaulting to 1 KB). When 0.log is renamed to 0.log.<timestamp>, the fingerprint is unchanged. The Collector recognises the renamed file as the same file it was already tracking, resumes reading from the last stored offset, and collects any remaining unread data. The new 0.log is detected as a new file and read from the beginning. No lines are duplicated.

Scenario 2 — Rotation whilst stopped: On restart, the wildcard pattern discovers the rotated file. The file_storage extension matches the rotated file's fingerprint against stored state and resumes from the last committed offset, collecting all lines written before rotation but not yet sent. Lines written to the new 0.log after rotation are also collected. No lines are lost and no lines are duplicated.

Behaviour summary

Important note on Scenario 1

A common misconception is that the rotated file is simply skipped when the Collector is running. In practice, the Collector follows the renamed file via fingerprint matching and reads any remaining unread lines. If the Collector is keeping pace with the log stream, there will typically be little or no unread data at the time of rotation — which can give the appearance of the rotated file being skipped — however the underlying mechanism is fingerprint-based file tracking, not file abandonment.

References

• OpenTelemetry Collector Contrib — filelog receiver
• OpenTelemetry Collector Contrib — file_storage extension
• Splunk OpenTelemetry Collector Helm chart