OpenTelemetry Collector filelog receiver may lose Kubernetes Pod logs when reading from /var/log/pods

Applicability

Description

The OpenTelemetry Collector filelog receiver intermittently loses Kubernetes Pod logs when reading log files directly from /var/log/pods.

As log volume increases, the Collector may report errors similar to:

Failed to open file "...": no such file or directory

As a result, some application logs are not ingested into Splunk even though the affected Pods continue to run.

The issue is observed when the filelog receiver monitors the Pod log directory rather than the container log directory.

Investigation

1. Verify the filelog receiver configuration

Review the OpenTelemetry Collector configuration and locate the filelog receiver.

Verify that the configured include path points to the Pod log directory.

For example:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log

If the receiver is configured to collect logs from /var/log/pods, continue with the remaining validation steps.

2. Review the OpenTelemetry Collector logs

Look for log messages similar to:

Failed to open file "...": no such file or directory

These messages indicate that the filelog receiver cannot continue reading one or more log files.

3. Verify where the log loss occurs

Determine whether the missing log entries reach the downstream ingestion pipeline.

Verify that no ingestion errors are reported by:

• Splunk HEC
• Splunk Forwarder
• the downstream ingest pipeline 

If the downstream components do not report ingestion errors while log entries are already missing, the issue originates in the filelog receiver.

4. Verify whether the affected Pods are still running

Verify whether the Pods associated with the missing log entries are still in the Running state.

Run:

kubectl get pods -A

or, if the affected namespace is known:

kubectl get pods -n <namespace>

If the affected Pods remain in the Running state while log entries are missing, the issue is not caused solely by Pod termination or log cleanup.

5. Validate the behavior using /var/log/containers

Temporarily update the filelog receiver configuration to collect logs from:

/var/log/containers/

instead of:

/var/log/pods/

Restart or redeploy the OpenTelemetry Collector Agent.

Generate application log traffic and verify that:

• new log events continue to be ingested,
• no log collection errors are reported,
• log loss is no longer observed.

If the issue no longer occurs after switching to /var/log/containers, the behavior is confirmed.

Resolution

1. Update the filelog receiver configuration to collect logs from the Kubernetes container log directory instead of the Pod log directory.

Replace the log source:

/var/log/pods/

with:

/var/log/containers/

2. Update the filelog receiver configuration to use the Kubernetes container log parsing pipeline.

3. If downstream processing depends on Kubernetes metadata, configure the Collector to populate the required resource attributes, for example:

k8s.pod.uid

k8s.pod.labels.app

4. If downstream routing depends on the original log source, configure the Collector to set a custom com.splunk.source attribute that reconstructs the original /var/log/pods/... source path.

This allows existing routing rules that rely on the source field to continue working without modification.

5. Apply the updated configuration and restart or redeploy the OpenTelemetry Collector.

6. After the Collector has restarted, verify that:

• new log events are continuously ingested,
• no additional log collection errors are reported,
• the required Kubernetes metadata is present,
• the expected source value is populated,
• downstream routing continues to function as expected.

Related Links

Filelog Receiver (OpenTelemetry Collector Contrib)

Kubernetes logging architecture