Splunk OpenTelemetry Collector does not collect Kubernetes pod logs when /var/log/pods is a symlink to a relocated path

Applicability

Description

On some Kubernetes nodes, the operating system relocates pod logs so that /var/log/pods is a symlink to another path, for example /data/var/log/pods (seen on nodes with a separate /data partition, and on other OS distributions such as BottleRocket for different relocated paths). The Splunk OpenTelemetry Collector Helm chart's agent DaemonSet bind-mounts the host's /var/log directory (and /var/lib/docker/containers) into the collector container by default. It does not mount the relocated target directory.

When the host /var/log/pods symlink points outside of the mounted /var/log tree, the symlink is dangling inside the collector container: the target path does not exist in the container's mount namespace. The filelog receiver's default include path (/var/log/pods/*/*/*.log) then matches zero files, so no pod-container log content is discovered or shipped, while metrics and Kubernetes object data (which do not depend on this file path) continue to ingest normally.

This is a host filesystem layout and mount configuration gap, not a defect in the filelog receiver. The receiver correctly reports no matches for a glob whose parent path is a dangling symlink.

In a multi-node cluster running an identical chart, values, and image, this can appear as a per-node split: nodes where /var/log/pods is a real directory (or resolves to a mounted path) continue to collect pod logs, while nodes where it is relocated to an unmounted path do not.

Investigation

Confirm host layout inside an affected collector pod:

kubectl exec -n <namespace> <agent-pod> -- ls -la /var/log
kubectl exec -n <namespace> <agent-pod> -- ls -la /var/log/pods

If /var/log is a populated real directory and /var/log/pods shows only a symlink line (no directory listing), the symlink target is not resolvable inside the container.

Enable file-discovery debug logging to confirm the receiver finds zero files:

service:
  telemetry:
    logs:
      level: debug

Look for repeating entries similar to:

fileconsumer/file.go  finding files    ... error: "no files match the configured criteria"
fileconsumer/file.go  matched files    ... "paths": []
fileconsumer/file.go  Consuming files  ... "paths": []

Check the internal metric otelcol_fileconsumer_open_files on the collector's Prometheus telemetry endpoint. A value of 0 confirms no files are open for reading; a healthy agent reports a value greater than 0 matching the number of log files on that node.

Confirm metrics and Kubernetes object data continue to ingest normally on the same node, which isolates the failure to host log file discovery rather than the collector process, network path, or exporter.

On the host, confirm whether /var/log/pods is a symlink and where it points:

ls -ld /var/log/pods
readlink -f /var/log/pods

Resolution

Add an extra hostPath volume and volume mount to the agent DaemonSet that exposes the symlink target at the identical in-container path, so the existing symlink resolves. Do not change the default filelog include path and do not mount over /var/log/pods directly.

In the chart values (adjust the hostPath path to match the environment's actual relocated target):

agent:
  extraVolumes:
    - name: data-varlog-pods
      hostPath:
        path: /data/var/log/pods
        type: DirectoryOrCreate
  extraVolumeMounts:
    - name: data-varlog-pods
      mountPath: /data/var/log/pods
      readOnly: true

Apply the change with a Helm upgrade so the DaemonSet rolls out the additional mount:

helm upgrade <release> splunk-otel-collector-chart/splunk-otel-collector -f values.yaml

Before rolling out to every affected node, confirm the individual per-container log files under the relocated path (for example /data/var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log) are regular files and not further symlinks to another unmounted path. If they are further symlinks, add an additional mount covering that target as well.

Verify the fix on one affected node or cluster before rolling out fleet-wide:

Debug logs show a non-empty matched files paths list.
otelcol_fileconsumer_open_files is greater than 0.
Pod-container log events resume arriving in the configured logs index.

Related Links

Filelog receiver README (opentelemetry-collector-contrib)
Splunk OpenTelemetry Collector chart advanced configuration (extraVolumes, extraVolumeMounts, extraFileLogs)
Kubernetes hostPath volumes