Drive space filling up Otel Connector control plane, causing cluster inaccessibility

Applicability

Description

Splunk OpenTelemetry Collector (splunk-otel-collector-agent) causing Openshift control plane issues and cluster inaccessibility.

Splunk OTel Collector has been functioning for about a year. Now, OpenShift Kubernetes cluster master servers have been in an unhealthy state, causing their Kubernetes cluster to be inaccessible. It seems splunk-otel-collector-agent (otelcol) processes running on the control plane/master servers are holding log files ope (some are very large (1GB+)) that would normally be closed and deleted. 

Investigation

Capture the following information for investigation into this issue:

1. Otel version, in case this becomes relevant during troubleshooting 
   

   splunk-otel-collector --version

2. Deployment details - the outputs of these commands may offer insights into the issue and/or validation of findings:
   

   kubectl describe pod splunk-otel-collector-agent-xxxxx -n <namespace>
   kubectl get pods -o wide -n <namespace>

3. Logs from problematic pods while the issue is occurring (the outputs of these commands assume you are working from the directory where you want the outputs saved):
   1. Splunk Otel collector agent log, if applicable (select the splunk pod name that contains "agent" in the name):
      

      kubectl logs <pod_name> -n splunk > splunk-otel-collector-agent.log

   2. Cluster receiver logs (select the splunk pod name that contains "k8s-cluster-receiver" in the name):

      kubectl logs <pod_name> -n splunk > splunk-otel-collector-k8s-cluster-receiver.log

   3. Current configuration settings:
      1. Values.yaml:

         helm show values splunk-otel-collector-chart/splunk-otel-collector > values.yaml

      2. Configmap.yaml
         from Kubernetes (replace my-configmap in the command below with the actual name of your configmap, do this for both the Cluster Receiver and Agent Collector):

         filekubectl get configmap my-configmap -o yaml > my-configmap.yaml

         or from Linux (capture the configmap.yaml file from this path):

         cp /etc/otel/collector/configmap.yaml > configmap.yaml

      Log file findings:

      1. Every 10 seconds, the splunk-otel-collector-k8s-cluster-receiver.log shows failures to scrape the Prometheus endpoint due to high memory utilization. Ingestion stops until memory falls back down.
         

         2025-07-31T11:28:50.349Z error scrape/scrape.go:1298 Scrape commit failed {"kind": "receiver", "name": "prometheus/k8s_cluster_receiver", "data_type": "metrics", "scrape_pool": "otel-k8s-cluster-receiver", "target": "http://XXXXXXXXXXXXX:8889/metrics", "error": "data refused due to high memory usage"} github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1 github.com/prometheus/prometheus@v0.54.1/scrape/scrape.go:1298 github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport github.com/prometheus/prometheus@v0.54.1/scrape/scrape.go:1376 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run github.com/prometheus/prometheus@v0.54.1/scrape/scrape.go:1253 
         2025-07-31T11:28:50.250Z warn internal/transaction.go:129 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus/k8s_cluster_receiver", "data_type": "metrics", "scrape_timestamp": 1753961330249, "target_labels": "{__name__=\"up\", instance=\"XXXXXXXXXXXXX:8889\", job=\"otel-k8s-cluster-receiver\"}"} github.com/prometheus/prometheus@v0.54.1/scrape/scrape.go:1376 github.com/prometheus/prometheus/scrape.(*scrapeLoop).run github.com/prometheus/prometheus@v0.54.1/scrape/scrape.go:1253 
         2025-07-30T16:07:01.291Z info memorylimiter@v0.117.0/memorylimiter.go:196 Memory usage back within limits. Resuming normal operation. {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics/collector", "cur_mem_mib": 484} 2025-07-30T16:07:10.165Z info internal/retry_sender.go:126 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "splunk_hec/platform_metrics", "error": "Post \"https://XXXXXXXXXXXXXXX:8088/services/collector\": EOF", "interval": "35.570898833s"} 

      2. Then, the splunk-otel-collector-agent.log reports HTTP 400 messages which appear to begin at the 10 second interval marker, then 3 or 4 times in 1-2 second intervals, stop again, and pick up once more at the 10 second mark:
         

         2025-07-31T11:28:50.917Z error internal/queue_sender.go:103 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "splunk_hec/platform_metrics", "error": "not retryable error: Permanent error: \"HTTP/1.1 400 Bad Request\\r\\nContent-Length: 60\\r\\nConnection: Keep-Alive\\r\\nContent-Type: application/json; charset=UTF-8\\r\\nDate: Thu, 31 Jul 2025 11:28:50 GMT\\r\\nServer: Splunkd\\r\\nVary: Authorization\\r\\nX-Content-Type-Options: nosniff\\r\\nX-Frame-Options: SAMEORIGIN\\r\\n\\r\\n{\\\"text\\\":\\\"Incorrect index\\\",\\\"code\\\":7,\\\"invalid-event-number\\\":1}\"", "dropped_items": 14} go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1 go.opentelemetry.io/collector/exporter@v0.117.0/exporterhelper/internal/queue_sender.go:103 go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1 go.opentelemetry.io/collector/exporter@v0.117.0/internal/queue/consumers.go:43

          

       

Findings:

The core issue in this case is that an index either does not exist in Splunk or is named differently in the otel configuration than it is on the Splunk indexers, so data is not getting ingested. If the collector cannot successfully offload the data it has collected, its internal buffers (even if persistent queue is enabled) will eventually fill, leading to backpressure that propagates back to the receivers, causing them to hold onto files or drop data. The fact that this happens at 10-second intervals (matching Prometheus scrape intervals) suggests that every attempt to send a batch of metrics is failing due to this misconfiguration.

With Splunk OTel Collector agents configured to use an in-memory queue for log export while also tailing a large volume of log files on the master nodes, when the export pipeline experiences backpressure, the collector holds onto file handles to prevent data loss, which in turn prevents the operating system from truly deleting rotated log files, leading to disk space exhaustion. The frequent restarts occur due to memory pressure from the accumulating in-memory data.

Here's a breakdown of the findings:

1. The splunk_hec/platform_metrics exporter is sending metrics to an index that either doesn't exist, is misspelled, or the HEC token used does not have permissions to write to that index.

   • This is a critical configuration error that will cause data to fail to get indexed. Since it's a "not retryable error", the collector is dropping this data immediately.
   • If the collector cannot successfully offload the data it has collected, its internal buffers (even if persistent queue is enabled) will eventually fill, leading to backpressure that propagates back to the receivers, causing them to hold onto files or drop data. The fact that this happens at 10-second intervals (matching Prometheus scrape intervals) suggests that every attempt to send a batch of metrics is failing due to this misconfiguration.

2. In-memory sending queue and backpressure:

   • The values.yaml configuration for the Splunk OTel Collector explicitly shows splunkPlatform.persistentQueue.enabled: false. This means that the collector's sending queue, which buffers data before exporting it to the Splunk HEC endpoint, is configured to operate in-memory.
   • The filelog receiver uses file_storage for its internal checkpoints (to track its position in the log files) in /var/addon/splunk/otel_pos. This is distinct from the exporter's sending queue.
   • When the collector experiences backpressure (e.g., if the network connection to the Splunk HEC endpoint is slow, or the Splunk HEC endpoint itself is overwhelmed or unreachable, or in this case data can't be sent because of a configuration issue with the index), its in-memory sending queue will fill up.

3. File handle retention and disk space exhaustion:

   • To prevent data loss when the in-memory queue is full, the filelog receiver will stop reading new data from the log files. Crucially, it keeps the file handles open to these log files so that it can resume reading from the last known position once the backpressure is relieved.
   • However, the underlying log rotation mechanisms (e.g., Kubernetes' own log rotation or logrotate for system logs) attempt to rotate and delete old log files. Because otelcol still holds open file handles to these "deleted" files, the operating system cannot truly free up the disk space occupied by them. The files are marked for deletion but remain on disk until all open handles are released. With the files open, they cannot be deleted from the node until closed, filling up the disk space until then. This explains why the disk usage increases despite logs being "rotated."

4. High restart counts and OOMKills:

   • The kubectl get pods output will show many splunk-otel-collector-agent pods with high restart counts.
   • The kubectl describe pod output and values.yaml will show the otel-collector container's memory limit. (for example, 500Mi).
   • An in-memory queue, especially when facing significant backpressure, can quickly consume available memory. It's highly probable that the collector is hitting it memory limit, leading to Out-Of-Memory (OOM) kills and subsequent restarts. While a restart might temporarily release some file handles, the collector will re-open them upon starting, perpetuating the problem.
   • The "Scrape commit failed... data refused due to high memory usage" messages in splunk-otel-collector-k8s-cluster-receiver.log explicitly state that the Prometheus receiver within the cluster receiver pod is unable to process scraped metrics because the collector itself is experiencing high memory usage. The memory_limiter processor (configured in both the cluster receiver configmap.yaml and collector agent configmap.yaml files) is actively dropping or refusing data due to memory constraints.
   • The collector is hitting its memory limit, pausing operations, and then resuming when memory frees up. This cycle indicates constant memory pressure.

Resolution

Implementing these changes should resolve the immediate backpressure issue and protect against future data loss by allowing the collector to buffer logs to disk during periods of backpressure, enabling it to release file handles to rotated logs more promptly and prevent disk space exhaustion.

1. Immediate Mitigation: Removing tolerations for master nodes is a highly advisable step, at least as an immediate mitigation, and potentially as a long-term strategy.
   1. Reasons to remove tolerations:
      1. Control Plane Stability: The primary goal is to ensure the stability and accessibility of your Kubernetes control plane. If the splunk-otel-collector-agent is causing disk space exhaustion and high resource usage on master nodes, it directly jeopardizes the cluster's core functionality. Isolating the control plane from this workload is paramount.
      2. Mitigation of Current Issue: This action would immediately prevent the splunk-otel-collector-agent from scheduling on master nodes, thereby stopping the log file handle issue and disk space exhaustion on those critical servers.
   2. To implement (in values.yaml), locate the tolerations section under the top-level tolerations (this applies to the agent DaemonSet) and remove or comment out the entries that allow scheduling on master/control plane nodes:
      

      tolerations:
        # - key: node-role.kubernetes.io/master # Remove or comment this line
        #   effect: NoSchedule
        #   operator: Exists
        # - key: node-role.kubernetes.io/control-plane # Remove or comment this line
        #   effect: NoSchedule
        #   operator: Exists
        # ... keep other necessary tolerations if applicable to other node types you want to monitor

      Important Considerations if you remove tolerations:

      • Loss of Control Plane Observability: By preventing the agent from running on master nodes, you will lose direct collection of host-level metrics, container logs, and system logs from those specific nodes. This creates a monitoring blind spot for your control plane components.

      • Underlying Issues Remain: Removing tolerations addresses the _symptom_ on master nodes, but it does not fix the underlying problems (memory pressure, HEC configuration errors, certificate issues). These problems will continue to affect the splunk-otel-collector-agent pods running on your worker nodes.

2. Root Cause Resolution: Prioritize fixing the underlying issues:
   • Correct Splunk HEC Configuration: Address the "Incorrect index" error for splunk_hec/platform_metrics. Verify the affected index exists in Splunk and the HEC token has the necessary write permissions to it. This will allow data to actually be accepted by Splunk.
   • Enable Persistent Queue: Once data can actually be exported, enabling the persistent queue will allow the collector to manage its internal buffers more effectively and release those log file handles, freeing up the disk space currently consumed by old logs. Increasing memory limits will also prevent OOM kills. To do this,
     Enable splunkPlatform.persistentQueue.enabled: true in your values.yaml and increase the agent's memory limits (e.g., to 1Gi or more) to provide robust buffering and prevent in-memory queue overflows.
     • Modify values.yaml to enable the persistent queue for splunkPlatform:"

       splunkPlatform:
         # ... other configurations ...
         sendingQueue:
           enabled: true
           numConsumers: 10
           queueSize: 1000 # This queue will now be persistent
         persistentQueue:
           enabled: true
           storagePath: "/var/addon/splunk/exporter_queue" # This path is already mounted to the agent

     • Important: It is advised to increase the agent.resources.limits.memory to at least 1Gi (or more, depending on your log volume) to accommodate the additional overhead of the disk-based storage.
   • Monitor Splunk HEC Performance: After implementing the above, you must monitor the disk usage of the persistent queue's storagePath (ex: /var/addon/splunk/exporter_queue). If this directory starts to grow uncontrollably, it means your export pipeline is still not keeping up with the ingestion rate, and further investigation into Splunk HEC capacity or network throughput is required.

Related Links

Here are documentation links that support the findings and recommendations discussed:

1. Persistent Queue for Data Durability and Disk Space Management

• Splunk OpenTelemetry Collector Helm Chart - Advanced Configuration (Data Persistence): This official documentation explicitly details the persistentQueue option and its implications, including the recommendation to increase memory limits.
• OpenTelemetry Collector Exporter Helper - Sending Queue: This explains the general mechanism of the sending queue used by exporters, which can be configured for persistence.
• OpenTelemetry Collector Contrib - File Storage Extension: This is the underlying extension that provides the disk-based storage for the persistent queue.

2. Memory Limiter Processor

• OpenTelemetry Collector Contrib - Memory Limiter Processor: This documentation describes how the memory_limiter processor works, including its role in preventing out-of-memory conditions by dropping data or pausing pipelines.

3. Splunk HEC "Incorrect Index" / HTTP 400 Errors

• Splunk Documentation - Troubleshoot HTTP Event Collector: This resource provides common issues and solutions for HEC, including error codes and messages related to incorrect indexes or tokens. While direct links to specific error codes can vary by Splunk version, this is the general troubleshooting area. (You may need to navigate to the specific version of Splunk you are using).

5. Tolerations and Running on Control Plane Nodes

• Kubernetes Documentation - Taints and Tolerations: This explains how taints are applied to nodes (like master nodes to prevent general workloads) and how tolerations allow pods (like your collector agent DaemonSet) to be scheduled on those tainted nodes.
• Kubernetes Documentation - DaemonSets: Describes how DaemonSets ensure that a copy of a Pod runs on all (or a subset of) nodes, and how tolerations affect their scheduling.

6. Log File Handling (Filelog Receiver, Held Handles, Disk Exhaustion)

• OpenTelemetry Collector Contrib - Filelog Receiver: This documentation details how the filelog receiver tails files, manages state, and uses storage for checkpoints. Understanding its operation helps explain why file handles are held.
• Linux File System Semantics (General Concept): While not a single official link, the behavior of files being "deleted" but space not being reclaimed until all open file handles are released is a fundamental aspect of Unix-like file systems. This is often explained in operating system design courses or advanced Linux administration guides.