OTel Collector (otelcol) Process Causing High CPU Usage on Windows Server

Applicability

Description

Users may observe that the otelcol.exe process periodically or continuously consumes up to 100% of the host CPU cores on a Windows Server. During these high CPU spikes, the server can become sluggish or unresponsive, and the splunk-otel-collector service may fail to respond to system commands.

Common symptoms and error messages include:

• The host CPU reaching 100% utilisation attributed to otelcol.exe.
  
  

• Windows System Event Log reporting Event ID 7011:

  A timeout (180000 milliseconds) was reached while waiting for a transaction response from the splunk-otel-collector service.

   

• Local diagnostics endpoints (such as http://localhost:8888/metrics or health check ports) timing out or becoming unreachable during the spike.

Investigation

Follow these steps to confirm if the CPU spikes are caused by memory starvation and Go runtime garbage collection thrashing:

1. Check for Service Timeout Events in Windows Event Viewer:
   Run the following PowerShell command as Administrator to check if the Service Control Manager reported timeout transactions during the CPU spike window:

   Get-WinEvent -FilterHashtable @{ LogName = 'System' Id = 7011 StartTime = (Get-Date).AddDays(-7) } | Where-Object { $_.Message -match 'splunk-otel-collector' } | Select-Object TimeCreated, Message

    

2. Verify Steady-State DNS and Network Egress:
   Confirm that outbound connectivity to the Splunk ingest endpoint is functioning normally and not suffering from ongoing DNS resolution failures:
   
   

   Resolve-DnsName ingest.<REALM>.signalfx.com

   (Replace <REALM> with the appropriate realm, e.g., us1, eu0).
   
   

3. Check the Current Allocated Memory Limit:
   Inspect the existing environment variable setting for the collector service:

    

   [Environment]::GetEnvironmentVariable("SPLUNK_MEMORY_TOTAL_MIB", "Machine")

    

   If this returns 512 (or is not configured, defaulting to 512 MiB), the collector is running on minimum baseline memory. On production systems processing multiple telemetry pipelines (such as IIS logs, OS metrics, and traces), standard traffic fluctuations or retry queues can push memory against this cap, causing the Go runtime to enter severe garbage collection loops that saturate the CPU.
   
   

4. Inspect for Redundant Metric Scrapers:
   Review C:\ProgramData\Splunk\OpenTelemetry Collector\agent_config.yaml to verify whether duplicate scrapers are active simultaneously (e.g. both the native iis receiver and the smartagent/windows-iis receiver enabled in the metrics pipeline).

Resolution

1. Increase Collector Memory Allocation

Increase the total allocated memory in the Windows Service environment from the default 512 MiB to at least 1024 MiB (or scale using the standard ratio of 1 CPU core to 2 GB of RAM for heavy workloads).

Open PowerShell as Administrator and set the system environment variable:

[Environment]::SetEnvironmentVariable("SPLUNK_MEMORY_TOTAL_MIB", "1024", "Machine")

2. Update Memory Limiter Configuration

Update the memory_limiter processor in C:\ProgramData\Splunk\OpenTelemetry Collector\agent_config.yaml so that it provides appropriate headroom below the total allocated memory:

processors:
  memory_limiter:
    check_interval: 2s
    limit_mib: 900
    spike_limit_mib: 100

3. Remove Duplicate Receivers (If Applicable)

If redundant receivers are configured in agent_config.yaml, disable one of the duplicate monitors from the service.pipelines section to reduce unnecessary heap allocations per scrape cycle.

4. Restart the Collector Service

Restart the service to apply both the environment variable and configuration updates:

Restart-Service splunk-otel-collector

5. Confirm Performance

Monitor the host in Windows Task Manager or Performance Monitor to ensure otelcol.exe CPU usage remains low and stable during peak telemetry periods.