High CPU consumption on Universal Forwarder

Applicability

Description

An unexpected activity of a Windows host process handling the Windows event I/O operations has been observed that constantly holds its CPU usage between 10% and 30%. Users have complained of degradation in endpoint user experience.

It is noted that the splunk-PowerShell process is constantly writing out to the open PowerShell pipeline.

Investigation

Event 4103CommandInvocation(Out-Null): "Out-Null"CommandInvocation(Out-Null): "Out-Null"

Event 4103CommandInvocation(Start-Sleep): "Start-Sleep"ParameterBinding(Start-Sleep): name="Milliseconds"; value="200"

Resolution

Modified the Splunk parent PS1 code $SPLUNK_HOME\bin\splunk-powershell.ps1 on a few machines by replacing the following code:
 

if($disposedNow -eq 0){start-sleep -m 200}

With a higher value, in our test: 250000ms ( ~4 min )


After that, the CPU was stable.