Splunk Add-on for Unix and Linux : lsof.sh script causing CPU Spike on Universal Forwarder

Applicability

Description

After enabling the lsof.sh script in Splunk Add-on for Unix and Linux (9.2.0). 

It is observed that the CPU usage is spiking every 10 minutes, which is set as the Interval of running the lsof.sh script.

Investigation

During the investigation, it was observed that the lsof.sh script executes the following command to list all open files on the server:

lsof -nPs +c 0

This can be confirmed by navigating to the script present in the Splunk Add-on for Unix and Linux. Path should be:  

$SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/lsof.sf   

Search for the script snippet below:

assertHaveCommand lsof
CMD='lsof -nPs +c 0'

While the script is running, it is recommended to use the top command to monitor CPU consumption. It was noted that the lsof command utilizes a significant portion of CPU resources during execution.

Also, verify the count of files listed on the server using the following command:

lsof | wc -l

If the count of the files is huge, this would explain the higher CPU usage. 

This is expected behavior for the lsof.sh script, as it executes the "lsof -nPs +c 0" command, and prints the results. 

Resolution

To resolve this issue related to the CPU spike caused by the execution of the lsof command, need to scale up the number of CPU cores on the server. 

This will help distribute the load more efficiently, allowing the system to handle the increased resource demand during the script’s execution.

Related Links

Splunk Add-on for Unix and Linux configuration