High CPU usage on Universal Forwarder

Applicability

Description

Introspection is enabled on the Universal Forwarder (UF) and one of the CPU cores is taking up 100% CPU and 5G memory.

Investigation

From the file monitor listing using the below command:

splunk list monitor

Nested directory monitoring is found: 

Monitored Directories:
		[No directories monitored.]
Monitored Files:
	//app0/paymentConnect/mws/uat2/logs/MWSCore.gMMworkflowservice-1a.log
	/app0/gmm/BillPay/logs
	/app0/gmm/BillPay/logs/archived
	/app0/gmm/logs/gmmCashBalance.log*
	/app0/gmm/logs/gmmCore*.log*
	/app0/gmm/logs/gmmSpringConfig.log*
	/app0/paymentConnect/logs/
	/app0/paymentConnect/mws/uat1/ *.log
	/app0/paymentConnect/mws/uat2/logs/DashboardREST.gMMworkflowservice-1a.log
	/app0/paymentConnect/mws/uat2/logs/MWSBam.gMMworkflowservice-1a.log
	/app0/paymentConnect/mws/uat2/logs/MWSIntegration.gMMworkflowservice-1a.log
	/apps/mwtools/scripts/*/logs/*.delta.log
	/apps/mwtools/scripts/*/var/logs/*.delta.log
	/apps/mwtools/scripts/mwrs/var/logs
	/home/oracle/dba/log/ora_breakfix_*
	/mwinfra0/mwtools/config/mwpVersionRegFile.txt
	/mwinfra0/mwtools/scripts/*/logs/*.delta.log
	/mwinfra0/mwtools/scripts/*/var/logs/*.delta.log
	/mwinfra0/mwtools/scripts/mwrs/var/logs
	/mwinfra0/tomcat*/*/tc*u/autoBlackOut/action.log
	/mwinfra0/tomcat*/*/tc*u/autoBlackOut/jvmBlackout.log
	/opt/selfHeal/daemon/logs
	/opt/selfHeal/modules/.../conf/*.txt
	/opt/selfHeal/modules/.../logs/*.log
	/opt/selfHeal/modules/MW/sslExpiryAlertNotify/logs/*.status
	/tmp/discover.txt.*
	/var/log/cfengine/build-fs.log*
	/var/log/cfengine/Infra_Setup.log*
	/var/log/cfengine/messages*
	/var/log/cfengine/webksh.log*
	/var/log/messages*
	/var/tmp/bfbBasicLogs/*.log
	/var/VRTSvcs/log/engine_A.log*
	/web0/mwtools/scripts/*/logs/*.delta.log
	/web0/mwtools/scripts/*/var/logs/*.delta.log
	/web0/mwtools/scripts/mwrs/var/logs

However, from the input status command (see below) the UF only monitors less than 10 files and all of them have finished status.

splunk list inputstatus

After collecting pstacks  most of the CPU usage is used to discover the learned source type.

This can be confirmed from below.
etc/apps/learned/local/ have a large sourcetypes.conf

etc/apps/learned/local/-rwxr-xr-x. 1 support support    951685 May  6 14:02 props.conf
-rwxr-xr-x. 1 support support 134364485 May  6 07:39 sourcetypes.conf

Resolution

Define the sourcetype for all monitoring stanza to avoid large sourcetype files that might consume a lot of resources