The forwarder throughput is not consistent

Applicability

Description

There are ups and downs in the throughput of each Heavy Forwarder.

Why is it not consistent?

Investigation

Verify there are no issues with saturated and blocked queues Processing queues are full: "The monitor input cannot produce data because splunkd's processing queues are full"

Verify the thruput option in the limits.conf for the forwarder.

Verify the load balancing methods used in the outputs.conf of the forwarder - Set up load balancing.

Resolution

The throughput will depend on the amount of data that is being forwarded during a period of time, if the indexers are not busy with data ingestion and/or expensive searches. Connection issues and hardware performance can also impact the data throughput.

The amount of data being transferred is not always the same each day, every hour. 

The following search query shows the average throughput from forwarders:

index=_internal host=<hostname> source=*metrics.log* name=thruput | timechart span=30s avg(kb)

The following search query can be used to see the throughput by source type:

index=_internal host=<hostname> source=*metrics.log* group=per_sourcetype_thruput
| eval ingest_pipe=if(isnotnull(ingest_pipe),ingest_pipe,"none")
| search ingest_pipe=*
| timechart minspan=30s per_second(kb) by series useother=false limit=15