Significant Delays in Log Forwarding Due to High CPU Utilization on Heavy Forwarder

Applicability

Description

When ingesting very large volumes of logs using a dedicated Heavy Forwarder, significant delays in log forwarding may occur. The delays are accompanied by continuously high CPU utilization on the HF, particularly on a TCP output thread, which can become a bottleneck. This issue arises because a single input stanza is processed by only one ingestion pipeline, concentrating workload on a single CPU core despite available capacity on others. 

Investigation

To determine the cause of the forwarding delays, CPU utilization was analyzed at the thread level using the top -H command. It was observed that a TCP output (TCPOUT) thread consistently consumed nearly one full CPU core. Further investigation confirmed that a monitored file is mapped to a single ingestion pipeline queue, meaning that a single input stanza cannot be processed in parallel across multiple pipelines. This was verified by querying Splunk internal metrics with the following SPL:

index=_internal source="/opt/splunk/var/log/splunk/metrics.log" host=<HF_host> group=queue name=tcpout*
| timechart max(current_size) AS current_size BY ingest_pipe

The results showed that only one ingestion pipeline was actively processing the data, confirming the workload concentration on a single pipeline.

Resolution

• Split Data into Multiple Streams/Files:
  Configure the source application or use external mechanisms (e.g., scripts) to split the firewall logs into multiple files or streams based on log types (e.g., incoming vs. outgoing traffic). This allows configuring multiple input stanzas in Splunk, each assigned to a separate ingestion pipeline, enabling parallel processing.
  
  

• Consider Switching to Universal Forwarder (UF):
  Using a Universal Forwarder bypasses parsing and typing queues on the forwarder, forwarding raw data chunks with lower CPU overhead. This reduces CPU load on the forwarder but shifts parsing workload to the indexers.

Related Links

https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/10.6/manage-indexes/manage-pipeline-sets-for-index-parallelization