Some of the Universal Forwarders stop sending data after Heavy Forwarder is restarted

Applicability

Splunk Enterprise Universal Forwarder

Description

When forwarding events from a Universal Forwarder to a Heavy Forwarder it is seen that when the Heavy Forwarder is restarted events from the Universal Forwarder stop being indexed

Investigation

Step 1
When searching for events from a Universal Forwarder it is found that events are missing for periods of time. In the example below when searching for over a week period for events from this Universal Forwarder a number of days show 0 events in the Timeline (see image) The issue is resolved when the problematic Universal Forwarder is restarted.




Step 2
Checked the output queue was full on the Universal Forwarder while the issue happening until the Universal Forwarder was restarted, by the following SPL.


<SPL to check the output queue>

index=_internal source=*metrics.log* group=queue
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size)
| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)
| eval fill_perc=round((curr/max)*100,2)
| timechart max(fill_perc) by name useother=false limit=15

Step 3
Confirmed the below warning messages started to be output in $SPLUNK_HOME/var/log/splunkd.log around the time the Heavy Forwarder was restarted until Universal Forwarder was restarted.
<$SPLUNK_HOME/var/log/splunkd.log on Universal Forwarder>

xx-xx-2023 xx:xx:xx.xxx +xxxx WARN TcpOutputProc [xxxx parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xxx.xxx.xxx.xxx inside output group test from host_src=xxx has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.xx-xx-2023 xx:xx:xx.xxx +xxxx WARN TcpOutputProc [xxxx parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=xxx.xxx.xxx.xxx inside output group test from host_src=xxx has been blocked for blocked_seconds=200000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.xx-xx-2023 xx:xx:xx.xxx +xxxx INFO ServerConfig [xxxx MainThread] - My GUID is xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Step 4
Since the message states there was a resource issue on the Heavy Forwarder side, checked it, but could not find any resource issues.
It can be checked using Monitoring Console.

Monitoring Console

Step 5
As there was no resource issues on the Heavy Forwarder side, tested the connection between the Universal Forwarder and the Heavy Forwarder while the issue is happening.
Confirmed that the connection was fine from Universal Forwarder to Heavy Forwarder and vice versa.

ping xxx.xxx.xxx.xxx (the Heavy Forwarder address from the Universal Forwarder)
ping xxx.xxx.xxx.xxx (the Universal Forwarder address from the Heavy Forwarder)telnet xxx.xxx.xxx.xxx:9997 (the Heavy Forwarder address from the Universal Forwarder)

Step 6
Checked any firewall settings or load balancer between the Universal Forwarder and the Heavy Forwarder, there is not any.
 

Step 7
Found a similar bug, SPL-237740, SPL-226003, where ingestion stops after some time with "Invalid ACK received from indexer=" error.
Checked that no corresponding logs are output and useACK setting is not configured.

<How to check if "Invalid ACK received from indexer=" error is there>

index=_internal source=*splunkd.log* "Invalid ACK received from indexer="

<How to check useACK setting with btool command>

$ $SPLUNK_HOME/bin/splunk btool outputs list --debug |grep -i useACK
/opt/splunk/etc/system/default/outputs.conf useACK = false

KnownIssue
 

Resolution

Found that forceTimebasedAutoLB is set to true in outputs.conf on Universal Forwarders.
The following doc recommends EVENT_BREAKER_ENABLE and EVENT_BREAKER settings in props.conf.
It seems like forceTimebasedAutoLB forcibly terminated a connection and tried to establish a new connection, which ended up with failure.
Updated forceTimebasedAutoLB to false in outputs.conf.

outputsconf

forceTimebasedAutoLB = <boolean>
* Forces existing data streams to switch to a newly elected indexer every auto load balancing cycle. 
* On universal forwarders, use the 'EVENT_BREAKER_ENABLE' and 'EVENT_BREAKER' settings in props.conf rather than 'forceTimebasedAutoLB' for improved load balancing, line breaking, and distribution of events. 
* Default: false

$SPLUNK_HOME/etc/system/local/outputs.conf

[tcpout]
defaultGroup = test
forceTimebasedAutoLB = false

 

Related Links

outputsconf

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release