TCP output processor has paused the data flow due to default-autolb-group configuration

Applicability

Splunk UniversalForwarder, Splunk Enterprise, Splunk Cloud

Description

If a forwarder cannot reach the server(s) referenced in its outputs.conf, it will send data to the last known receiver (as long as it is available) until its attempts to reach the allowable receiver reaches connectionTimeout value in outputs.conf (by default, 20 seconds). If the forwarder is restarted, it will again send to the last known receiver until the timeout is reached again. That means, if outputs.conf is configured to point to a specific server (a cluster manager or a single indexer instance), and the forwarder cannot communicate with that instance for any reason, as long as  connectionTimeout has not expired, it will STILL send data if the last known receiver is different from that instance and can be reached. 

Forwarder WARN and ERROR Messages

During an outage where certain indexers become unavailable, if the following error occurs, posted repeatedly by a forwarder, Splunk is not sending data from this instance:

05-27-2023 13:15:29.778 -0400 WARN  TcpOutputProc [16824 parsing] - The TCP output processor has paused the data flow. Forwarding to host_dest=10.200.0.100 inside output group default-autolb-group from host_src=THIS_FORWARDER has been blocked for blocked_seconds=1800. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Additional messages show the forwarder is continuing to try contacting indexers defined in outputs.conf. Indexer connections the forwarder cannot complete are eventually placed into "quarantine". When quarantine is lifted (after 30 seconds by default), the forwarder will try that connection again:

05-27-2023 13:15:00.934 -0400 ERROR TcpOutputFd [9600 TcpOutEloop] - Connection to host=10.200.0.111:9997 failed
05-27-2023 13:15:00.934 -0400 WARN TcpOutputFd [9600 TcpOutEloop] - Connect to 10.200.0.111:9997 failed. No connection could be made because the target machine actively refused it.
05-27-2023 13:12:32.500 -0400 WARN AutoLoadBalancedConnectionStrategy [9600 TcpOutEloop] - Applying quarantine to ip=10.200.0.111 port=9997 connid=0 _numberOfFailures=205-27-2023 13:15:00.934 -0400 ERROR TcpOutputFd [9600 TcpOutEloop] - Connection to host=10.200.0.100:9997 failed 05-27-2023 13:15:00.934 -0400 WARN TcpOutputFd [9600 TcpOutEloop] - Connect to 10.200.0.100:9997 failed. No connection could be made because the target machine actively refused it. 
05-27-2023 13:12:32.500 -0400 WARN AutoLoadBalancedConnectionStrategy [9600 TcpOutEloop] - Applying quarantine to ip=10.200.0.100 port=9997 connid=0 _numberOfFailures=2

If successful connections to available indexers are posted, but no data is transmitted, this suggests multiple autolb groups where the default group member(s) are not available, as the forwarder will not transmit data from other autolb groups if the default group is not available:

05-27-2023 13:14:31.126 -0400 INFO AutoLoadBalancedConnectionStrategy [9600 TcpOutEloop] - Connected to idx=GOOD.IDX.IP.ADDRESS:9997:2, pset=0, reuse=0. autoBatch=1
05-27-2023 13:14:35.446 -0400 INFO Metrics - group=tcpout_connections, name=csra_indexers:GOOD.IDX.IP.ADDRESS:9997:3:0, sourcePort=8089, destIp=GOOD.IDX.IP.ADDRESS, destPort=9997, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00, kb=0.00, interval_sec=60
05-27-2023 13:15:01.012 -0400 INFO AutoLoadBalancedConnectionStrategy [9600 TcpOutEloop] - Closing stream for idx=GOOD.IDX.IP.ADDRESS:9997:2
05-27-2023 13:15:01.012 -0400 INFO AutoLoadBalancedConnectionStrategy [9600 TcpOutEloop] - Connected to idx=GOOD.IDX.IP.ADDRESS:9997:0, pset=0, reuse=0. autoBatch=1

Investigation

Review outputs.conf on an affected forwarder for possible configuration issues.

• Use ./splunk btool list outputs --debug to find the configuration a specific instance is using. By default, $SPLUNK_HOME/etc/system/local configs take precedence.
• If outputs.conf has defined multiple autolb groups, ensure the default-autolb-group is defined correctly, and that multiple autolb groups were intended. Are settings in etc/system/local overriding an app deployed by the Deployment Server?
• If the server defined or default-autolb-group is presented as a hostname:
  • verify the hostname and port are correct
  • test connection by changing this value to the host's IP address instead of its hostname. If IP address works, then this is an OS configuration issue (/etc/hosts) or a DNS or firewall issue.

EXAMPLE
In the configuration below without indexer discovery, two autolb-groups are defined (default-autolb-group and our_indexers). In this scenario, only one indexer is listed in in default-autolb-group. Even though other indexers are available in the other group, Splunk will still stop sending data to the indexing layer if no hosts in the default group are available. As the peer listed in default-autolb-group is also an indexer, as noted in the messages above, the forwarder will continue polling all indexers defined in all autolb groups, and will make successful connections to any that are available-- but it will not send data to ANY indexer unless a member of the default group is available.

Using ./splunk btool list outputs --debug, in the example below, the location of outputs.conf for each configured value. An outputs.con file located in $SPLUNK_HOME/etc/system/local is overriding configs pushed out by a deployment app:

$SPLUNK_HOME/etc/system/local/outputs.conf                 [tcpout]
$SPLUNK_HOME/etc/system/local/outputs.conf                 defaultGroup = default-autolb-group$SPLUNK_HOME/etc/system/local/outputs.conf                 [tcpout:default-autolb-group] 
$SPLUNK_HOME/etc/system/local/outputs.conf                 server = 10.200.0.100:9997$SPLUNK_HOME/etc/apps/forwarder-apps/local/outputs.conf    [tcpout:our_indexers]
$SPLUNK_HOME/etc/apps/forwarder-apps/local/outputs.conf    autoLB = true
$SPLUNK_HOME/etc/apps/forwarder-apps/local/outputs.conf    autoLBFrequency = 30
$SPLUNK_HOME/etc/apps/forwarder-apps/local/outputs.conf    forceTimebasedAutoLB = true
$SPLUNK_HOME/etc/apps/forwarder-apps/local/outputs.conf    server = 10.200.0.100:9997,10.200.0.111:9997,10.200.100.100:9997

Resolution

1. If a setting in $SPLUNK_HOME/etc/system/local/outputs.conf is unintentionally overriding an app configuration, remove or rename the outputs.conf file and restart Splunk on the forwarder
2. If multiple autolb groups are defined, make sure this is intentional and configured correctly. Remove excess groups if they are not needed.
3. If hostnames are used, confirm they are correct and test the connection by updating outputs.conf to use the host's ip address instead of its hostname. If it works with the IP address, this is a DNS, routing or firewall issue. Network or OS system administrators should be able to resolve the issue. Be sure to restore the hostname in outputs.conf when the issue is corrected

Related Links

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf
https://community.splunk.com/t5/All-Apps-and-Add-ons/Forwarder-reconnect-time-after-quarantined-connection/m-p/83980
 

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release