Load Balancers are not supported between Splunk 2 Splunk communication

Available Languages

Updated:September 29, 2026

Document ID:SPLK045440

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise Universal Forwarder

Description

The user had outputs.conf configured with a single IP address pointing to a load balancer, which resolved to the Indexers' IP addresses.

While this setup initially worked, Splunk frequently experienced connection issues, causing the Indexers to be repeatedly quarantined and removed from quarantine.

Messages similar to the following were observed in $SPLUNK_HOME/var/log/splunk/splunkd.log

01-03-2022 18:48:12.515 +0000 INFO AutoLoadBalancedConnectionStrategy [8280 TcpOutEloop] - Found currently active indexer. Connected to idx=xxx.xxx.xxx.xxx:9997, reuse=1.
01-03-2022 18:48:42.395 +0000 INFO AutoLoadBalancedConnectionStrategy [8280 TcpOutEloop] - Found currently active indexer. Connected to idx=xxx.xxx.xxx.xxx:9997, reuse=1.
01-03-2022 18:49:03.081 +0000 WARN TcpOutputProc [8279 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=xxx.xxx.xxx.xxx inside output group splunkcloud from host_src=hostname.example.com has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
01-03-2022 18:49:04.922 +0000 INFO AutoLoadBalancedConnectionStrategy [8280 TcpOutEloop] - Connection to xxx.xxx.xxx.xxx:9997 closed. context=write sock_error=104. ssl_error=error:00000000:lib(0):func(0):reason(0)
01-03-2022 18:49:13.004 +0000 WARN TcpOutputProc [8279 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=xxx.xxx.xxx.xxx inside output group splunkcloud from host_src=hostname.example.com has been blocked for blocked_seconds=20. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
01-03-2022 18:49:23.027 +0000 WARN TcpOutputProc [8279 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=xxx.xxx.xxx.xxx inside output group splunkcloud from host_src=hostname.example.com has been blocked for blocked_seconds=30.

Investigation

The user's workaround, although not a viable solution, involved manually resetting the TLS connection with the following command.

# timeout 5 openssl s_client -connect xxx.xxx.xxx.xxx:9997 &>/dev/null

However, This approach is unsupported and not recommended.

Resolution

The Splunk-to-Splunk (S2S) connections do not support external load balancers.

Instead, the user should use Splunk's built-in load balancing capabilities as outlined in the official documentation:

Setup load balancing

If a security requirement drives the need for intermediary control, the user can consider configuring the forwarder to use a SOCKS proxy, as described below.

Configure a forwarder to use a SOCKS proxy