Restart of Intermediate Heavy forwarder stops data forwarding from Universal forwarder to Intermediate Heavy forwarder

Applicability

Description

Whenever the Intermediate Heavy forwarder gets restarted, The data forwarding from Universal forwarder is halted until the manual restart is performed on Universal forwarder.

Investigation

1.  When Intermediate Heavy Forwarder is restarted, it encounters a connection failure with Universal Forwarder, the below log is observed in splunkd.log.

 WARN TcpOutputFd [281950 TcpOutEloop] - Connect to ab.abc.ab.a:9997 failed. Connection refused

2.  Further validate if the TcpOut queue is blocked on Universal Forwarder.

 WARN TcpOutputProc [281949 parsing_0] - The TCP output processor has paused the data flow. Forwarding to host_dest=eopx-spl-lin01-mig inside output group orlen_indexers from host_src=aa.abc.a.ab has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

3. Validate if post restart of Universal Forwarder, the connection between the Universal forwarder and Intermediate Heavy Forwarder is established.

INFO AutoLoadBalancedConnectionStrategy [281950 TcpOutEloop] - Connected to idx=ab.abc.ab.a:9997:2, pset=0, reuse=0. autoBatch=1

4. Validate if the queue blockage between the Universal Forwarder and Intermediate Heavy Forwarder still continued.

WARN TcpOutputProc [281949 parsing_0] - The TCP output processor has paused the data flow. Forwarding to host_dest=abc inside output group orlen_indexers from host_src=aa.abc.ab.a has been blocked for blocked_seconds=1000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Resolution

The problem has been fixed in version 9.3.x, as one of the feature updates since 9.3 addressed the issue. This is a known issue on the in older versions of  9.1.x and 9.2.x 

To resolve this, it is recommended to upgrade either the indexer or the forwarder on the receiver side to version 9.3.x.