TCPOutAutoLB error message on DS

Applicability

Description

From the error message "Possible duplication of events" and from the splunkd logs we could see ACK time outs which are also indications of duplicate events.

Investigation

Resolution

From the error message "Possible duplication of events" and from the splunkd logs we could see ACK time outs which are also indications of duplicate events.

When the pipeline queues fill up the memory starts climbing rapidly on the DS and CM due to a large number of ACK timeouts.

"04-04-2022 13:36:13.987 +0530 WARN TcpOutputProc - Read operation timed out expecting ACK from 10.200.147.16:9997 in 300 seconds.
04-04-2022 10:00:20.160 +0530 WARN TcpOutputProc - Read operation timed out expecting ACK from 10.200.147.16:9997 in 300 seconds.
04-04-2022 10:08:18.999 +0530 WARN TcpOutputProc - Read operation timed out expecting ACK from 10.200.147.16:9997 in 300 seconds.
04-04-2022 09:58:34.017 +0530 WARN TcpOutputProc - Read operation timed out expecting ACK from 10.200.147.16:9997 in 300 seconds.
04-04-2022 10:06:18.278 +0530 WARN TcpOutputProc - Read operation timed out expecting ACK from 10.200.147.16:9997 in 300 seconds."

If useACK is set to false then the problem will disappear.

etc/system/local/outputs.conf
useACK = false

please set it as false.
The removal of useAck will improve throughput between uf to next Splunk components. Enabling this setting might slow down the delivery of data and will force the instance to use more memory. so before disabling this setting make sure that you have no pipeline blocks or network fluctuation on the target instances. Disabling this setting may chance to resolve the 99% of delay issues.

Related Links