The TCP output processor has paused the data flow - Heavy Forwarder queues are blocked while indexer is empty.

Applicability

Description

Errors seen on Indexer:

ERROR TcpInputProc [13891 FwdDataReceiverThread] - Encountered S2S Exception=Failed to parse observed latency with value=18446744073709524

Queues are blocked due to the Ingestion latency feature being added to the heartbeat.

On indexers, the error found  is:

ERROR TcpInputProc [13891 FwdDataReceiverThread] - Encountered S2S Exception=Failed to parse observed latency with value=18446744073709524 in json doc, doc="{"green":{"count":207},"red":{"count":1,"splunkd.file_monitor_input.ingestion_latency.ingestion_latency_gap_multiplier":{"ids":["4DD6F0C8-F61C-4719-98CD-4EA9F59C7B3C:10.95.237.32:60706"],"values":[18446744073709524]}}}" for data received from src=xx.xx.xx.xx:52664.

Which points to the TcpInputProc component and Splunk To Splunk communication exception. So when receiving heartbeat / data from HF indexers throw this exception/error.
Following that the HF puts the indexers under quarantine:

WARN TcpOutputProc [17851 indexerPipe_1] - The TCP output processor has paused the data flow. Forwarding to host_dest=10.95.245.10 inside output group idx from host_src=STG-EMS-MW-N2 has been blocked for blocked_seconds=8820. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

AND

WARN AutoLoadBalancedConnectionStrategy [17892 TcpOutEloop] - Applying quarantine to ip=10.95.245.10 port=9997 connid=0 _numberOfFailures=2
08-10-2022 13:20:35.805 +0300 INFO AutoLoadBalancedConnectionStrategy [17892 TcpOutEloop] - Removing quarantine from idx=10.95.245.10:9997 connid=0

As per findings on health.conf and the errors above on Indexer for TcpInputProc, 
The issue is with the heartbeat:

health.conf - Splunk Documentation
 

ingestion_latency_send_interval = <integer>* The amount of time, in seconds, that splunkd waits before it sends ingestion latency data as part of a heartbeat message.

So now ingestion latency is part of the heartbeat.

Issue is still present for on-prem and Splunk Cloud.  When issue hits the HF it starts to send the overflown integer to the Indexer and the Indexer stops responding to heartbeats due to error.
Reload helps temporary but when disabled ingestion latency feature on HF it resolves the issue.

Investigation

1. Checking the HF queues and they are all full starting with Output queue. 
2. Checking the indexer and it's queues are empty and clean. At the time of the issue the TCP input from the HF which is facing issue drops to 0. 
3. useACK was disabled first and did not help - SPL-226003, SPL-237740 issue and workaround described in the documentation under Known Issues.
4. Correlating the quarantine messages in HF, to blocked queues in HF, to errors in Indexer provides the following error in the indexer:

ERROR TcpInputProc [13891 FwdDataReceiverThread] - Encountered S2S Exception=Failed to parse observed latency with value=18446744073709524 in json doc, doc="{"green":{"count":207},"red":
{"count":1,"splunkd.file_monitor_input.ingestion_latency.ingestion_latency_gap_multiplier":{"ids":["4DD6F0C8-F61C-4719-98CD-4EA9F59C7B3C:10.95.237.32:60706"],"values":[18446744073709524]}}}" for data received from src=10.95.245.11:52664.

Resolution

[health_reporter]
aggregate_ingestion_latency_health = 0
* A value of 0 disables the aggregation feature for ingestion latency health reporter. * Default: 1 (enabled).

[feature:ingestion_latency]
alert.disabled = 1
disabled = 1

Related Links