Indexer not receiving data due to port 9997 down on Intermediate Forwarder (IF)

Applicability

Description

During periods of high load on the IF, the instance might experience intermittent downtime. Although the other servers are able to successfully send data once the IF recovers, there might be an issue observed with data sent via port 9997 not being ingested into the indexer. 

Investigation

1.After careful observation, it was noted that when the IF went down and came back up, the other logs were ingested via different ports, but not through port 9997.

2.Observed queue blockages in the IF

WARN TcpInputProc [2164271 FwdDataReceiverThread] - Stopping all listening ports. Queues blocked for more than 300 seconds
WARN TcpInputProc [2164347 FwdDataReceiverThread] - Stopping all listening ports. Queues blocked for more than 300 seconds

3. Found that port 9997 is  listening on the indexer still no data is observed

4. Checked the port 9997 on IF and observed that the port was down

5. After restarting the IF, the IF port 9997 is listening and data was ingested into the indexer

6. Checked the outputs.conf and observed that the below parameter is configured on the IF:

useAck=true

7. When the useACK parameter is set to true, the universal forwarder caches events in the queue and waits for acknowledgments. If the universal forwarder does not receive the acknowledgment in time (default 300 seconds, defined by readTimeout on outputs.conf), the queue will be blocked, leading to a halt in data ingestion. Consequently, port 9997 will become inaccessible.

The log below can be seen on IF.

01-23-20xx 12:34:56.122 +0900 WARN AutoLoadBalancedConnectionStrategy [1234567 TcpOutEloop] - Read operation timed out expecting ACK from xxx.xxx.xxx.xxx:9997 connid 3 in 300 seconds.

Resolution

1. Consider adjusting the useAck setting in the outputs.conf to false, to potentially streamline the process of sending data from the Universal Forwarder to the indexer without needing an acknowledgement from the indexer.

2.  When utilizing an indexer cluster, it is possible to enable the forceTimeBasedAutoLB option on the universal forwarder (UF) with useAck. This setting in the outputs.conf file compels the UF to transition to another indexer before receiving all ACKs.

forceTimebasedAutoLB = <boolean>
* Forces existing data streams to switch to a newly elected indexer every
  auto load balancing cycle.
* On universal forwarders, use the 'EVENT_BREAKER_ENABLE' and
  'EVENT_BREAKER' settings in props.conf rather than 'forceTimebasedAutoLB'
  for improved load balancing, line breaking, and distribution of events.
* Default: false

Related Links

1. Splunk Validated Architectures - Intermediate data routing using universal and heavy forwarders

Part to note:

You should not employ useAck with UF as intermediate, and if using as part of the data path for Splunk Cloud Victoria, you should review the provided splunkclouduf.spl TA, as useAck is enabled in the default settings.