Failed to send logs from Heavy Forwarder when set dual-pipeline/Persistent Queues

Applicability

Description

After a network outage, a Heavy Forwarder is unable to forward events from different sources after recovering. Leading heavy forwarders to not send data as expected to indexers. 

This issue was discovered when persistent queues were present in inputs.conf

persistentQueueSize = 250MB

In this case, it can apply to any case where persisteQueueSize has any value.
And dual pipeline in server.conf :

[general]
parallelIngestionPipelines = 2

Investigation

To demonstrate this issue check $SPLUNK_HOME/var/log/splunk/splunkd.log or  run the following SPL for indexer hosts:

index=_internal host=<indexer's hostname> sourcetype=splunkd log_level!=info component=TcpInputProc "S2S"

The following errors can be listed:

ERROR TcpInputProc [1556 FwdDataReceiverThread] - Encountered Streaming S2S error=38481 is not a valid combined field name/value type for data received from src=XXX.XXX.XXX.XXX:XXXX.
ERROR TcpInputProc [12167 FwdDataReceiverThread] - Encountered S2S Exception='Client is unable to process ACKs' for data received from src=XXX:XXX:XXX:XXX:XXXXX. Terminating client connection.

However, errors are not limited to those, many other S2S streaming errors can appear like

1. "Cannot register new_channel"2. "Invalid payload_size"3. "Too many bytes_used"4. "Message rejected. Received unexpected message of size"5. "not a valid combined field name/value type for data received"

Resolution

To resolve this issue there are two different actions that can be taken to address it:

1. As a first step, it was noticed when dual-pipeline changed to a single pipeline, Heavy forwarder resumed sending logs to indexers. This can be done by performing the following change in $SPLUNK_HOME/etc/system/local/server.conf
   

   [general]
   parallelIngestionPipelines = 1

2. If step above does not work or due to business needs is impossible to perform. An upgrade to one of the latest versions will fix the issue. Consider upgrading to one of the following Splunk Enterprise versions:
   
   1. Splunk Enterprise 9.3.2
   2. Splunk Enterprise 9.2.4
   3. Splunk Enterprise 9.1.7

Related Links

Persistent queue problems