"ERROR TcpInputProc - Encountered Streaming S2S" in splunkd.log when sending data from Splunk to Splunk

Applicability

Description

06-21-2019 11:10:31.738 +0200 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register 
new_channel="source::perfdata|host:: xxxx.xx.xxx.xxx.xx (https://xxxxx.xx.xxxx.xxx.xx/) |sourcetype:
:nmon_data|145804": desired_field_count=14 conflicts with existing_field_count=0 for data received 
from src=10.121.200.154:50626.

Investigation

S2S communication might not be directly related to the ingestion of the nmon and IIS, which components like TcpInputProc is one of the log channels that report network information, such as the S2S protocol messages. 

There are several events reporting the TcpOutputProc (TCP Output Processor and FD) from the HF is experiencing connection timeouts to the indexer x.x.x.x, when sending the cooked data. 

It started to report warnings with Connection Refused and finally, a connection failed.
 

WARN TcpOutputFd - Connect to x.x.x.x:10101 failed. Connection refused 
ERROR TcpOutputFd - Connection to host=x.x.x.x:10101 failed 
WARN TcpOutputFd - Connect to x.x.x.x:10101 failed. Connection refused 
ERROR TcpOutputFd - Connection to host=x.x.x.x:10101 failed 
WARN TcpOutputFd - Connect to x.x.x.x:10101 failed. Connection refused 
WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:10101 timed out 
WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:10101 timed out 
WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:10101 timed out 

Resolution

This error message can be triggered when Splunk is expecting some amount of fields and it receives the wrong number, in this particular case, it was 0. This might be caused by a lack of communication, network or busy resources.
​​​​​​
As TCPOuputProc and TcpOutputFd components are the output processors that manage the forwarding of data to configured servers:port, mostly used by UF and HF. They manage the file descriptor on TCP Ports and perform operations like read/write.

The S2S (splunk to splunk) error messages can be triggered when Splunk is expecting some amount of fields and it receives the wrong number, in this particular case, it was 0. This might be caused by a lack of communication caused by either busy resources, I/O lack of space to write data or commonly network fluctuation.