Forwarder stops sending data or data loss due to S2S errors.

Applicability

Description

The HF stopped sending data to indexers and was blocked until HF restart.  During troubleshooting, it was also determined that data was missing from CSV file inputs when these errors and blocking occurred.  The main errors observed on the indexers were "ERROR TcpInputProc [#### FwdDataReceiverThread] - Encountered Streaming S2S error=Cannot register new_channel="source::<source>|host::<host>|sourcetype::<sourcetype>|####": desired_field_count=## conflicts with existing_field_count=## for data received from src=<source IP:port>." at the time of the issues.  This issue has multiple possible causes and resolutions as described in the investigation/resolution sections.

This specific S2S cannot register new channel error causes the indexer to tear down the connection with the HF.  The HF then has to establish a new connection to the indexer.  If there are many repeated S2S cannot register new channel errors, the HF will see the connection to the indexer as blocked and quarantine it.  After it does this with all indexers, the HF can go into a state of constant blocking and must be restarted.

Investigation

There is a pattern of errors that can typically be found when one of these scenarios is encountered. Use this search against the indexers to find the S2S cannot register new channel errors:

index=_internal host IN (<index host list>) sourcetype=splunkd "*Encountered Streaming S2S error=Cannot register new_channel*"

In another search tab or window, use this search to find the INFO messages on the HF that correspond to the connection errors.

index=_internal host=<HF hostname> sourcetype=splunkd component=AutoLoadBalancedConnectionStrategy ("*Connection closed by server*" OR "*sock_error*")

The indexers will show this error:

<date/timestamp> ERROR TcpInputProc [#### FwdDataReceiverThread] - Encountered Streaming S2S error=Cannot register new_channel="source::<source>|host::<host>|sourcetype::<sourcetype>|####": desired_field_count=## conflicts with existing_field_count=## for data received from src=<source IP:port>.

Then the HF that is sending that data reports this error first:

<date/timestamp> INFO AutoLoadBalancedConnectionStrategy [#### TcpOutEloop] - Connection to <idx ip:port> closed. Connection closed by server. context=read type=zero_return

The HF then switches indexers and the S2S cannot register new channel error will appear on the new indexers.

Then the HF shows this sock_error:

<date/timestamp> INFO AutoLoadBalancedConnectionStrategy [#### TcpOutEloop] - Connection to <idx ip:port> closed. context=write sock_error=104. ssl_error=error:00000000:lib(0):func(0):reason(0)

This pattern is repeated any time the field count changes. If the field count is 0 when expected is >0, it only throws the connection closed by server message. If the field count is >0 when expected is 0, that is when the sock_error=104 is thrown. Since the source is a CSV file and the field count should never change. Something is corrupting the data between the HF and the indexer or between any UF and the HF.

Scenario 1:

The HF or UF has persistent queue configured on the [splunktcp://<port] input stanza.  Typically the UF will not be receiving from another Splunk instance, but the persistent queue on the splunktcp input for any forwarder can cause this issue.

	
[splunktcp://9997]
_rcvbuf = 1572864
connection_host = ip
host = <hostname>
persistentQueueSize = <PQ size>

When the HF has a persistent queue enabled on the splunktcp input, any time the parsingQueue is blocked, data is dropped into the persistent queue.  Any time data is dropped into the persistent queue on the HF, the events dropped into the persistent queue will cause the S2S cannot register new channel errors.

Scenario 2:

There was a change in 9.1.x to how the S2S negotiation happens between older versions of Splunk and the 9.1.x versions of Splunk.  Due to this change, if there are multiple older versions of Splunk sending through a 9.1.x HF, the S2S headers can become corrupted and cause the S2S cannot register new channel error.  Use this search to see if there are older versions sending data through the HF:

index=_internal source=*metrics.log* tcpin_connections 
| stats count by version, fwdType

If there are any versions 9.0.x or below sending that through this HF, set enableOldS2SProtocol to true in the outputs.conf on the 9.1.x HF.

Scenario 3:

For this scenario, there is no data loss.  If there is data loss, then the issue is either scenario 1 or 2.  This is also a very rare case where very few events encounter the S2S cannot register new channel errors.  This scenario happens when there are too many input channels being created and the in-memory map is unable to maintain all of the metadata maps to the event data.  Each input channel (host/source/sourcetype combination) maintains an in-memory metadata map.  When there are too many channels, the metadata can be truncated which breaks the link between data input and sourcetype.  Use this search to determine if this is potentially an issue:

index=_internal host IN (<idx host list>) source=*metrics.log* group=map name=pipelineinputchannel current_size > 5000 
| bin current_size as size span=1000 
| stats count by host,size

The threshold for encountering this is typically around 10,000 channels.  If there are many bins over 10,000 and they can be correlated to the same times as S2S cannot register new channel errors, then the limits.conf needs a few parameters updated to handle the large number of channels.  The max_inactive,  lowater_inactive, and inactive_eligibility_age_seconds need to be adjusted since the default values may not handle such large numbers of channels appropriately.

Resolution

Scenario 1:

Remove persistent queue configuration from the [splunktcpin://<port>] input stanza on the HF.

[splunktcp://9997]
_rcvbuf = 1572864
connection_host = ip
host = <hostname>

Scenario 2:

Set enableOldS2SProtocol to true under the [tcpout] stanza on the 9.1.x HF.

[tcpout]
enableOldS2SProtocol = false

Scenario 3:

Set the max_inactive, lowater_inactive, and inactive_eligibility_age_seconds to these values in the limits.conf under [input_channels] stanza on both indexers and HF's.

[input_channels]
max_inactive=300001
lowater_inactive=300000
inactive_eligibility_age_seconds=120