Splunk UF Can't Connect to Splunk Cloud, Error: Connection reset by peer (FIPS)

Applicability

Description

Some Splunk Cloud deployments have an issue whereby a new or particular Forwarder cannot send logs.

The following warnings and errors are seen in the splunkd.log.
 

WARN TcpOutputProc - Applying quarantine to ip=<Instance_IP> port=9997 _numberOfFailures=2INFO TcpOutputProc - Removing quarantine from idx=<Instance_IP>:9997

03-15-2019 15:36:11.695 +1100 ERROR TcpOutputFd - Read error. Connection reset by peer03-15-2019 15:36:12.235 +1100 ERROR TcpOutputFd - Read error. Connection reset by peer03-15-2019 15:36:12.778 +1100 ERROR TcpOutputFd - Read error. Connection reset by peer03-15-2019 15:36:13.322 +1100 ERROR TcpOutputFd - Read error. Connection reset by peer03-15-2019 15:36:13.865 +1100 ERROR TcpOutputFd - Read error. Connection reset by peer03-15-2019 15:36:13.866 +1100 WARN TcpOutputProc - Applying quarantine to ip=<Instance_IP> 
port=9997 _numberOfFailures=2

WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output 
group splunkcloud has been blocked for 300 seconds. This will probably stall the data flow 
towards indexing and other network outputs. Review the receiving system's health in the 
Splunk Monitoring Console. It is probably not accepting data.

Investigation

- Confirmed that the Internal Firewalls are not blocking the traffic and are being allowed through via Packet Capture and UI.

- A SYN/ACK was seen but not getting the third handshake.

-A new forwarder being configured had FIPS enabled set whereas another nearly identical forwarder running without issue did not have the FIPS mode set.

The following information was found in this article: Securing Splunk Enterprise with FIPS

The Federal Information Processing Standard (FIPS) uses government-certified versions of some algorithms to meet regulatory guidelines.

It should not be considered a security enhancement by itself, and might potentially reduce performance on your system.
Enable FIPS if it is a regulatory requirement for your environment.

Splunk Enterprise and the Universal Forwarder use an embedded FIPS 140-2-validated cryptographic module.

Thus you need FIPS enabled and running on both the Forwarder side and the Indexer side.

 

Resolution

In these cases, as FIPS was not set or enabled on the Splunk Cloud side, any forwarder with FIPS turned on will fail to be allowed to connect.

On the mis-configutred forwarders disable FIPS and reboot.

Check FIPS is disabled with the next command:

# cat /proc/sys/crypto/fips_enabled 0

Related Links

Securing Splunk Enterprise with FIPS