Splunk forwarder is not sending events to Splunk Cloud

Applicability

Description

A Splunk forwarder is no longer sending events to Splunk Cloud. This can affect a Universal Forwarder, Heavy Forwarder, or an intermediate forwarding tier.

The following error appears in splunkd.log:

ERROR TcpOutputFd - Connection to host=1.22.122.222:9997 failed. sock_error = 104. SSL 
Error = No error

The sock_error = 104 message indicates that the TCP connection was reset while the forwarder was attempting to communicate with the Splunk Cloud receiving endpoint.

Investigation

Verify basic network connectivity from the affected forwarder to the Splunk Cloud input endpoint:

telnet inputs1.<stack_id>.splunkcloud.com 9997

A successful telnet connection confirms that the port is reachable. However, it does not fully validate the SSL/TLS connection or confirm that traffic is not being reset by an intermediary network device.

To further validate the connection, run one of the following commands from the affected forwarder:

openssl s_client -connect inputs1.<stack_id>.splunkcloud.com:9997

OR

$SPLUNK_HOME/bin/splunk cmd openssl s_client -connect inputs1.<stack_id>.splunkcloud.com:9997

If the following output is returned, the connection is being reset:

CONNECTED(00000003)
write:errno=104

This confirms that the forwarder can initially establish a TCP connection, but the connection is forcibly closed before the SSL/TLS session completes.

Also review the forwarding configuration on the affected forwarder, including outputs.conf, to confirm that the forwarder is using the correct Splunk Cloud receiving endpoint and port.

Resolution

Error 104 is ECONNRESET, which means “Connection reset by peer.” This indicates that the connection was forcibly closed with a TCP reset packet. The reset may be sent by the destination endpoint or by an intermediary network device, such as a firewall, proxy, load balancer, IDS/IPS, SSL inspection device, or network appliance.

Review the full network path between the affected forwarder and Splunk Cloud. Confirm that:

• Outbound traffic to the Splunk Cloud input endpoint is allowed on port 9997.
• Firewalls, proxies, or network appliances are not resetting TCP sessions.
• SSL inspection is not interfering with the Splunk forwarding connection.
• Any recent network, firewall, proxy, routing, or certificate-related changes are reviewed.
• The Splunk Cloud forwarder app and certificates are current and correctly deployed.
• Intermediate forwarders can communicate with Splunk Cloud and are not dropping or blocking forwarded traffic.

After correcting the network, firewall, or certificate configuration, restart the affected forwarder if needed, or allow it to reconnect automatically. Then confirm that events are being indexed successfully in Splunk Cloud.

Related Links

Connection problems with Universal Forwarder for Linux ARM and Splunk Cloud (SSL error)