Testing forwarder network connectivity to Splunk Cloud at the application level with openssl

Applicability

Description

The symptom manifests as no events from an on-prem forwarder are being indexed in Splunk Cloud, mostly importantly logs from _internal index, which indicate a total lack of connection.

Error logs from splunkd on the forwarder are found similar to the one below:
 

xx-xx-xxxx xx:xx:xx.xx -0700 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=inputs1.<mystack>.splunkcloud.com:9997 inside output group splunkcloud from host_src=Universal/Heavy_Forwarder.com has been blocked for blocked_seconds=XXX. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

Investigation

One of the first aspects that needs to be confirmed before further troubleshooting is whether there is a successful connection from the forwarder to Splunk Cloud.

When testing basic network connectivity with telnet or nc (netcat), regardless if a successful telnet/TCP connection is established between the forwarder and the cloud, always test connectivity at the application layer.  Use the following Splunk command to test this connection:

splunk cmd openssl s_client -connect inputs1.<stackname>.splunkcloud.com:9997

Palo Alto Firewalls are very popular and out-of-the-box will block self-signed certs on non-standard ports like TCP/9997.  It is common for other, newer firewalls to also block self-signed certs as well. 

Testing an openssl connect from the Splunk app itself will speed up the troubleshooting process. Testing only a TCP connection from the OS-layer may not identify issues with the firewall regarding the SSL certificates.

The output will look like this when the firewall is blocking the connection when testing with OpenSSL using the command above:
 

user@host:~# /opt/splunk/bin/splunk cmd openssl s_client -connect inputs1.<stackname>.splunkcloud.com:9997CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 306 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1582307496
Timeout : 300 (sec)
Verify return code: 0 (ok)

There may be a log entry or message from Palo Alto similar to the one below:

CONNECTED(00000003)
depth=1 CN = Splunk Cloud Certificate Authority, L = San Francisco, O = Splunk Inc, ST = CA, OU = Splunk Cloud
verify error:num=19:self signed certificate in certificate chain
140548168603296:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1493:SSL alert number 40
140548168603296:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
-----------------------------------------------See URL Troubleshooting Dump for additional details ----------------------------------------

Notes:
To test the TCP connection on the OS layer if needed, the following commands may be used (depending on the OS):
 

nc -v inputs1.<stackname>.splunkcloud.com:9997
telnet inputs1.<stackname>.splunkcloud.com 9997
curl -v telnet://inputs1.<stackname>.splunkcloud.com:9997

Resolution

If the Splunk openssl command fails, ask to verify that the firewall rules on the network allow self-signed certificates.  Again, many firewalls--especially Palo Altos--block self-signed certs by default

To allow self-signed certs with Palo Alto:

In the Palo Alto firewall log view under the monitor tab in the traffic logs, filtering with (addr.src in <forwarder IP>), the user can see the traffic being blocked for a policy rule. To allow this access they need to edit an application policy. In the policy tab with Application Override selected they need to add port 9997 to the SSLStandard Ports. Once this setting is put into place their forwarders will be able to send data to their cloud indexers.

Once the firewall rules have changed, if the connection is successful, the Splunk openssl command will return the cert.