Universal forwarder unable to send data to Splunk Cloud due to duplicate credentials package

Applicability

Description

The following error message is available in indexer logs under the TcpInputProc component:

ERROR TcpInputProc - Error encountered for connection from 
src=xxx.xxx.xxx.xxx:99999. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

In the Universal forwarders, the following messages is on splunkd.log, confirming that TCP connectivity between instances can be established:

INFO TcpOutputProc - Removing quarantine from idx=xxx.xxx.xxx.xxx:9997ERROR TcpOutputFd - Read error. Connection reset by peerWARN TcpOutputProc - Applying quarantine to ip=xxx.xxx.xxx.xxx port=9997 _numberOfFailures=2WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output 
group splunkcloud has been blocked for 82150 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address 
for host="inputsN.<SplunkDeployment>.splunkcloud.com"

Investigation

This is how the outputs.conf looks after using the credentials package, which seems to be correctly configured:

[tcpout:splunkcloud]
compressed = false
server = inputs1.<SplunkDeployment>.splunkcloud.com:9997, inputs2.<SplunkDeployment>.splunkcloud.com:9997, 
inputs3.<SplunkDeployment>.splunkcloud.com:9997, inputs4.<SplunkDeployment>.splunkcloud.com:9997, 
inputs5.<SplunkDeployment>.splunkcloud.com:9997, inputs6.<SplunkDeployment>.splunkcloud.com:9997, 
inputs7.<SplunkDeployment>.splunkcloud.com:9997, inputs8.<SplunkDeployment>.splunkcloud.com:9997
sslCertPath = $SPLUNK_HOME/etc/apps/100_splunk_splunkcloud/default/splunk_server.pem
sslCommonNameToCheck = *.<SplunkDeployment>.splunkcloud.com
sslPassword = DIAG_REDACTED_ENCRYPTED_PASSWORD
sslRootCAPath = $SPLUNK_HOME/etc/apps/100_splunk_splunkcloud/default/splunk_cacert.pem
sslVerifyServerCert = true etc/apps/100_splunk_splunkcloud/default/outputs.conf 
useClientSSLCompression = true

The following messages are also on splunkd.log trying to authenticate the (100_<SplunkDeployment>_splunkcloud app), even knowing that the path to the server.pem was correct.

ERROR SSLCommon - Can't read key file /opt/splunk/etc/apps/100_splunk_splunkcloud/default/<SplunkDeployment>_server.pem errno=151404653 error:0906406D:PEM routines:PEM_def_callback:problems getting password.

Resolution

The errors on the Universal forwarder that cannot connect to Splunk Cloud seem to be caused by the credential package not being configured correctly.

To resolve it the credentials package should be pushed to instances. Place the new package in the Deployment Server to be pushed to the instances, do not install the credential packages in the Deployment Server.

The credential package can also be installed individually by following the steps:
1) Log into your Splunk Cloud deployment
2) Download the Splunk Universal Forwarder for Linux.
3) Install the Splunk Universal Forwarder for Linux.
4) Download and install the universal forwarder credentials.

Install and configure the Splunk Cloud Platform universal forwarder credentials package

Port 9997 won't open due to an invalid cert password, therefore it is concluded that the credentials package SPL was re-used in another forwarder hashing the password twice and now the indexers cannot recognize the new forwarder.