Common forwarder certificate renewal failure

Applicability

Description

While renewing the universal forwarder, getting an error as cannot read the certificate file.
 

"ERROR SSLCommon - Can't read certificate file /opt/splunkforwarder/etc/apps/dp_outputs_app/local/forwarder_21122.pem errno=33558530 error:02001002:system library:fopen: No such file or directory

Followed the below steps to generate a private key and CSR on one of the UF servers from /opt/splunkforwarder/bin location as Splunk user

Get it signed with a certificate authority, once receive the .pem file add the private key to the .pem file as per the order below same as the old cert, and replace the certificate in outputs.conf

Server certificate
Private key
Intermediate certificate (applicable in NAB)
CA root certificate (also called CA public key)

Using custom inputs and outputs apps on the deployment server from which they push the configuration to the forwarders from outputs.conf and also they are using a CA-signed certificate for forwarders and Indexers.

Investigation

Tried to renew the certificate by using the below steps:

1. Run the command

$SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem

2. Check the expiry date of the output if it expired then please do the below steps:

3. Go to $SPLUNK_HOME/etc/auth/

4. Rename server.pem to server.pem_backup

5. Restart the Splunk using the command "./splunk restart"

6. After restarting you will be able to see a new server.pem file.

7. Check the expiry date of the Certificate now using the command

$SPLUNK_HOME\bin\openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem

8. The expiry date will be extended.

Found that the path to the .pem certificate is mentioned in sslCertPath.

The sslCertPath is deprecated in the 8.2.6 Splunk version, have to change it to clientCert and observe the status.

Please find an example for your reference.

sslCertPath = $SPLUNK_HOME/etc/apps/100_<stack>EU_splunkcloud/default/<stack>EU_server.pem

To

clientCert = $SPLUNK_HOME/etc/apps/100_<stack>EU_splunkcloud/default/<stack>EU_server.pem

changed the sslCertPath to clientCert but it is still not working it gives the same error "error reading the file" as below.
 

WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
WARN  TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead
INFO  TcpOutputProc - tcpout group idx using Auto load balanced forwarding
INFO  TcpOutputProc - Group idx initialized with maxQueueSize=7340032 in bytes.
ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/apps/dp_outputs_app/local/forwarder1.pem errno=151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line.
ERROR TcpOutputProc - Error initializing SSL context - check splunkd.log regarding configuration error for server splunk-tmem-hf3.npp.esptest.aurtest.national.com.au:9997
ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/apps/dp_outputs_app/local/forwarder1.pem errno=151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line.
ERROR TcpOutputProc - Error initializing SSL context - check splunkd.log regarding configuration error for server splunk-tmem-hf4.npp.esptest.aurtest.national.com.au:9997
INFO  TcpOutputProc - tcpout group tmem-hf using Auto load balanced forwarding

Resolution

Combine the certificate and keys into a single file.

Combine your server certificate and public certificate into a single PEM file.

Set up certificate chains
To use multiple certificates, append the intermediate certificate to the end of the server's certificate file in the following order:

[ server certificate]
[ intermediate certificate]
[ root certificate (if required)]
For example, a certificate chain might look like the following:


-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the root certificate for the CA)...
-----END CERTIFICATE-----

The root CA that signed the intermediate certificate and all intermediate certificates must be in browser certificate stores.

The certificate was renewed successfully after combining the private key with PKCS#12 certificate. 

Related Links