Splunk web UI is down due to certificate expired

Applicability

Description

ERROR SSLCommon [ WebuiStartup] - Can't read key file E:\Program Files\Splunk\splunkcert.key SSL error code=33558530 message="error:02001002:system library:fopen:No such file or directory"  ERROR HTTPServer [ WebuiStartup] - SSL context could not be created - error in cert or password is wrong  ERROR HTTPServer [ WebuiStartup] - SSL will not be enabled

Investigation

Splunk web UI is down due to the certificate used in the web.conf file being expired.

The issue was observed on Search Head.

Web.conf under /opt/splunk/etc/system/local/web.conf

[settings]
enableSplunkWebSSL = true
privKeyPath = Cert PATH/splunk_cert.key
caCertPath = Cert PATH/SPLUNK.pem

After the certificate was renewed the splunkd service was up and still unable to connect the server via splunk web UI.

Steps were taken to resolve the issue:

1. Verified the certificate validity by executing the following command in Searchhead.

/opt/splunk/bin/openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

2. User configured caServerPath in web.conf for the .pem file. It should be serverCert.
For example:

[settings]
enableSplunkWebSSL = true
privKeyPath = /opt/splunk/etc/auth/mycerts/mySplunkWebPrivateKey.key 
serverCert = /opt/splunk/etc/auth/mycerts/mySplunkWebCertificate.pem

3. Then we verified the password for the private key and added sslpassword parameter in the web.conf file.

To verify the password for private key:

openssl rsa -noout -modulus -in private.key

Under path $SPLUNK_HOME/etc/system/local/web.conf 

[settings]
sslpassword = < private-key password>

4. Still not able to connect via splunk webUI and getting the same errors in splunkd.logs

ERROR SSLCommon [ WebuiStartup] - Can't read key file E:\Program Files\Splunk\splunkcert.key SSL error code=33558530 message="error:02001002:system library:fopen:No such file or directory"
 ERROR HTTPServer [ WebuiStartup] - SSL context could not be created - error in cert or password is wrong
 ERROR HTTPServer [ WebuiStartup] - SSL will not be enabled

Resolution

We have changed the Splunk.pem to lower letters and the naming for the .key file was also changed by renaming it to splunkcert.key. Then restarted the splunk service on SH.

After that, we were able to log in to SH webUI without any issues.

Example:

Before change:

Under that $SPLUNK_HOME/etc/system/local/web.conf 

[settings]
privKeyPath = Cert PATH/splunk_cert.key
serverCert = Cert PATH/SPLUNK.pem

After change:

[settings]
privKeyPath = Cert PATH/splunkcert.key
serverCert = Cert PATH/splunk.pem

Related Links

SecureSplunkWebusingasignedcertificate


What-are-digital-certificates-and-their-trust-chains