Search Head cluster, KVstore status shows error and Splunk web not loading

Applicability

Splunk Enterprise

Description

When running the following Splunk command "splunk show kvstore-status" the below error is seen:

Encountered some errors while trying to obtain kvstore status.
Couldn't complete HTTP request: Connection reset by peer.

Investigation

The following error can be seen in splunkd.log:

CONTROL [PeriodicTaskRunner] Server certificate is now invalid. It expired on YYYY-MM-DDTHH:MM:SS.000Z

From the below logs .pem certificate used in the search head is a self-signed certificate and the below log shows errors and warnings related to the self-signed certificate:

CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead.
CONTROL [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead.
CONTROL [main] Option: sslPEMKeyPassword is deprecated. Please use tlsCertificateKeyFilePassword instead.
CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.
NETWORK [main] cannot read PEM key file: /opt/splunk/etc/auth/<certFolder>/<SplunkCertificate>.pem error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
CONTROL [main] Failed global initialization: InvalidSSLConfiguration: Can not set up PEM key file.

The following event indicating a "Bad decrypt" was also seen:

ERROR SSLCommon [0 MainThread] - Can't read key file /opt/splunk/etc/auth/<certFolder>/<SplunkCertificate>.pem SSL error code=101077092 message="error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt"

The following command will fail:

curl -k -s https://localhost:8089/services/server/info | grep kvStore

This is the message expected when it fails:

Encountered some errors while trying to obtain kvstore status.
Couldn't complete HTTP request: Connection reset by peer.

Resolution

After renewing the new certificate, "sslPassword" under the stanza "[sslConfig]"  in the server.conf needs to be updated. A splunk restart is required after making these changes to re-encrypt the password and to update the server.conf file settings.

Running "splunk show kvstore-status --verbose" shows the status of the KVstore and the SHcluster.

Please find the below screenshot for reference:


Note: The "sslPassword"  will be generated if the "sslPassword" in the server.conf  is cleared and a restart the Splunk service is performed.

Related Links

https://community.splunk.com/t5/Getting-Data-In/splunk-fowarder-SSL-error-ERROR-TcpOutputProc/m-p/101157 
https://community.splunk.com/t5/Security/How-do-I-set-up-SSL-forwarding-with-new-self-signed-certificates/td-p/57046 
http://splunk-base.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certificates-and-authentication?_gl=1*ngoczy*_ga*MjAyMzQxMDUyOS4xNzAwNTM5ODk2*_ga_5EPM2P39FV*MTcwNTQ2NzUzNy41OC4xLjE3MDU0NzAyMzUuMC4wLjA.*_ga_GS7YF8S63Y*MTcwNTQ3MDAxNi4xMjIuMS4xNzA1NDcwMjM0LjYwLjAuMA..&_ga=2.260925198.2068403530.1705281356-2023410529.1700539896

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release