KV Store initialization failed

Applicability

Splunk Enterprise

Description

Splunk heavy forwarder was configured to send the data to the Splunk cloud through apps.

It has been observed that few of the apps are not sending the logs to Splunk cloud and with the following error logged in splunkd.log:
 

 "HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator."

 

Investigation

On checking the status of KV Store, the status displayed was failed.

Running the following command will help us to understand the KV Store status:

C:\Program Files\Splunk\bin>splunk show kvstore-status
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Splunk username: splkadmin
Password:This member:
backupRestoreStatus : Ready
disabled : 0
guid : 572C383A-B2BB-4EE5-BDB7-EAC2BE1EACA0
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger

Following is the error message from Splunkd.log (HF):

"raise HTTPError(response)
solnlib.packages.splunklib.binding.HTTPError: HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator."

Following are the messages from Mongod.log:

W NETWORK [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
F NETWORK [main] The provided SSL certificate is expired or not yet valid.
F - [main] Fatal Assertion 50755 at src\mongo\util\net\ssl_manager_windows.cpp 1609
F - [main] \n\n***aborting after fassert() failure\n\n

 

Resolution

The KV-store is failed due to the SSL certificates expired.

Following the below steps will resolve the issue:

1. Open Windows certificate management MMC 
2. Navigate to Personal > Certificates
3. Delete any entries named SplunkServerDefaultCert
4. Restart Splunk.
5. Check the expiry date for the SSL certificates.
6. Check the KV Store status.

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release