KV Store fails to start after Splunk Upgrade to 10.x due to SSL Certificate Configuration

Applicability

Description

  After upgrading a standalone Splunk environment from version 9.4.3 to 10.x, the KV Store service may fail to start, resulting in errors in the UI and logs, and preventing dependent features from functioning. In this scenario:

* The KV Store status remains "failed" with errors referencing SSL handshake failures, certificate validation, or version compatibility.

* The Splunk instance may use the default server.pem certificate or a custom certificate.

* Common log messages include "could not start mongod," "TLS handshake failed," or "certificate verify failed (19): self-signed certificate in certificate chain."

* Troubleshooting and log review do not show relevant errors in mongod.log, but splunkd.log repeatedly reports that the mongod process could not start and was terminated.

* The configuration often includes a mix of default and custom/self-signed certificates between the serverCert and sslRootCAPath parameters.

Investigation

As part of the investigation, the following was performed:

1 - Certificate and Configuration Review

* Verified that server.pem contained the correct server certificate and private key.

* Ensured combined_cacert.pem only included root and intermediate CA certs (no private keys or server cert).

* Paths for certificates and keys in server.conf were made absolute.

* Permissions and file ownership for all certificate and key files were set correctly for the Splunk user.

2 - Test Matrix—Key/Password Handling

* Unencrypted key, blank sslPassword: Splunk process crashed with SSL context initialization errors.

* Encrypted key with new sslPassword generated in Splunk 10: Splunk process crashed.

* Encrypted key with old sslPassword from Splunk 9: Splunk started, but KV Store failed.

* Confirmed placement of server.pem at both default and custom paths.

3 - CA Chain and Parameter Testing

* Swapped sslRootCAPath between custom/self-signed and organizational CA certs.

* Used built-in Splunk server.pem with both self-signed and default CA chain files.

* Noted that using the default server.pem with a custom/self-signed sslRootCAPath resulted in persistent failures.

4 - Log Analysis

* mongod.log did not record error messages for the KV Store startup attempts.

* splunkd.log reported errors such as:

- "Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout"

- "TLS handshake failed: certificate verify failed (19): self-signed certificate in certificate chain"

- "No suitable servers found: serverSelectionTimeoutMS expired"

5 - Parameter Placement Review

* Confirmed that specifying sslRootCAPath under [kvstore] can override [sslConfig] and may cause trust chain mismatches if not aligned with serverCert.

* Removing sslRootCAPath from [kvstore] allows it to inherit the global [sslConfig] value, which can resolve mismatches.

Resolution

1 - Align Certificate and CA Chain Configuration

* Ensure that both serverCert and sslRootCAPath reference files that are part of the same trust chain (either both default or both custom).

* Do not mix the default Splunk certificate with a custom/self-signed CA chain unless the CA chain explicitly includes the CA that signed the default certificate.

2 - Parameter Placement

* Remove the sslRootCAPath parameter from the [kvstore] stanza in server.conf.

* Rely on the global [sslConfig] stanza to provide the correct CA path. This allows the KV Store to inherit the trusted CA settings and prevents mismatches.

Example configuration:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/selfsigned/combined_cacert.pem
sslPassword = <password>
serverCert = /opt/splunk/etc/auth/selfsigned/server.pem
sslVersions = tls1.2

[kvstore]
sslPassword = <password>
sslVerifyServerCert = true
serverCert = /opt/splunk/etc/auth/selfsigned/server.pem # (Do not specify sslRootCAPath here)

3 - Restart Splunk

* After making these changes, restart Splunk.

* Verify that the KV Store service starts successfully with splunk show kvstore-status --verbose.

4 - If switching to a custom server certificate and CA chain, ensure the certificates are valid, not expired, and that the CA chain is complete. Regenerate certificates if necessary and confirm all settings match across [sslConfig] and [kvstore]. Use openssl s_client to test SSL connectivity and certificate trust on the relevant port (e.g., 8191) and monitor splunkd.log and mongod.log for any further errors.

 The root cause of the KV Store startup failure after upgrading to Splunk 10.x, in this specific case, was an SSL certificate configuration mismatch, particularly the placement and trust chain referenced by sslRootCAPath. Aligning the CA and server certificate and relying on the global [sslConfig] parameter resolved the issue. This approach should be used for similar SSL/KV Store failures following a Splunk upgrade.