KVStore failed to start with default certificates after upgrading to 9.4.x

Applicability

Description

SHC KV Store did not upgrade to version 7. The environment had FIPS enabled with the use of the usual Splunk default certificate.

Investigation

1) The following configuration was observed under splunk-launch.conf - 

[default]
SPLUNK_FIPS = 1

2) There is custom PEM configuration under kvstore stanze via server.conf, even it is default PEM

etc/system/local/server.conf [kvstore]
etc/system/local/server.conf serverCert = $SPLUNK_HOME/etc/auth/server.pem
etc/system/local/server.conf caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem
etc/system/default/server.conf kvstoreUpgradeOnStartupEnabled = true

3) Validated and check [sslConfig] and [kvstore] stanza under opt/splunk/etc/system/local/server.conf on all the SH's.

4) Default SSL certs was being used and the certs were valid.

5) Added some attributes and the final entries looked like -

[sslConfig]
sslPassword = <pass>
serverCert = /opt/splunk/etc/auth/server.pem
sslRootCAPath = /opt/splunk/etc/auth/cacert.pem

[kvstore]
sslPassword = <pass>
#caCertFile = /opt/splunk/etc/auth/cacert.pem
serverCert = /opt/splunk/etc/auth/server.pem

6) Restarted the instances.

7) KV Store was still in 'starting' status.

8) Navigated to /opt/splunk/bin and observed the mongod7.0 binary is present.

9) Navigated to /opt/splunk/var/lib/splunk/kvstore_upgrade and noticed that VersionFile4.2 exists, but not VersionFile7.0.

10) Also, checked via splunk cmd mongod --version to check the mongod version used. Its showing 7.0.14

11) Moved the VersionFile4.4 to a /tmp directory and restarted the splunk instances. The splunk restart created a VersionFile7.0 entry.

12) So, then checked the KV Store status and it was still in 'starting' state.

13) Checked
-mongod.log | grep 'db version' (db version 4.2.25)
-splunkd.log | grep 'MongodRunner' (observed the logs)

-splunkd.log | grep "Splunkd start"
-mongod.log | grep "Wrong mongod version"

14) ps -ef | grep splunkd to check the mongod process.

15) During the troubleshooting, the SH cluster was not stable. Some members went out of the cluster and unable to establish connection with captain. So, restarted those servers and it came back up in the cluster.

16) Next, did the following changes and added kvstoreUpgradeOnStartupEnabled = false in [kvstore] stanza -

[kvstore]
sslPassword = <pass>
#caCertFile = /opt/splunk/etc/auth/cacert.pem
serverCert = /opt/splunk/etc/auth/server.pem
kvstoreUpgradeOnStartupEnabled = false

17) ls -lrt /opt/splunk/var/log/splunk/mongo* to check the errors in Mongod -

Error observed - ERROR: KVStore Upgrade is not supported on this instance! Reason=Instance is configured with custom KVStore certificates!

-Even though there is no custom KV Store certificates being used, the certificates used are all default.

18) Also, tried with the following command, but it did not work.

./splunk start-shcluster-upgrade kvstore -version 7.0 -isDryRun true

19) Next, did the following -

-Stopped the service.
-Reverted back VersionFile7.0 to VersionFile4.2 on all of them using mv command.
-Started the service

20) One SH migrated to 7.0.14 version, but the other 2 were still at 4.2

21) Tested by renaming the VersionFile4.2 to VersionFile5.0 for the step by step upgrade keeping kvstoreUpgradeOnStartupEnabled = false but it did not work (The manual method of migrating KV Store version did not work).

22) Validated for mongo version 4.2.25, 5.0 and 6.0 as the migration involves automated migration to these mongo versions. These versions were not present as the mongo version could not upgrade to 7.0. It was 4.2.25.

23) So, reverted back to VersionFile4.2 on all the 3 SH's and made sure that the KV Store is in ready status. serverVersion reflecting 4.2.25 and the cluster is stable.

Resolution

Splunk 9.4.2 is the one that contains supporting an upgrade with certificates defined under kvstore stanza and in this scenario with FIPS being enabled.

1) Kept and added the following configurations - 

[sslConfig]
serverCert = /opt/splunk/etc/auth/server.pem
sslRootCAPath = /opt/splunk/etc/auth/cacert.pem[kvstore]
serverCert = /opt/splunk/etc/auth/server.pem
kvstoreUpgradeOnStartupEnabled = false

2) Upgrade to 9.4.2

3) Did the manual upgrade - Manual Upgrade

Related Links

Manually Upgrading the KV Store