KV Store Status Shows “Failed” After Upgrading Splunk Enterprise from 9.3.5 to 9.4.4

Applicability

Description

After upgrading Splunk Enterprise from 9.3.5 to 9.4.4 in a search head cluster, the KV Store displays a failed status due to a stale version file, KV Store metadata corruption and SSL certificate configuration mismatch. 

Investigation

1) After the upgrade, the KV Store status showed failed on the instance.

./splunk show kvstore-status --verbose
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Splunk Enterprise 9.4 and higher no longer support KV store server version 4.2. Upgrade to KV store server version 7.0 for continued support and security, and to comply with Splunk Support Policy. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/MigrateKVstore in the Admin manual to plan your upgrade.
 
This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                   featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [Failed to connect to target host: x.x.x.x:8191]
                                          guid : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
                                          port : 8191
                                    standalone : 1
                                        status : failed
                                 storageEngine : wiredTiger
                      versionUpgradeInProgress : 0

2) The directory $SPLUNK_HOME/var/run/splunk/kvstore_upgrade contained an old version file (versionFile42).

3) The stale version file was removed, but the KV Store continued to show a failed state.

4) A review of mongod.log indicated SSL certificate errors.

5) The system was using Splunk default certificates, while a third-party root certificate remained configured in server.conf, creating a configuration conflict.

Resolution

1) Remove stale KV Store version file :- 

a) Navigate to $SPLUNK_HOME/var/run/splunk/kvstore_upgrade

b) Back up the existing version file (versionFile42).

c) Removed the stale file.

d) Restarted Splunk.

Outcome: KV Store remained in a failed state.

2) Performed KV Store clean :-

a) Backup required before cleaning.

cp -r $SPLUNK_HOME/var/lib/splunk/kvstore /tmp/kvstore_backup

b) Stopped Splunk

c) Clean KV Store:

./splunk clean kvstore --local

d) Started Splunk.

Outcome: KV Store moved to starting, confirming recovery progress.

3) Correct SSL certificate configuration :-

a) Review mongod.log and confirm SSL certificate errors.

b) Edit server.conf and remove the conflicting SSL root CA setting:

[kvstore]
#sslRootCACert = /path/to/thirdparty/rootCA.pem

c) Save the file and restart Splunk. 

Final Outcome: KV Store status changed to ready, confirming full resolution.

Related Links

Upgrade the KV store server version

KV store troubleshooting.