KV Store Process Failing to Start After Upgrade to Splunk Enterprise Version 9.4

Applicability

Description

Splunk Enterprise version 9.4 and higher require KV store server version 7.0 or higher. If using a custom certificate or IPv6 configuration, the upgrade to server version 7.0 will fail. Upgrades for KV store deployments with custom certificates and IPv6 configurations will be available in future Splunk Enterprise releases.

Investigation

Both in splunkd.log and mongod.log it indicates that KV store can't be started, see the error below:

E NETWORK [conn937] SSL peer certificate validation failed: self signed certificate in certificate chainERROR KVStoreBulletinBoardManager [240773 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details.

Resolution

Steps for the workaround:

1. Navigate to /opt/splunk/etc/apps/100_****_splunkcloud/
2. copy the line sslRootCAPath = $SPLUNK_HOME/etc/apps/100_****_splunkcloud/default/**_cacert.pem  from server.conf.
3. From /opt/splunk/etc/apps/100_uky_splunkcloud/default/outputs.conf copy the line serverCert = $SPLUNK_HOME/etc/apps/100_***_splunkcloud/default/***_server.pem
4. From /opt/splunk/etc/apps/100_***_splunkcloud/local/outputs.conf get the encrypted password from the line sslPassword = <****>
5. And use the below command to decrypt the encrypted password and copy it:
splunk show-decrypted --value '<encrypted_value>' 
6. Now, vi /opt/splunk/etc/system/local/server.conf 
and paste all the above copied value under the stanza [sslconfig]
Example:

[sslConfig]
serverCert = $SPLUNK_HOME/etc/apps/100_****_splunkcloud/default/****_server.pem
sslRootCAPath = $SPLUNK_HOME/etc/apps/100_****_splunkcloud/default/****_cacert.pem
sslPassword = <paste the copied decrypted value of the password> 

7.  Restart the splunk service.

8. check the KV store status with the below command:
./splunk show kvstore-status --verbose 

Related Links

Mongo_DB_7.0 (https://community.splunk.com/t5/Deployment-Architecture/KVStore-does-not-start-when-running-Splunk-9-4/m-p/708304/thread-id/29016/highlight/false) 

Migrate KV Store