KVStore not starting after upgrade to 9.4.x

Applicability

Description

After splunk upgrade to 9.4.0, on single instance in FIPS mode KVStore is not starting. This issue can happen in Search Head Cluster (SHC) environment as well. 

splunkd.log has the error:

01-11-2025 00:06:01.200 +0000 ERROR KVStorageProvider [46628 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [Failed to receive length header from server. calling hello on '127.0.0.1:8191']

The instance is using splunk default certs which are not expired. 

Investigation

Follow the below commands to verify the certs: 

cd $SPLUNK_HOME
source bin/setSplunkEnv
cd etc/auth
$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile cacert.pem  server.pem

After running the above command the following error is seen: 

server.pem: CN = SplunkServerDefaultCert, O = SplunkUsererror 7 at 0 depth lookup:certificate signature failure140624707415680:error:04097077:rsa routines:RSA_private_encrypt:wrong signature length:fips_rsa_sign.c:349:140624707415680:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:218:

When using the ca.pem and not the cacert.pem the response was OK.

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile ca.pem server.pem
OK

Resolution

The solution was to use the ca.pem instead of the cacert.pem:

mv $SPLUNK_HOME/etc/auth/cacert.pem  $SPLUNK_HOME/etc/auth/cacert.pem.invalidcp $SPLUNK_HOME/etc/auth/ca.pem $SPLUNK_HOME/etc/auth/cacert.pem$SPLUNK_HOME/bin/splunk restart

In some cases, kvstore will continue to fail using the default Splunk certificate. This can happen if ca.pem was expired (it expires on installs older than 10 years). If errors are seen after trying the above solution, a new rootCA will need generated as well as the rest of the default certificates. Follow this solution:

1. Run this command to generate a new root certificate:

   $SPLUNK_HOME/bin/genRootCA.sh -d /opt/splunk/etc/auth/ -l 2048

2. Remove server.pem:

   rm $SPLUNK_HOME/etc/auth/server.pem

3. Restart Splunk.
4. Verify the certificate is working properly with these two commands:

   $SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile cacert.pem server.pem$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile ca.pem server.pem

If the above commands show an OK status when running them, kvstore should no longer be restricted by certificate problems. KVstore should start at this time unless there is another issue preventing it from starting.

Related Links

KVStore is failed and caused by expired CA certificate