Unable to start the KVstore in a cluster after upgrading to 9.4.X

Applicability

Description

The Kvstore fails to connect while in a cluster after upgrading to Splunk 9.4.x, standard procedure was followed but the fail to start continues

Investigation

It was found that the default certificates were expire, but after renewing them following the instructions below

How do I renew an expired Splunk Certificate?

As well as trying a manual reissue of the certificated by using:

/opt/splunk/bin/genRootCA.sh -d /opt/splunk/etc/auth/ -l 2048 and ./splunk createssl server-cert -d /opt/splunk/etc/auth/ -n server -l 2048 

the following error kept showing when trying the following command  show kvstore-status --verbose:

An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [Failed to receive length header from server. calling hello on 'cgrapp941t.atc.esd:8191'] 

As well as the status failed for the kvstore,

After confirming the error and that the default certificates are the only ones in use, the standard instructions were followed:

Custom Certs KVstore

the results for:

$SPLUNK_HOME\bin\splunk.exe cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
was

$SPLUNK_HOME/etc/auth/server.pem: OK

Which confirmed that the certificates are still valid but the fail status continues, It was also confirmed that there was an ssl password which was not suppose to be set up,

Resolution

The Workaround that was found is the following:

Download the latest version of Splunk 9.4.x but don't install it, only extract the following files from /opt/splunk/etc/auth/ :

ca.pem.default

cacert.pem.default

then copy the files into the real /opt/splunk/etc/auth/ folder installed in the environment, after removing the original files and rename them as follow:

ca.pem.default -> ca.pem

cacert.pem.default -> cacert.pem 

Then restart Splunk, kvstore should come up,