Post-Upgrade Issue: KV Store Fails to Start in Splunk 9.4.x

Applicability

Description

Following an upgrade to Splunk 9.4.x, data ingestion unexpectedly stopped, and the KV Store service failed to start, even though the upgrade process showed KV Store had successfully upgraded to version 7.0.

Investigation

Expired Default Certificates:

Default certificates can expire if they have been used for more than 3 years. If default certificates are used, follow these instructions: 

• To check KV Store status, run the following command:
  

  $SPLUNK_HOME/bin/splunk show kvstore-status

  Result: KV Store status showed as failed.
  
  

• To review certificate configuration, determine which certificate was being used (default or custom) and execute
  

  /opt/splunk/bin/splunk btool server list sslConfig --debug

  If default certificates are being used (serverCert = $SPLUNK_HOME/etc/auth/server.pem caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem), proceed to the next bullet point. If custom certificates are used, adjust the path in the following commands to check the expiration date:

• Validated certificate expiry based on the output from btool and confirmed the expiration dates of the configured certificates using
  

  --> openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/cacert.pem
  --> openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem

  Note: If any certificates were expired, they needed to be regenerated with valid parameters, ensuring proper Extended Key Usage (EKU) values (e.g., TLS Web Client Authentication for client certificates).

Server Purpose:

• To confirm this, run the following command (adjust the path and file name for custom certificates):
  a. Go to the "/etc/auth" directory:
  

  cd $SPLUNK_HOME/etc/auth

  b. Run the below command:

  openssl x509 -noout -in server.pem -purpose

  Expected output:

  Certificate purposes:
  SSL client : Yes
  SSL client CA : No
  SSL server : Yes
  SSL server CA : No

If the above lines are returned with different values, the certificate purpose is not correct.

Resolution

• Expired Certificates: To resolve, generate new certificates per your organizations requirements or use new default certificates:
  • If the certificates are default certificates, a new server.pem can be generated by removing the expired one and restarting Splunk. A new certificate will be created.
  • If the certificate is a custom certificate, renew them per the security requirements for the infrastructure and be sure they also follow Splunk's requirements. The certificate must be both a client and server purposed certificate, and the certificate will need to be prepared for use before deploying.
• Incorrect server purpose
  
  • If the purpose is incorrect, a new certificate will need to be created. Please follow the directions in the Expired Certificates section to create a new certificate. Be sure it has the purpose set properly in it when you create the certificate. 
• If any certificate changes were made and mongo is still failing to start
  • Regenerated splunk.key (if applicable). In cases where new certificates were generated and the existing splunk.key was still tied to the previous one, it had to be removed:
    a. Go to the directory:
    

    cd /opt/splunk/var/lib/splunk/kvstore/mongo

    b. Move the spkunk.key file to the tmp directory:

    mv splunk.key /tmp

    c. Restart the Splunk service to take the change into effect:

    $SPLUNK_HOME/bin/splunk restart

     

Related Links

Splunk documentation on Upgrade the KV store server version 7.0 : here

steps for securing your splunk enterprise deployment with tls

How to prepare TLS certificates for use with the Splunk platform

Renew existing tls certificates