KV Store Failed After Upgrading Splunk Enterprise from 8.2.5 to 9.4.0

Applicability

Description

Following the upgrade, it is being noticed that data wasn’t reaching Splunk Cloud. Upon digging in, it was found that the KV Store service was stuck in a “starting” state and not coming online.

Since multiple add-ons are relying on KV Store, including the Splunk Add-on for Stream Forwarders, this meant data ingestion came to a complete halt.

Investigation

•  KV Store Status Check:
  Ran the below command to check the KV Store health:
  

  $SPLUNK_HOME/bin/splunk show kvstore-status --verbose

  KV Store showed status: starting, along with a mongod error saying no suitable servers were found due to a timeout.
  
  

• Check and verify the Mongod logs with the below SPL query:
  

  index=_internal sourcetype=mongod

  Look for the following SSL error:
  

  SSL peer certificate validation failed: self signed certificate in certificate chain

   

• Checked and verified splunkd.log by running the below Splunk query:
  

  index=_internal component=KVStoreUpgradeStartupThread log_level!=INFO

  Error Logs found: 

  ERROR KVStorageProvider - No suitable servers found (serverSelectionTryOnce set): [connection closed calling hello on '127.x.x.x:xxxx']

  Logs showed KV Store trying to talk to localhost on port xxxx but failing due to TLS issues.
  
  

• Verified Cert Validity:
  

  openssl x509 -noout -enddate -in /opt/splunk/etc/auth/server.pem
  openssl x509 -noout -enddate -in /opt/splunk/etc/auth/cacert.pem

The output confirmed that neither the server nor the CA certificates were expired.

Further log analysis revealed a conflict involving the CA certificate bundled within the app 100_<stack_name>_splunkcloud.

To mitigate this, the app was temporarily moved out of the Splunk apps directory to prevent it from loading during startup, and then the Splunk Service was restarted.

Resolution

After the restart of the Splunk services, the KV store was healthy, and then reinstalled the latest version of the custom app back on the heavy forwarder.

Reconfigured the Splunk Add-on for Stream Forwarders, and events started flowing again into Splunk Cloud without any further issues.

Takeaways & Best Practices:

• Starting from Splunk 9.x, TLS requirements have become stricter, and components like KV Store (Mongod) are now more sensitive to certificate mismatches.
• Custom or third-party apps bundling their own certificates can unintentionally break internal services after an upgrade.
• Always validate and test app level SSL configurations before and after upgrading Splunk, especially in production environments.
• Use internal logs (splunkd.log, mongod.log) and _internal SPL searches to trace startup issues early.

Related Links

Splunk Documentation : here