KV store fails to start after upgrading Splunk Enterprise on Windows platform to version 9.0.0

Applicability

Description

Consider the following scenario:

• Splunk Enterprise on Windows platform was upgraded to version 9.0.0. The upgrade was successful, however, KV store failed to start.
• Other Splunk functionalities appear to work but every task related to KV store does not work.
• mongod.log showed:

2022-07-01T01:14:06.089Z F CONTROL [main] Failed global initialization: InvalidSSLConfiguration: Could not read private key attached to the selected certificate, ensure it exists and check the private key permissions

• May also see the following in splunkd.log:(*)

08-03-2022 09:59:22.202 +0930 ERROR MongodRunner [9636 KVStoreConfigurationThread] - Failed to add PFX to certificate store
08-03-2022 09:59:22.202 +0930 ERROR MongodRunner [9636 KVStoreConfigurationThread] - Failed to open system certificate store, reason=Access is denied.\r\n

Investigation

In Splunk for Windows version prior to 9.0.0, KV Store (which is a customized version of MongoDB) relies on the certificate specified in the server.conf.

In Splunk for Windows version 9.0.0 onwards, Splunk generates pfx file from the certificate specified in the server.conf. This pfx file will be automatically imported to the Windows Certificate Store on splunkd restart. MongoDB then fetches the certificate information from the Windows Certificate Store (and not from the certificate specified in server.conf).

The account that runs Splunkd service must have access to the Windows Certificate Store. If it doesn't, Splunk cannot register the pfx file and as a result, MongoDB fails to start.

By default the Local Administrators group and Local System account have access to the Windows Certificate Store. 

Resolution

To solve the problem, therefore, the account that runs Splunkd service must be added to the Local Administrators group. If the account cannot be part of the Local Administrators group due to security requirements, the administrator needs to give the account access to the local Windows Certificate Store. The steps are outside of this article's scope.

(*) NOTE: If these errors are not in splunkd.log and the user running that runs Splunkd service has already been added to the Local Administrators group and the issue persists, renaming the certificates on Windows Personal Store (to keep a backup) and restarting Splunk should solve the issue. When Slunk restarts, it generates new certificates.

Steps: 
1. Go to $SPLUNK_HOME\etc\auth and rename server.pem to server.pem.old
2. Go to Local computer\Personal\Certificates and rename the Splunk certificates that exist there to .old.
3. Restart Splunk.