KV Store fail to initialize when mongod received shutdown signal during startup

Applicability

Splunk Enterprise

Description

KV Store fails to initialize when MongoDB process (mongod.exe) received shutdown signal (shutting down with code:12) from splunkd.

mongo.log

I CONTROL  [thread881] CTRL_BREAK_EVENT signal
I CONTROL  [consoleTerminate] got CTRL_BREAK_EVENT, will terminate after current cmd ends
I NETWORK  [consoleTerminate] shutdown: going to close listening sockets...
I REPL     [consoleTerminate] shutting down replication subsystems
I REPL     [consoleTerminate] Stopping replication reporter thread
I REPL     [consoleTerminate] Stopping replication fetcher thread
I REPL     [consoleTerminate] Stopping replication applier thread
I REPL     [consoleTerminate] Stopping replication storage threads
I FTDC     [consoleTerminate] Shutting down full-time diagnostic data capture
I STORAGE  [consoleTerminate] shutdown: waiting for fs preallocator...
I STORAGE  [consoleTerminate] shutdown: final commit...
I JOURNAL  [consoleTerminate] journalCleanup...
I JOURNAL  [consoleTerminate] removeJournalFiles
I JOURNAL  [consoleTerminate] old journal file will be removed: 
           D:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\journal\j._0
I JOURNAL  [consoleTerminate] Terminating durability thread ...
I JOURNAL  [journal writer] Journal writer thread stopped
I JOURNAL  [durability] Durability thread stopped
I STORAGE  [consoleTerminate] shutdown: closing all files...
I STORAGE  [consoleTerminate] closeAllFiles() finished
I STORAGE  [consoleTerminate] shutdown: removing fs lock...
I CONTROL  [consoleTerminate] now exiting
I CONTROL  [consoleTerminate] shutting down with code:12

splunkd.log

ERROR KVStoreConfigurationProvider - Could not get ping from mongod. 
ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed.
ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. Failed to start KV 
      Store process. See mongod.log and splunkd.log for details..
ERROR KVStoreBulletinBoardManager - Failed to start KV Store process. See mongod.log and 
      splunkd.log for details.

Investigation

1. cipherSuite setting in server.conf has been hardened with the removal of non-ECDHE cipherSuite to pass vulnerability assessment scan.
For example: 

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:
ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:
ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256

2. MongoDB for the installed Splunk version does not support ECDHE cipherSuite and TLS communication could not be established between splunkd and the mongod.exe process. 

INFO  KVStorageProvider - Mongo ping failed. This might be expected. The error is:  (domain: 
'15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): 
[TLS handshake failed: error:00000000:lib(0):func(0):reason(0) calling ismaster on 
'xx.xx.xx.xx:8191']

Resolution

Workaround
To workaround the issue, add 'AES256-GCM-SHA384' to the list of cipherSuite in $SPLUNK_HOME/etc/system/local/server.conf and restart Splunk. 
For example: 

[sslConfig]
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:
ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:
AES256-GCM-SHA384

NOTE: By default, 'AES256-GCM-SHA384' is included in the cipherSuite setting in server.conf.

Resolution
ECDHE cipherSuite support has been added to the MongoDB bundled with Splunk version 7.2.11 / 7.3.5 / 8.0.1 onward. To harden the cipherSuite configuration, patch to the latest Splunk version.
 

Related Links

Splunk Community
Why did the KVstore fail to start after using a more secure cipher in server.conf?
https://community.splunk.com/t5/Knowledge-Management/Why-did-the-KVstore-fail-to-start-after-using-a-more-secure/td-p/429656

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release