Mongodb not starting on search head

Applicability

Splunk Enterprise

Description

Mongodb not starting on one of the search heads out of three search heads in a cluster.

The kv store port 8191 is being used and there seems to be connectivity with the other search heads.

The port 8191 is not in LISTEN state and the mongod process is not running and  it dies after a few minutes of restarting.
 

Investigation

Run the below search in the search head and found that fails to start mongod, and it is not listening on port 8191.

index=_internal sourcetype=mongod

Verified that the status on the port 8191 mongod is down
server down not listening and it is not running

Found the below errors in splunkd.log

ERROR KVStoreLookup - Failed to execute KV Store lookups: External command based lookup 'nmon_inventory' is not available because KV Store initialization has not completed yet.REPL_HB [replexec-274] Error in heartbeat (requestId: 1863695) to msplunksh01:8191, response status: HostUnreachable: Connection refused

Found the below errors in mongod.log

[repl writer worker 9] DBException::toString(): UnknownError: assertion src/mongo/db/storage/mmap_v1/extent.h:83
Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)8, mongo::AssertionException>
0x55d0d64d2e21 0x55d0d64d2805

 

Resolution

1. Verify that server.pem not expired using the below command

/opt/splunk/bin/splunk cmd openssl x509 -enddate-in-opt/splunk/etc/auth/server.pem

2. Ensure enough disk space is available

3. stop splunk ,clean kv store and start splunk

splunk stop
splunk clean kvstore --local
splunk start

4. Change the splunk.key permission to 600

chmod 600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

5. Change the user permission to splunk directory.

chown -R user:group /opt/splunk/

6. Restart splunk

splunk restart

 

Related Links

Splunk-Enterprise KVstore Fails to start

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release