MongoDB error I ACCESSS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open

Applicability

Product: Splunk Enterprise Component: KV Store (MongoDB) Operating System: Linux Applies To: All Splunk Enterprise deployments (Search Head Cluster, Standalone, License Master)

Description

When Splunk starts, it launches the KV Store service using an embedded MongoDB process. MongoDB validates that its security key file (splunk.key) has strict permissions for security reasons.

If the key file is accessible by other users (for example, 777 or 644), MongoDB refuses to start, causing the KV Store to fail and Splunk Web to become partially unavailable.

KV Store status output:
 

Investigation

1.Check the KV Store logs

grep -i "too open" $SPLUNK_HOME/var/log/splunk/mongod.log

The error can be found by looking in mongod.log for the words "too open":

index=_internal source=*mongod.log too open

If permissions are too open,  error similar to the following will be present:

permissions on splunk.key are too open
Error creating service context

Example error in mongod.log (pre 9.4.x):

2025-10-04T00:53:18.019Z I ACCESS  [main] Read security file failed
2025-10-04T00:53:18.019Z I ACCESS  [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
2025-10-04T00:53:18.020Z F CONTROL [main] Error creating service context
2025-10-04T00:53:18.020Z F CONTROL [main] Unable to acquire security key[s]

This is what the error looks like in 9.4.4:

{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ACCESS", "id":20254, "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open"}}}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":6529201, "ctx":"main","msg":"Network interface redundant shutdown","attr":{"state":"Stopped"}}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":22582, "ctx":"main","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":6529201, "ctx":"main","msg":"Network interface redundant shutdown","attr":{"state":"Stopped"}}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":22582, "ctx":"main","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":6529201, "ctx":"main","msg":"Network interface redundant shutdown","attr":{"state":"Stopped"}}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":22582, "ctx":"main","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"SHARDING", "id":5847201, "ctx":"main","msg":"Balancer command scheduler stop requested"}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":6529201, "ctx":"main","msg":"Network interface redundant shutdown","attr":{"state":"Stopped"}}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"I", "c":"ASIO", "id":22582, "ctx":"main","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"YYYY-MM-DDTHH:MM:SS.sssZ"},"s":"F", "c":"CONTROL", "id":20575, "ctx":"main","msg":"Error creating service context","attr":{"error":"Location5579201: Unable to acquire security key[s]"}}

2. Verify the file permissions on splunk.key

Run the following command to check ownership and permission mode:

ls -l $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

Example Output:

-rwxrwxrwx 1 splunk splunk 64 Oct 4 00:53 splunk.key

3. Validate Splunkd logs for cascading impact

Because the KV Store startup fails, you may also see the following in splunkd.log:

ERROR KVStoreBulletinBoardManager - KV Store process terminated abnormally (exit code 1)
ERROR KVStoreBulletinBoardManager - KV Store changed status to failed.

 

Root Cause

The file $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key must have restricted permissions because it contains security credentials used by MongoDB for internal authentication and encryption.

If the file is world-readable or writable, MongoDB enforces a hard security check and refuses to start the KV Store process, leading to service failure.

Resolution

1. Reset the file ownership and permissions to the correct values:
   

   chmod 600 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

   1. 600 = Read and write only by the owner (splunk).

   2. Do not use 777, 755, or 644 — these are considered insecure.
2. Restart Splunk to reinitialize the KV Store:
   

   $SPLUNK_HOME/bin/splunk restart

3. Verify KV Store status after restart:
   

   $SPLUNK_HOME/bin/splunk show kvstore-status

   Expected output:
   

   KV Store status: ready

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release