ERROR: "Permission denied: "/opt/splunk/var/lib/splunk/kvstore/mongo/journal/lsn", terminating."

Applicability

Splunk Enterprise

Description

KVStore won't start with the next error:

Permission denied: "/opt/splunk/var/lib/splunk/kvstore/mongo/journal/lsn", terminating

Investigation

The journal and diagnostic data directories appear to have strange permissions:

/opt/splunk/var/lib/splunk/kvstore/mongo/diagnostic.data:
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? metrics.2020-08-31T09-15-14Z-00000
-????????? ? ? ? ?            ? metrics.2020-09-01T23-30-22Z-00000
-????????? ? ? ? ?            ? metrics.2020-09-06T12-24-06Z-00000
-????????? ? ? ? ?            ? metrics.2020-09-06T12-41-29Z-00000
-????????? ? ? ? ?            ? metrics.2020-09-29T06-44-49Z-00000
-????????? ? ? ? ?            ? metrics.2020-10-01T07-04-56Z-00000
-????????? ? ? ? ?            ? metrics.interim

 

/opt/splunk/var/lib/splunk/kvstore/mongo/journal:
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? j._0
-????????? ? ? ? ?            ? lsn

But it is not the permissions that are at fault, look at the parent directory:

/opt/splunk/var/lib/splunk/kvstore/mongo:
total 4947490
drwx------. 4 splunk splunk     16384 Oct  7 13:16 .
drwx------. 4 splunk splunk        32 Aug 31 13:15 ..
-rw-------. 1 splunk splunk  16777216 Aug 31 13:15 admin.0
-rw-------. 1 splunk splunk  16777216 Aug 31 13:15 admin.ns
drw-------. 2 splunk splunk       281 Oct  1 11:09 diagnostic.data
drw-------. 2 splunk splunk        29 Oct  1 11:06 journal

​​​​​​​

Resolution

Notice how the journal and diagnostic.data are directories but they do not have the executable bit set, hence they are not traversable.

So we will run:

chmod 700 /opt/splunk/var/lib/splunk/kvstore/mongo/journal
chmod 700 /opt/splunk/var/lib/splunk/kvstore/mongo/diagnostic.data
chown -R splunk:splunk /opt/splunk/var/lib/splunk/kvstore/mongo

Then restart Splunk.

If still errors then please move the contents of
/opt/splunk/var/lib/splunk/kvstore/mongo/diagnostic.data
and
/opt/splunk/var/lib/splunk/kvstore/mongo/journal
to somewhere outside of Splunk, just the contents, leave the directories where they are, and then restart Splunk.
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release