KV Store does Not Start in Standalone Search Head
Available Languages
Applicability
Splunk Enterprise KvStoreDescription
Issue:
Attempting to start KvStore on a standalone search head, found in the logs the following messages:
W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release. I ACCESS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
Running splunk as root, but the $SPLUNK_HOME directory is owned by user "splunk". A further revision on the mongo directory shows that permission across the whole directory (mongo) are 650, as per documented in [REF 1] these permissions should be set on -rw-------. It is easily seen that this correlated well whit the above log message. Verify if permissions changed on the splunk.key file to its defaults, after a splunk restart permissions where back set to 650.
Applies to:
Version 8.1.6
Linux OS
Investigation
For checking the strange behavior, of the changed permissions after the restart, a test environment was set using the same version in this case Splunk Enterprise 8.1.6. Also, deliberately, setting the user Splunk owner of the $SPLUNK_HOME directory and starting splunk as the root user.
[splunker@ip-x-x-x-x ~]$ cd /opt/splunk/ [splunker@ip-x-x-x-x splunk]$ ls -l total 3884 drwxr-xr-x. 4 splunker splunker 4096 Sep 14 23:10 bin -r--r--r--. 1 splunker splunker 57 Sep 14 19:21 copyright.txt drwxr-xr-x. 16 splunker splunker 4096 Jan 19 02:56 etc drwxr-xr-x. 4 splunker splunker 62 Sep 14 23:07 include drwxr-xr-x. 8 splunker splunker 4096 Sep 14 23:10 lib -r--r--r--. 1 splunker splunker 85440 Sep 14 19:21 license-eula.txt drwxr-xr-x. 3 splunker splunker 58 Sep 14 23:07 openssl -r--r--r--. 1 splunker splunker 525 Sep 14 22:48 README-splunk.txt drwxr-xr-x. 4 splunker splunker 108 Sep 14 23:07 share -r--r--r--. 1 splunker splunker 3867956 Sep 14 23:10 splunk-8.2.2.1-ae6821b7c64b-linux-2.6-x86_64-manifest drwxr-xr-x. 2 splunker splunker 54 Sep 14 23:08 swidtag drwx--x---. 7 splunker splunker 72 Jan 19 02:56 var
Verify the permissions are the defaults under the mongo folder (for simplicity sake, only relevant output is shown):
[splunker@ip-x-x-x-x mongo]$ pwd /opt/splunk/var/lib/splunk/kvstore/mongo [splunker@ip-x-x-x-x mongo]$ ls -l total 1310736 -rw-------. 1 splunker splunker 16777216 Jan 19 02:56 admin.0 -rw-------. 1 splunker splunker 16777216 Jan 19 02:56 admin.ns -rw-------. 1 splunker splunker 16777216 Jan 19 03:06 config.0 -rw-------. 1 splunker splunker 16777216 Jan 19 03:01 config.ns drwx------. 2 splunker splunker 48 Jan 19 03:10 diagnostic.data drwx------. 2 splunker splunker 6 Jan 19 03:10 journal -rw-------. 1 splunker splunker 16777216 Jan 19 02:56 local.0 -rw-------. 1 splunker splunker 268435456 Jan 19 03:10 local.1 -rw-------. 1 splunker splunker 16777216 Jan 19 03:10 local.ns -rw-------. 1 splunker splunker 0 Jan 19 03:10 mongod.lock -rw-------. 1 splunker splunker 88 Jan 19 02:56 splunk.key
Starting splunk as the root user and then verifying:
[splunker@ip-x-x-x-x ~]$ ps -ef | grep splunkd splunkuf 979 1 0 02:41 ? 00:00:05 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd splunkuf 1862 979 0 02:41 ? 00:00:00 [splunkd pid=979] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_syste md [process-runner] root 13141 1 5 03:23 ? 00:00:04 splunkd -p 8089 start root 13143 13141 0 03:23 ? 00:00:00 [splunkd pid=13141] splunkd -p 8089 start [process-runner] root 13406 13143 0 03:23 ? 00:00:00 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore root 13632 13143 0 03:23 ? 00:00:00 [splunkd pid=13141] [search-launcher] root 13633 13632 0 03:23 ? 00:00:00 [splunkd pid=13141] [search-launcher] [process-runner]
Verify kvstore status:
[splunker@ip-x-x-x-x ~]$ sudo /opt/splunk/bin/splunk show kvstore-status
Splunk username: admin
Password:
This member:
backupRestoreStatus : Ready
date : Wed Jan 19 03:33:19 2022
dateSec : 1642563199.319
disabled : 0
guid : 1FFA6400-1ADB-4897-88BB-352FA36FA956
oplogEndTimestamp : Wed Jan 19 03:33:11 2022
oplogEndTimestampSec : 1642563191
oplogStartTimestamp : Wed Jan 19 02:56:16 2022
oplogStartTimestampSec : 1642560976
port : 8191
replicaSet : 1FFA6400-1ADB-4897-88BB-352FA36FA956
replicationStatus : KV store captain
standalone : 1
status : ready
storageEngine : mmapv1
KV store members:
x.x.x.x:8191
configVersion : 1
electionDate : Wed Jan 19 03:23:11 2022
electionDateSec : 1642562591
hostAndPort : 127.0.0.1:8191
optimeDate : Wed Jan 19 03:33:11 2022
optimeDateSec : 1642563191
replicationStatus : KV store captain
uptime : 608
And as expected the permissions of /opt/splunk/var/lib/splunk/kvstore/mongo remain unchanged. By changing the permissions manually:
[splunker@ip-x-x-x-x mongo]$ chmod 650 splunk.key [splunker@ip-x-x-x-x mongo]$ ls -l total 1310736 -rw-r-x---. 1 splunker splunker 88 Jan 19 02:56 splunk.key
The kvstore fails and the mongod.log shows the ACCESS errors stated in the case:
[splunker@ip-x-x-x-x mongo]$ sudo /opt/splunk/bin/splunk show kvstore-status
Your session is invalid. Please login.
Splunk username: admin
Password:
This member:
backupRestoreStatus : Ready
disabled : 0
guid : 1FFA6400-1ADB-4897-88BB-352FA36FA956
port : 8191
standalone : 1
status : failed
storageEngine : mmapv1
The logs /opt/splunk/var/log/splunk/mongod.log
I CONTROL [signalProcessingThread] shutting down with code:0 W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release. I ACCESS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open
Simply by changing the permission back to -rw------- for the splunk.key the kv store is able to start normally. At this point, it has been shown that Splunk itself cannot change the splunk.key permissions by itself. At first, the workaround proposed was changing the ownership of the splunk.key to the root user. This worked, and the ACCESS errors stoped appearing in the log. Subsequently, the kvstore was not able to start still, checking the logs again:
I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided I CONTROL [initandlisten] ** Please specify an sslCAFile parameter. I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. I CONTROL [initandlisten] I CONTROL [initandlisten] I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. I CONTROL [initandlisten] ** suggest launching mongod like this to avoid performance problems: I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] I CONTROL [initandlisten] I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. I CONTROL [initandlisten] ** suggest setting it to 'never' I CONTROL [initandlisten] I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. I CONTROL [initandlisten] ** suggest setting it to 'never' I CONTROL [initandlisten] F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details. I NETWORK [initandlisten] shutdown: going to close listening sockets... I REPL [initandlisten] shutdown: removing all drop-pending collections... I REPL [initandlisten] shutdown: removing checkpointTimestamp collection... I REPL [initandlisten] shutting down replication subsystems W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startup() finished. Shutting down without cleaning up the replication system I STORAGE [initandlisten] shutdown: waiting for fs preallocator... I STORAGE [initandlisten] shutdown: final commit... I JOURNAL [initandlisten] journalCleanup... I JOURNAL [initandlisten] removeJournalFiles I JOURNAL [initandlisten] old journal file will be removed: /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0 I JOURNAL [initandlisten] Terminating durability thread ... I JOURNAL [journal writer] Journal writer thread stopped I JOURNAL [durability] Durability thread stopped I STORAGE [initandlisten] shutdown: closing all files... I STORAGE [initandlisten] closeAllFiles() finished I STORAGE [initandlisten] shutdown: removing fs lock... I CONTROL [initandlisten] now exiting I CONTROL [initandlisten] shutting down with code:62
Resolution
The above was solved by performing a kvstore cleaning [REF 2] as follows :
- Stop the search head that has the stale KV store member.
- Run the command splunk clean kvstore --local.
- Restart the search head. This triggers the initial synchronization from other KV store members. (purple does not apply because is standalone)
- Run the command splunk show kvstore-status to verify synchronization.
This solves the issue and the kvstore is ready to go, then perform a restart and the permission issue arises again, this is solved by locking the splunk.key permissions:
chattr +i /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
It was recommended by the TSE to look for any other service that may be changing the permissions and ownership of these files.
Related Links
REF : ResyncKVstore
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)