Permission Denied after a Restart of the Splunk Service

Applicability

Splunk Enterprise

Description

When trying to start the Splunk Instance, multiple errors appear like below:
 

$ sudo ./splunk start
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: 
Permission deniedChecking http port [8000]: Cannot initialize: /opt/splunk/etc/apps/splun 
k_instrumentation/metadata/local.meta: Permission denied

Investigation

The complete error that is displayed when trying to start Splunk with the $SPLUNK_HOME/bin/splunk start command:

$ sudo ./splunk start
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission deniedSplunk> Winning the War on ErrorChecking prerequisites...
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Checking http port [8000]: Cannot initialize: 
/opt/splunk/etc/apps/splun k_instrumentation/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot initialize: 
/opt/splunk/etc/apps/splun k_instrumentation/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: 
Cannot initialize: /opt/splunk /etc/apps/splunk_instrumentation/metadata/local.meta: 
Permission denied
open
Checking kvstore port [8191]: Cannot initialize: 
/opt/splunk/etc/apps/sp lunk_instrumentation/metadata/local.meta: Permission denied
open
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbu cket antivirus azure 
cim_modactions cim_summary endpoint_summary fireeye_sb hist ory ioc main network_acs 
network_checkpoint network_fortinet network_ise network _juniper network_paloalto 
network_proxy network_vpn network_wlc notable notable_ summary os_dhcp os_dns os_esxi 
os_linux os_windows rapid7 risk salesforce summar y threat_activity trustar windows_defender 
zscaler
Done
Bypassing local license checks since this instance is configured with a remote license masterChecking filesystem compatibility... Done
Checking conf files for problems...
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Cannot open file to check: /opt/splunk/etc/apps/splunk_instrumentation/local/tel emetry.conf
Your indexes and inputs configurations are not internally consis tent. For more information, 
run 'splunk btool check --debug'
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Done
Checking default conf files for edits...
Validating installed files against hashes from 
'/opt/splunk/splunk-7.3.4 -13e97039fb65-linux-2.6-x86_64-manifest'
All installed files intact.
Done
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Checking replication_port port [8080]: Cannot initialize: 
/opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
open
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
All preliminary checks passed.Starting splunk server daemon (splunkd)...
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.me ta: 
Permission denied
Done

We can check the user and group owner of the files by running the following:

ls -l /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta

In the output below we can see that the file is owned by user1 and group1

-rw------- 1 user1 group1 69 Jan 23 03:01 /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta

It is found that user1 and group1 is not the expected user that was configured to run the Splunk service.

Resolution

To resolve the issue the permissions need to set to the user that has been configured to run the Splunk service, this is typically "splunk"
1. Stop Splunk:

cd $SPLUNK_HOME
./bin/splunk stop

2. Modify permissions using the following command replacing group:user with the required group and user.

chown -R group:user $SPLUNK_HOME

For example, if the user and group are splunk this would be:

su -
chown -R splunk:splunk $SPLUNK_HOME
su splunk

3. Start Splunk

./bin/splunk start

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release