Deploying apps to a Cluster Manager "Manager-Apps" directory may cause ERRORS or a Cluster Master Restart

Applicability

Splunk Enterprise

Description

When deploying apps from the Deployment Server to a Cluster Manager's "Manager-Apps" directory (which can then be distributed to the index cluster), an ERROR is logged in splunkd.log, " Failed to install app".

splunkd.log:

01-01-2019 10:10:00.001 +0000 INFO DeployedApplication - Checksum mismatch 0 <> 13853077245268154211 for app=NEW-APP01. 
Will reload from='xxxxx:8089/services/streams/deployment?name=default:ServerClass-ClusterManager:NEW-APP01'
01-01-2019 10:00:00.002 +0000 INFO DeployedApplication - Downloaded url=xxxxxxxx:8089/services/streams/deployment?name=
default:ServerClass-ClusterMaster:NEW-APP01 to file='/opt/splunk/var/run/ServerClass-ClusterManager/NEW-APP01-1546354800.bundle' sizeKB=10
01-01-2019 10:00:00.003 +0000 INFO DeployedApplication - Installing app=NEW-APP01 to='  /opt/splunk/cmanager/splunk/etc/manager-apps/NEW-APP01'
01-01-2019 10:00:00.004 +0000 ERROR DeployedApplication - Failed to install app=/opt/splunk/etc/manager-apps/NEW-APP01; reason=
Application does not exist: NEW-APP01
01-01-2019 10:00:00.005 +0000 ERROR DeployedServerclass - name=ServerClass-ClusterManager Failed to install app=NEW-APP01

Investigation

The Cluster Master's deploymentclient.conf configuration is configured correctly to redirect the app repository to the master-apps directory:

[deployment-client]
 repositoryLocation = $SPLUNK_HOME/etc/manager-apps
 serverRepositoryLocationPolicy = rejectAlways[target-broker:deploymentServer]
 phoneHomeIntervalInSecs = 60
 targetUri = https://Deployment-Server.domain.com:8089

Reviewing the /opt/splunk/etc/master-apps directory on the Cluster Master finds the app is in fact there:

[splunk@cmanager ~] cd /opt/splunk/etc/manager-apps/
[splunk@cmanager manger-apps] ls -ltr
total 2
drwxr-xr-x  3 splunk    splunk    01 Jan  1  2019 NEW-APP01
drwxr-xr-x  3 splunk    splunk    01 Jan  1  2019  _cluster

 
However, the Deployment Server's serverclass.conf configuration for this serverclass.conf does not appear to be correct for this scenario:

[serverClass:ClusterManagers]
filterType = blacklist
restartSplunkd = true
blacklist.0 = *
whitelist.0 = cmaster.domain.com
[serverClass:ClusterManagers:app:NEW-APP01]
stateOnClient = enabled

Resolution

In serverclass.conf, please include the following at the top level for the Cluster Master's server class to prevent the reported ERROR:

stateOnClient = noop

More info on this setting

• serveclass.conf:

stateOnClient = enabled | disabled | noop
* If set to "enabled", sets the application state to enabled on the client,
  regardless of state on the deployment server.
* If set to "disabled", set the application state to disabled on the client,
  regardless of state on the deployment server.
* If set to "noop", the state on the client will be the same as on the
  deployment server.
* Can be overridden at the serverClass level and the serverClass:app level.
* Defaults to enabled.

The recommended settings for a serverclass in this scenario would be:

[serverClass:ClusterMasters]
stateOnClient = noop
restartSplunkWeb = 0
restartSplunkd = 0
[serverClass:ClusterMasters:app:NEW-APP01]
[serverClass:ClusterMasters:app:NEW-APP02]
[serverClass:ClusterMasters:app:NEW-APP03]
[serverClass:ClusterMasters:app:NEW-APP04]

It is important to configure both restartSplunkd and restartSplunkWeb (legacy) to 0 (false) in order to prevent the restart of those services on the Cluster Manager during the application deployment.  Restarting those service on the Cluster Manager is unnecessary and could lead to unintended consequences.
 

Related Links

Splunk Documentation

serverclass.conf
Serverclassconf

Splunk Answers

DeployedApplication - Failed to install ... reason=Application does not exist: APPNAME 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release