Windows Universal Forwarder Restart Permissions Using Managed Service Account.

Applicability

Description

Splunk Universal forwarder (UF) instances on Windows server 2012R2 and Server 2019 are not restarted when a new deployment is pushed from the deployment server. The Universal forwarder was installed with a Group Service Managed Account.

The splunkd log file would show the instance being restarted:

9-29-2020 09:12:53.136 +0100 WARN DC:DeploymentClient - Restarting Splunkd...

But then never came back up.

Investigation

When the UF is restarted after a deployment server push, the restart is done via external process:
 

splunk _relaunch restart --accept-license --answer-yes --no-prompt --waitonpid=<MOTHERSHIP PID>

To confirm if it is a permissions issues by starting the process using the PsExec.exe (if sysinternals is installed) for example:

PsExec.exe -u domain\MsaAccount$ <cmd.exe>

To see the service's security descriptors using the sc sdshow command, for example:

sc sdshow SCManager
sc sdshow SplunkForwarder

The following documents outline how Splunk should be configured when using a Managed Service Account (MSA)

Use managed service accounts

Install Splunk with a managed system account

The above documents state:

• The MSA requires the same permissions as a domain account on the machine that runs Splunk Enterprise.
• The MSA must be a local administrator on the machine that runs Splunk Enterprise.

It was determined for the service to restart that BOTH the "SCManager" and "SplunkForwarder" process needed the following:

MSA 0x34 (SERVICE_QUERY_STATUS | SERVICE_START | SERVICE_STOP) and LCRPWP

Resolution

The issue was resolved by using the "sc.exe sdset" to change BOTH "SCManager" and "SplunkForwarder" such that the MSA's SID has an entry which allows SC_MANAGER_ALL_ACCESS (0xF003F).

The sdshow should then translate to A;;KA;;;<SID>