Enterprise Security - missioncontrol updating thousands of times per hour causing replication issues

Applicability

Description

Splunk search head cluster stability is impacted, and health.log is reporting configuration replication issues. The problem is not easy to spot on the surface, but searching conf.log can uncover the issue quickly. This is a known issue with Enterprise Security 8.6.1 and 8.7.0. 

Investigation

Review conf.log. Are there a lot of apps updating configurations?

index=_internal source=*conf.log* source=*/conf.log* data.task=addCommit  inputs 
| rex field=_raw "asset_uri\":\[\"\w+\",\"(?<APP_GETTING_UPDATED>\w+)\",\"(?<CONF>\w+)\",\"(?<STANZA>.+)\"\]"
| stats count by APP_GETTING_UPDATED  CONF STANZA data.payload.children.disabled.alias  data.payload.children.disabled.value 
| sort - count

Look at the results with a line chart visualization. If missioncontrol shows a large amount of updates over the span of an hour, the environment is affected by a known issue that is affecting search head cluster configuration replication. In some cases, the count can get in to the thousands on this graph. 

Resolution

OPTION 1:

Apply the following configuration to missioncontrol/local/inputs.conf and restart Splunk:

[mc_setup_tasks_manager://main]
disabled = 1

OPTION2:

A fix for this issue is in Enterprise Security version 8.8. Upgrading to 8.8 or later will also resolve the issue.