Search head cluster resync overwrites node-specific inputs.conf host setting

Applicability

Description

Enterprise Security includes the following setting in DA-ESS-ThreatIntelligence/default/server.conf:

[shclustering]
conf_replication_include.inputs = true

This allows Threat Intelligence Framework input changes made through the user interface or REST API to replicate across search head cluster members.

When splunk resync shcluster-replicated-config is run on a member, the captain's replicated configuration bundle can overwrite the target member's /opt/splunk/etc/system/local/inputs.conf. If that file contains a static [default] host value, the target member can begin attributing its internal events to the captain's host name.

Investigation

1) Inspect DA-ESS-ThreatIntelligence/default/server.conf for the following setting:

[shclustering]
conf_replication_include.inputs = true

2) Confirm whether the affected member's /opt/splunk/etc/system/local/inputs.conf contains a static host value:

[default]
host = <host_name>

3) Correlate the changed host value with execution of splunk resync shcluster-replicated-config or search head cluster member addition.

Resolution

1) Restore the correct node-specific [default] host value in /opt/splunk/etc/system/local/inputs.conf, or set it with:

splunk set default-hostname <actual_hostname>

2) Restart splunkd on the affected search head after restoring the setting.

3) Optional: To reduce the risk of an incorrect static value after a future resync, configure the default host dynamically:

[default]
host = $decideOnStartup

Note: Do not disable conf_replication_include.inputs without evaluating Threat Intelligence Framework requirements. Setting it to false prevents inputs.conf replication, but can prevent automatic replication of threat intelligence inputs configured through the UI or REST API.