Cleaning up conflicting configuration settings

Applicability

Description

Examples of conflicting configurations resulting in unwanted results can include (but are not limited to) the following:

• Some data not being ingested by forwarders
• Data not reaching indexers
• Event duplication or missing events

Investigation

While the issues listed above can be caused by a wide range of issues, a simple check of configuration settings on impacted instances can rule out whether the issue is caused by configuration settings.

Because Splunk collects configs for many settings from multiple locations based on precedence and collates them into a single file, understanding how configurations impact the final output can become challenging.

• Except for clustered indexers, all Splunk instances consider $SPLUNK_HOME/etc/system/local config files with highest priority¹
• Clustered indexers
  • consider $SPLUNK_HOME/etc/peer-apps (or slave-apps for most pre 9.x versions) configurations with highest priority, THEN $SPLUNK_HOME/etc/system/local.
  • within their respective peer-apps or slave-apps directory (whichever is in use² - the other will be ignored entirely), indexers consider app configurations based on lexicographic order
  • within a specific folder location, settings in their local directory will take precedence over settings in their default locations
  • Splunk documentation says configs provided in an app/user context follow REVERSE lexicographical order. Applicable to this discussion: app.conf

For example, configurations on files such as server.conf, web.conf, limits.conf, inputs.conf and outputs.conf on all Splunk and Splunk Forwarder instances (excluding clustered indexers) is compiled in the following order of precidence:

1. $SPLUNK_HOME/etc/system/local
2. In reverse lexicographical order: $SPLUNK_HOME/etc/apps/<app-name-in-lexicographical-order>/local
3. In reverse lexicographical order: $SPLUNK_HOME/etc/apps/<app-name-in-lexicographical-order>/default
4. $SPLUNK_HOME/etc/system/default

Alternatively, configuration files such as those listed above are considered differently in clustered indexers:

1. In reverse lexicographical order: $SPLUNK_HOME/etc/peer-apps/<app-name-in-lexicographical-order>/local
2. $SPLUNK_HOME/etc/system/local
3. In reverse lexicographical order: $SPLUNK_HOME/etc/apps/<app-name-in-lexicographical-order>/local
4. In reverse lexicographical order: $SPLUNK_HOME/etc/peer-apps/<app-name-in-lexicographical-order>/default
5. In reverse lexicographical order: $SPLUNK_HOME/etc/apps/<app-name-in-lexicographical-order>/default
6. $SPLUNK_HOME/etc/system/default

¹Some configurations such as app.conf stay within the app itself and are not modified by $SPLUNK_HOME/etc/system/default or /local at all 
²Use of peer-apps vs master-apps in indexer clusters is based on configuration settings and/or Splunk version, which is further complicated when cluster participants employ a mixture of mode = peer/slave (indexers), mode = master/manager (Cluster Manager) and manager_uri/master_uri (indexers and search heads). It is suggested to standardize nomenclature across ALL cluster participants.

When configurations are supplied by multiple applications (as is common for inputs.conf or outputs.conf), this becomes much more complicated.

Splunk offers the btool command as a method for determining exactly what configurations are getting picked up by Splunk, and from exactly what files. It works like this:

$SPLUNK_HOME/bin/splunk btool <config-file-name-without-the-file-extension> list

• adding --verbose after "list" will output the exact file producing a specific attribute
• adding a stanza name or partial stanza name after list (or after list --verbose) will produce an output for just that stanza or stanza's containing that string (no wildcard required)
• If the output is too large to view via CLI, adding >> /path/to/file.name will output to a file instead.

Sample commands:

• server.conf [clustering] and [shcluster] stanzas: $SPLUNK_HOME/bin/splunk btool server list cluster
• full outputs.conf with source file locations: $SPLUNK_HOME/bin/splunk btool outputs list --verbose
• full inputs.conf with source file locations, output to a file: $SPLUNK_HOME/bin/splunk btool inputs list --verbose >> /tmp/inputs-output.txt
• indexes.conf volume stanzas only with source file locations $SPLUNK_HOME/bin/splunk btool indexes list --verbose volume

Resolution

Determine what your specific configuration file to look like, then compare this with the outputs of the btool command above, and make adjustments accordingly.

Common issues:

• $SPLUNK_HOME/etc/system/local variation of a config file is overriding the desired file (could apply to any instance except clustered indexers)
• A specific app's config settings are taking precedence over another app (and/or as with clustered indexers, is overriding $SPLUNK_HOME/etc/system/local variation of that file)
• an app's default configs are still in place after removing its local config settings, when the intention was actually to disable the app itself. An app can be disabled by:
  • running this command from the applicable app's CLI (for a standalone app, not deployed by a management instance such as Deployment Server, Deployer or Cluster Manager):
    

    $SPLUNK_HOME/bin/splunk disable app [app_name] -auth <username>:<password>

  • Or, by setting the following in its local/app.conf file:
    

    [install]
    state = disabled

Related Links

• Splunk Documentation: Where to find the configuration files
• Managing app objects
• Splunk community article: Disabling a custom app