[Splunk ES] Duplicated Notables

Applicability

Description

This KB will address concerns related to duplicated notables, which occur when they are indexed twice: once in the indexer and again in the Search Head.

Investigation

1. To determine if the root cause is indeed the notable being indexed twice, run the following SPL and note the fields 'splunk_server' and 'host'.

index=notable 
| table _time search_name splunk_server host

2. The 'splunk_server' should be the hostname of the indexer, while the 'host' should be the hostname of the Search Head. If the value of 'splunk_server' matches the Search Head's hostname, it would indicate that indexing is enabled on the Search Head.

3. This can be confirmed by checking the local configuration of the search head using the following btool command.

$SPLUNK_HOME/bin/splunk btool --debug outputs list | grep indexAndForward

Resolution

1. Remove the indexAndForward properties or set it to false for the Search Head in 

$SPLUNK_HOME/etc/system/local/outputs.conf

like so:

[tcpout]
indexAndForward = false

2. Restart Splunk

$SPLUNK_HOME/bin/splunk restart