Splunk Search Results Have Duplicate Fields for a Specific JSON-Formatted Sourcetype.

Applicability

Description

The searched data is JSON-formatted, and the sourcetype assigned to the data is/are defined by an app/add-on which is installed on the first Splunk node (instance) receiving the data, or indexers (in the case of HEC data in Splunk Cloud).

In the screenshot below you can see the fields being duplicated: 

Investigation

The sourcetype(s) has the INDEXED_EXTRACTIONS setting set to JSON. Now, the problem is that Splunk search heads by default have KV_MODE = auto. Then, the events already got their data extracted at index-time, but the search head is extracting it again at search-time, and that causes the duplicated fields on the search results.

This behavior is seen in any app that does not have KV_MODE disabled. That is why, when an app/add-on defines a sourcetype with INDEXED_EXTRACTIONS = JSON, it also sets KV_MODE =, that way the search head will not extract the fields at search-time, and the search results will show the index-time extracted fields.

Resolution

Disable the INDEXED_EXTRACTIONS setting by removing it from props.conf (since default value is not set) on the Splunk node ingesting the data, and enable KV_MODE = json on the search head. Or set  INDEXED_EXTRACTIONS=JSON and KV_MODE=none

Props.conf example solution 1:

[Sourcetype]
KV_MODE = json
AUTO_KV_JSON = false

Props.conf example solution 2:

[Sourcetype]
INDEXED_EXTRACTIONS=JSON 
KV_MODE = none
AUTO_KV_JSON = false

Note: When INDEXED_EXTRACTIONS = JSON for a particular source type, do not also  set KV_MODE = json for that source type. This causes the Splunk software to extract the JSON fields twice: once at index time, and again at search time.
 

Related Links

For more information about KV_MODE and props:
 Splunk Documentation: Props.conf

Splunk Answers

Why is my sourcetype configuration for JSON events with INDEXED_EXTRACTIONS making each extracted field multivalue with duplicate values?
Why is my sourcetype configuration for json events