Ingest Actions not working properly. Redacted data appearing in the UI in interesting fields

Applicability

Description

Ingest Actions - 

The ingest actions feature is a data pipeline that runs on the indexer, heavy forwarder, or in Splunk Cloud. It executes before the data is written to the disk. The function is to mask, filter, and route the data to one or more destinations, including AWS S3 or another Splunk index. 

Ingest Actions should apply the changes at the time of indexing and these fields should ideally be masked despite the ‘index-time field extraction’ as both happen at the same time (as part of the indexing phase). The source is having an impact on this as the source is more on the “input phase” while the filed extraction and masking happen much later.

Investigation

With INDEXED_EXTRACTIONS setting present in the props.conf, set the parsing and transforming of the data on the instance that indexes data will have no effect as the forwarded data will arrive at the indexer already parsed. When the INDEXED_EXTRACTIONS is specified meaning that the forwarding data is already structured, hence it skips the parsing pipeline and indexes the incoming data with key-value pairs in the index.

INDEXED_EXTRACTIONS = <CSV|TSV|PSV|W3C|JSON|HEC>
* The type of file that Splunk software should expect for a given sourcetype, and the extraction and/or parsing method that should be used on the file.

As routing, filtering, and masking data is part of the parsing pipeline, the INDEXED_EXTRACTIONS defies the pipelines and indexes the forwarded data as it is set to JSON.

1. Tried disabling INDEXED_EXTRACTIONS and use the KV_MODE to auto and retrieve the relevant fields at search time, but it did not work.

KV_MODE = [none|auto|auto_escaped|multi|multi:<multikv.conf_stanza_name>json|xml]
* Used for search-time field extractions only.
* Specifies the field/value extraction mode for the data.

Resolution

The structured data, or INDEXED_EXTRACTIONS=JSON, maps key-value pairs at index time. This implies that the key-value pairs have already been mapped and stored; hence, removing the key-value pair from _raw does not remove the extraction during search time as the value has already been indexed, and the Splunk UI does not create an ingest action that removes an already indexed field.

Even the INDEXED_EXTRACTIONS specified in props.conf on the indexer, structured data that is forwarded to it isn't processed by the indexer upon arrival. Data that is forwarded skips the following indexer pipelines, making it unlikely for the indexer to parse that data: Since HEC event endpoint events are already structured, the search field time extraction with KV_MODE = _json and remove indexed extractions can be allowed.

The UI shows the raw data plus all fields that have been extracted at search time plus all so-called indexed fields that have been created at index time.

Using INDEXED_EXTRACTIONS, the fields get indexed before ingesting actions, the same way the index, source type, source, and host fields are observed, but it does not exist in the _raw data. That's because it is all in _meta which are stored as index-time fields.

Related Links

How data moves through Splunk deployments: The data pipeline

Ingest Actions for Splunk platform

Extract fields from files with structured data