Universal Forwarder with Structured Data and INDEXED_EXTRACTIONS - Props/Transforms Forwarding not working

Applicability

Description

1) The below example (from Keep specific events and discard the rest) of forwarding indexed extractions was found to be working on a Heavy Forwarder (HF), however not on a Universal Forwarder (UF). 

props.conf

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

2) The configuration was the same on both - so why will this work on the Heavy Forwarder and not the Universal Forwarder? The text from the documentation below was interpreted as a Universal Forwarder can perform the same functionality and configuration as long as this is INDEXED_EXTRACTIONS of structured data.

Caveats for routing and filtering structured data
"The forwarded data must arrive at the indexer already parsed. To achieve this, you must also set up props.conf on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS and any other parsing, filtering, anonymizing, and routing rules. Universal forwarders are capable of performing these tasks solely for structured data."

Field extraction settings for forwarded structured data must be configured on the forwarder
"If you want to forward fields that you extract from structured data files to another Splunk instance, you must configure the props.conf settings that define the field extractions on the forwarder that sends the data. This includes configuration of INDEXED_EXTRACTIONS and any other parsing, filtering, anonymizing, and routing rules. Performing these actions on the instance that indexes the data will have no effect, as the forwarded data must arrive at the indexer already parsed."

Investigation

You can use props.conf and transforms.conf on the UF for structured data INDEXED_EXTRACTIONS. 

However, the example provided in Keep specific events and discard the rest can be read as applicable to UF, however this example will not work on the UF.

The UF does not have an indexQueue and if this is applied, events will not be forwarded (See referenced links for the MASA diagrams for UF/HF/IDX flow).

Resolution

You can use either of the below options of using _TCP_ROUTING or changing the output queue on the UF:

For Example:

a) _TCP_ROUTING

props.conf

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf:

[setnull]
REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT = noGroup[setparsing]
REGEX = bbbb
DEST_KEY=_TCP_ROUTING
FORMAT = default-autolb-group

See Routing and filtering capabilities of forwarders for further information on _TCP_ROUTING

b) Queue change

If you cannot use _TCP_ROUTING, you can change to the parsingQueue as per the example below. FORMAT = parsingQueue

props.conf

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue[setparsing]
REGEX = name\=splunk
DEST_KEY = queue
FORMAT = parsingQueue

Related Links

Keep specific events and discard the rest

Caveats for routing and filtering structured data

Field extraction settings for forwarded structured data must be configured on the forwarder

MASA Diagrams

Routing and filtering capabilities of forwarders