Heavy forwarder not sending internal logs to Indexers

Applicability

Description

The internal logs are not being forwarded from the heavy forwarder to the Indexers, but from the same heavy forwarder, other logs are being forwarded.

Investigation

1. Review the forwardedindex.2.whitelist present in outputs.conf setting on both the Universal Forwarder (UF) and Heavy Forwarder (HF) and verify that _internal index is present.

An example in a universal forwarder (UF), under $SPLUNK_HOME/etc/apps/all_forwarder_outputs/local/outputs.conf

[tcpout]
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

In the HF, under $SPLUNK_HOME/etc/apps/cts_all_forwarder_outputs/local/outputs.conf 

[tcpout]
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

2. The below warning is observed in splunkd.log:

WARN TcpOutputProc [30277 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=xxx.xxx.xxx.xxx inside output group primary_indexers from host_src=<hostname> has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

3.  In  $SPLUNK_HOME/etc/system/local/outputs.conf on HF,  it is observed that the default group is set to the incorrect target group name :

4. The correct target group name is set under $SPLUNK_HOME/etc/apps/<app_name>/local/outputs.conf on HF :

# BASE SETTINGS
[tcpout]
defaultGroup = primary_indexers ( this is correct)
compressed = true
forceTimebasedAutoLB = true
forwardedindex.filter.disable = false
forwardedindex.2.whitelist = (_audit|_introspection|_internal)[tcpout:primary_indexers]
server = xxx.xxx.xxx.xxx:9997, xxx.xxx.xxx.xxx:9997, xxx.xxx.xxx.xxx:9997

5. As per Splunk configuration file precedence, $SPLUNK_HOME/etc/system/local/outputs.conf is taking precedence over $SPLUNK_HOME/etc/apps/<app_name>/local/outputs.conf. More information about file precedence under Configuration file precedence.

Resolution

1. In HF, navigate to $SPLUNK_HOME/etc/system/local/outputs.conf and comment out, or delete, the incorrect parameter :

[tcpout]
#defaultGroup=indexerTest

2. Save the changes and restart the Splunk service in the HF to apply the changes.

The _internal logs should be forwarded from the HF to the Indexers.

Related Links

Why are _internal logs from heavy forwarder(HF) not getting to indexers

Why is heavy forwarder not sending _audit and _internal logs to indexer?