Troubleshooting file monitor issues on Universal Forwarder

Applicability

Description

Upon completing the required configuration in inputs.conf to monitor files in an instance with Splunk Universal Forwarder installed, the logs are not ingested in the environment.

Investigation

In order to confirm which is the issue, please run the following commands: 

1. Check the input status by running the following command: 

$SPLUNK_HOME/bin/splunk list inputstatus 

Here is the meaning of the output:

• file position: The file pointer position that Splunk is currently reading at. If this is the same as file size, that means Splunk reach end of file (EOF).
• file size : Total file size of the monitored file.
• parent : If you are monitoring a directory, this tell you from which monitoring stanza the file is come from.
• percent : The progress of the monitoring. If it is less than 100%, Splunk will re-visit the file again.
• type: The monitoring status. It can be ‘finished reading’, ‘open file’, ‘missing’, ’directory’, 'reading (batch)'.
  • reading (batch): If you saw 'reading (batch)', that means the file that you are monitoring is greater than min_batch_size_bytes under limits.conf (default is 20M in size) and Splunk is using batch processor to process the file. It is a single thread process and will process one file at a time. (TailingProcessor is multi thread process).
  • Open file: means that the files are being tailed, file is open, which means tailreader is watching for new events.
  • Missing: logs or file intended to be monitor by Splunk was not found in the location (path) indicated in the monitor stanza.

2. Check the error messages by running this query 

index=_internal source=*splunkd.log tailreader ERROR

Check the paths detailed in the error messages.

3. Check if FIPS is enable on the instances on the environment by running this command: 

$SPLUNK_HOME/bin/splunk show fips-mode

If FIPS mode is on, the CLI command returns FIPS mode enabled. Otherwise, it returns FIPS mode disabled. 

4. Run this command and check that the file monitor stanza is complete and without typos:

$SPLUNK_HOME/bin/splunk btool inputs list --debug | grep local

Resolution

1. Follow  the instructions according to the result of the type field from the output of inputstatus command :

•  Reading (batch):  the batch reader is used when the file is over 20 MB in size. Otherwise, the regular tailing processor queue is used. The batch reader only processes one file at a time, while the tailing processor can do many. The limit exists to prevent a bunch of large files for using up all slots and starving out new smaller files but the threshold can be changed in limits.conf by adjusting the argument min_batch_size_bytes with the required value.
• Open file: means that the files are being tailed, file is open, which means tailreader is watching for new events.
  This behavior is related with the rate at which logs are written and then rotated. Splunk is unable to match this load.  Try to track the global counters and log forwarder statistics to prove that the log rate is too high for Splunk to be able to complete each forward. The solution should be to increase the size, CPU and memory of the instance.
• Missing: check the path detailed here, the file to be monitored should have been moved or deleted. Place again in the path or update the monitor stanza with the right path.

2. If the error message indicates issue with the /var/log directory, then it could be due to permission issues since /var/log is not typically open to non-root accounts. The easiest way to fix this is to set an Access Control List on the /var/log directory for the Splunk or Splunkfwd user (or the one you are using), for example:

sudo setfacl -Rdm u:splunk:r /var/log
sudo setfacl -Rd -m u:splunkfwd:r /var/log

-d and -R flag to the setfacl command to set the defaults for the directory and make the change recursive.

Check the ACL status of a file with the command below

 getfacl /var/log/ 

Splunk user account must be able to read files and directories configured for monitoring by Splunk.

More information about ACLs:
Ubuntu
RH

3. If FIPS is enabled (OS level or Splunk level) in the instance (universal forwarder, heavy forwarder or indexer), then it should be enabled everywhere. Check if the Operative system of the instance has FIPS enabled by default, if so, enable it on the rest of the instances.  For example, Linux 4 Rocky Linux release 8.9 (x86-64) Green Obsidian (build 4.18.0) has FIPS enabled by default. Check NIST Implementation Under test list for more information.

4. Add the sourcetype to the into the file monitor stanza in $SPLUNK_HOME/etc/system/local/inputs.conf, for example:

[monitor:///var/log]
disabled = 0
index = linux
sourcetype=var_logs

Once modified the inputs.conf, please restart the instance. It is necessary to add the argument sourcetype while monitoring a file because declaring the sourcetype is important both for searchability and for applying the relevant formatting for this type of data during parsing and indexing. The Splunk platform picks a source type based on various aspects of the data, as there is no default.
Examples of monitor input stanzas is the best way to have your input stanza validated.

Related Links

FIPS

FIPS in UF

Examples of monitor input stanzas