Unable to send events from CSV file by using input file monitoring

Applicability

Description

The inputs.conf and props.conf mentioned below are examples of the configuration setup on the Universal Forwarder (UF) - 

inputs.conf:

[monitor:///home/jisan/rolechange.csv]
_TCP_ROUTING = nfvGroup
sourcetype = application:idmappcsv
index = application
disabled = 0
crcSalt = <SOURCE>

props.conf:

[application:idmappcsv]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %m-%d-%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TIME_PREFIX = ^
category = Structured
disabled = false
pulldown_type = true
REPORT-blah = application_idm_csv
description = Comma-separated value format.

This is the content of the CSV file mentioned below - 

rolechange.csv:

22-02-2022 05:05:20,Ticket22,App22,Role22
10-01-2010 05:05:20,Ticket10,App10,Role10
11-01-2011 05:05:20,Ticket11,App11,Role11
12-01-2012 12:39:00,Ticket12,App12,Role12
13-10-2013 05:05:05,Ticket13,App13,ROle13
14-09-2014 09:19:19,Ticket14,App14,Role14
15-09-2014 09:19:19,Ticket15,App15,Role15
16-09-2014 09:19:19,Ticket16,App16,Role16
18-09-2014 09:19:19,Ticket18,App18,Role18

Investigation

For structured file extraction, props.conf should be set on the UF in the distributed environment. So, kept the props.conf on UF, but still, Splunk is not reading the CSV file.
 

Found the following messages in the splunkd.log:

+0600 DEBUG TailingProcessor - Skipping itemPath='/home/jisan/rolechange.csv', does not match path='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log' :Not a directory :Not a symlink

Added the below parameter in inputs.conf:

Navigated to /opt/splunk/etc/apps/GP_TA_IDM/local/inputs.conf

time_before_close=1

After adding the above parameter, partial logs are only ingesting from rolechange.csv into Splunk.

Resolution

1. Suggested changing the below parameter in props.conf:

2. Go to /opt/splunk/etc/apps/GP_TA_IDM/local/props.conf

3. Append the following parameter:- 

MAX_TIMESTAMP_LOOKAHEAD = 20

4. Restart the instance.

- Even though the timestamp has 19 characters MAX_TIMESTAMP_LOOKAHEAD did not work with the value of 19 because TIME_PREFIX was set to '^'.
 

- In props.conf, if TIME_PREFIX=^, it will be calculated as 1, and time and date will be calculated from 2 in MAX_TIMESTAMP_LOOKAHEAD i.e., timestamp extraction will be constrained to characters 2 through 20.
 

Related Links

Configure Timestamp Recognition