Not Receiving Data from New Monitor Input.

Applicability

Splunk Enterprise Heavy Forwarder

Description

A common approach to splunking syslogs is to have these log files sent to a volume on a splunk forwarder, which then has a series of monitor inputs.

These monitor inputs are then often delineated or broken down in directories by hostname or source.  E.g.

/data/host1/ 100.log 101.log 102.log
/data/host2/ 100.log 101.log 102.log
/data/host3/ 100.log 101.log 102.log

In this example a new syslog monitor had been added to the setup, but none of the data was being received by the Splunk indexers.

 

Investigation

How to identify it:

1. Check the inputs.conf file for the new data source and compare it to the existing working monitor inputs.
    The format was the same:

[monitor:///data/host3/*.log]
disabled = 0
host_segment = 2
index = newindex
sourcetype = newsourcetype

2.  On the Splunk search head check for data:  | index = newindex

3.  Check for any events related to the new host:  | index=_internal sourcetype = splunkd  hostname = host3

4.  One can also check ~/var/log/splunkd.log  on the Splunk forwarder  for any events related to the host:  
   

# more splunkd.log | grep host3

Resolution

In this case the splunkd.log on the forwarder reported the warning:

WARN FileSystemChangeWatcher - error reading directory "/data/host3/": Permission denied

The  /data/host3  directory had 700 permissions ( drwx------ )  compared to  /data/host1 and  /data/host2  directories which had  755 permissions ( drwxr-xr-x )

Ran:

# chmod 755 /data/host3

And confirmed that the data was now being indexed:  | index = newindex

 

Related Links

Multiple *nix files in one directory with a wildcard   https://docs.splunk.com/Documentation/Splunk/8.0.2/Data/Monitorfilesanddirectorieswithinputs.conf
 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release