Splunk Universal Forwarder / Monitor Input log files not ingesting due to ignoreOlderThan mtime Boundary

Applicability

Description

On a standalone Splunk instance, a recursive directory monitor input was failing to reliably ingest log4j application logs from a deeply nested directory structure containing both active and archived (rotated) files. The data resided on an externally mounted volume, and the input was responsible for forwarding audit relevant .log files into a specific index.

Investigation

 Over the course of the investigation, the problem revealed itself in three progressive phases rather than as a single failure, which made it initially difficult to isolate a single root cause.

 In the first phase, newly added stanzas broke previously working inputs, and the instance reported File descriptor cache is full warnings. This was traced to a combination of invalid inputs.conf attributes (checkMethod, scanFreq, detectTruncation, alwaysOpenFile) and the default file-descriptor limit being exhausted.

• Invalid inputs.conf attributes: checkMethod, scanFreq, detectTruncation, alwaysOpenFile = true (not valid for monitor inputs in this context).
• Use of the legacy ... wildcard instead of recursive = true.
• Default max_fd = 100 exhausted due to the large number of files in the recursive tree.

What we did:

• Removed all invalid attributes.
• Replaced ... wildcard with recursive = true.
• Set initCrcLength = 2048 to better differentiate log4j files with identical header boilerplate.
• Added blacklist = (/lock/).
• Increased max_fd = 300 in limits.conf.
• Aligned the stanza with a known-working stanza already running on the same instance.
• Validated with btool, then restarted Splunk.

  In the second phase, after the configuration was cleaned up, files that physically existed on disk still were not being tracked, btprobe --validate returned record not found, and splunk list monitor did not list them. This was ultimately tied to a filesystem mount (fstab) change that was silently preventing the TailingProcessor from properly stat-ing and tracking files; reverting the change and remounting the volume restored discovery. So:

1. We needed to confirm Splunk service account can actually read the files: 

sudo -u splunk find /data/curss-prod/ -type f -name "*.log" 2>&1 | head -5

 If result: files listed fine, so it was not a basic permissions problem. 

2.  Probe the fishbucket for a known-existing file: 

./splunk cmd btprobe -d /data/splunk/var/lib/splunk/fishbucket/<db> \
  --file <full_path_to_log_file> --validate

 If:

validation result: OK
record not found

 Then, Splunk had never created a fishbucket record, meaning the TailingProcessor had never successfully tracked the file.

3. Check whether Splunk was actively monitoring the path: 

./splunk list monitor 2>/dev/null | grep curss-prod

 If the result is empty, then it confirmed the path was not being monitored despite a valid stanza.

 This means that a prior change to /etc/fstab for the data drive was silently interfering with how the TailingProcessor could stat and track files on the mounted volume. The stanza was valid and the user could read files, but Splunk's file-tracking layer couldn't establish the records.  To resolve this:

• Revert the fstab change for the affected drive.
• Unmount and remount the volume.
• Restart Splunk.
• Result: Files across all folders (including archived/ subdirectories) began appearing. The recursive = true setting correctly traversed nested subdirectories, confirming no separate stanzas were needed for log files

In the third and final phase, even after recent data began flowing correctly, a significant portion of files, roughly two-thirds in a daily sample (e.g., 52 of 151 files), remained missing. Critically, these were predominantly the log files required for audit compliance, and notably, no CRC collision warnings appeared in index=_internal, which ruled out the most common cause of missing-file symptoms. So:

1. Rule out CRC collisions:

index=_internal sourcetype=splunkd component=TailingProcessor host=<hostname> earliest=-2h
("CRC" OR "duplicate" OR "already" OR "seekcrc")
| head 50

2. List what Splunk actually ingested (Splunk Web → export to CSV): 

index=<index> host=<hostname> earliest=-90d 
| stats count by source 
| sort source

 This revealed the missing files clustered in archived/ directories with filenames dated late March / early April 2026 — right at the ignoreOlderThan = 65d cutoff. 

 ignoreOlderThan = 65d filters files based on modification time (mtime), not the date embedded in the filename. Archived files with old mtimes were silently skipped during discovery and never revisited — and crucially, ignoreOlderThan writes no error to splunkd.log, which is why nothing obvious appeared in any log. 

Resolution

 By comparing the Splunk ingestion output against the raw filesystem listing, the missing files were found to cluster around the ignoreOlderThan = 65d boundary. The root cause was that ignoreOlderThan filters files by their modification time (mtime), not by the date embedded in the filename, so archived files with old mtimes were being silently skipped at discovery time, with no error written to splunkd.log. The resolution involved widening the ignoreOlderThan threshold, refreshing the mtime of the audit-critical archived .log files via touch (safe because Splunk identifies files by CRC and path, not mtime), and restarting Splunk, recovering all required files without producing any duplicate events.

Resolution:

1. Widened the threshold in inputs.conf: 

[monitor://<path>]
disabled = false
crcSalt = <SOURCE>
initCrcLength = 2048
ignoreOlderThan = 90d
recursive = true 

2. Refreshed mtime on audit-critical archived .log files:

sudo -u splunk find <path> -path "*<log_name>*.log" ! -name "*_error.log" -mtime +60 -exec touch {} \;

3. Restart Splunk