Ingested Events Gone Missing - Resolving Timestamp Parsing Issues in Splunk: A Case Study on WinHostMon

Applicability

Description

When monitoring disk data from WinHostMon, significant discrepancies were observed between event timestamps (_time) and ingestion timestamps (_indextime). Some events had their _time set to dates far in the past, resulting in indexing delays and misleading search results.

This issue arose because the numeric value in the 'FreeSpaceKB' field was being misinterpreted as epoch time due to the default 'DATETIME_CONFIG=\etc\datetime.xml' setting in props.conf.

Investigation

Symptoms:
- Search results for 'index=windows sourcetype=WinHostMon' returned no results at all.
- Search for all time for the same SPL showed large delays between `_time` and `_indextime`.
- Events with `_time` far in the past appeared despite no issues with ingestion or indexing processes.
- The issue persisted even though the ingestion and indexing occurred on the same server.

Key Findings:
- Splunkd internal log found events ingested OK:

INFO ExecProcessor [4664 ExecProcessorSchedulerThread] - message from “”C:\Program Files\Splunk\bin\splunk-winhostinfo.exe”" splunk-winhostinfo - Output event Name=“D:“\nDriveType=“fixed”\nTotalSpaceKB=2147483019\nFreeSpaceKB=1607839214\nFileSystem=“NTFS”\n

- This log confirmed the correct ingestion of events, but searches showed incorrect `_time` values.
- Inspection of the raw event logs revealed the field 'FreeSpaceKB=1607839214', that corresponds to the epoch time for December 13, 2020.
- Analysis of the props.conf file showed DATETIME_CONFIG pointing to '\etc\datetime.xml', allowing Splunk to mistakenly interpret 'FreeSpaceKB' as a timestamp.

Resolution

1. Updated Configuration:
   - Modified props.conf for the WinHostMon source type:

     [WinHostMon]
     DATETIME_CONFIG = CURRENT

The change instructed Splunk to use the ingestion time (_indextime) as the event's timestamp instead of parsing one from the raw data.

2. Validation Steps:
   - Restarted Splunk to apply the updated configuration.
   - Reindexed test data to verify the fix.
   - Observed that events now had accurate _time values aligned with _indextime.

Key Takeaways
- Always validate how Splunk parses timestamps when encountering indexing delays or timestamp anomalies.
- Use `DATETIME_CONFIG = CURRENT` when events lack valid timestamp fields or when numeric fields might be misinterpreted.