Applicability

Description

How to identify it

Firstly confirm the delta between event time (_time) and Splunk indextime (_indextime), modify the following depending on which index and sourcetype is impacted. This example is querying Security channel event logging:

index=<splunk_index> sourcetype="WinEventLog:Security" host=<splunk_forwarder>
| fields _time, _indextime, host
| eval delay=(_indextime - _time)/60
| timechart avg(delay) by host

Investigation

Important things to verify:

• When did the indexing delay first appear?
• What changes, if any, where made to the deployment around the time the delay first appeared?
• Does latency increase over time (linear), or fluctuate?

Confirm the max and average Event Per Second (eps):

index=_internal source=*metrics.log* security per_source_thruput host=<splunk_forwarder>
| timechart max(eps) max(max_age)

If latency varies over time, a high EPS value may be indicative of bottlenecks in data throughput.

Resolution

Quick Fix

This option applies only if you want to get current events and don't care about missing historical data. The following must be performed one at a time on each problem forwarder:

1. Set current_only = 1 in $SPLUNK_HOME\etc\system\local\inputs.conf:
   

   [WinEventLog://Security]
   current_only = 1

2. Stop the forwarder
3. Rename the stale checkpoint file:
   

   ren "C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\modinput\WinEventLog\Security" Security.bak

4. Start the forwarder
5. Make sure a new Security checkpoint file is generated
6. Confirm the checkpoint & security logs are w/in a few hundred events of each other
7. Rename the $SPLUNK_HOME\etc\system\local\inputs.conf file to revert the current_only=1 setting back to default.
8. Restart Splunk
9. Search to confirm data is ingesting

Why performing regular UF reboots won't likely help

Rebooting the UF simply forces it to re-read checkpoints and briefly empty memory buffers. Because the event generation rate exceeds the processing/thruput limit, the forwarder will immediately start accumulating lag again within minutes of the restart.

Long Term Solutions

The quick fix requires manual intervention on each affected UF, which is not generally feasible in a large deployment (and some administrators don't have command line access to UF's) applied. On top of this, it's highly likely the UFs will fall behind again. Your best option may be to provide a long term solution. As every environment is different, there is no simple setting that can be applied to every scenario. However, the suggestions in this KB should work for many use cases. All config suggestions in this article require Splunk to be restarted on the UF in order to take effect, so ensure that custom app on the Deployment Server has restart Splunkd checked.

Try this first (limits.conf)

Open maxKBps 

Before tweaking the options below, look at the limits.conf/[thruput]maxKBps. The default setting on a UF is 256KB/s, which limits the forwarder to ~22 GB/day total network transmission. A busy server generating Security logs easily produces more than 256 KB/s, making it physically impossible for the forwarder to keep up.

So, first thing: Set this value to 0 (unlimited) in limits.conf to uncap the network bandwidth:

[thruput]
maxKBps = 0

As most Universal Forwarders are phoning home to a Deployment Server, best practice is to set this config via a deployment app so all forwarders receive it at once (first on a test environment if possible, then in production). Another option if a test environment is not available, is to deploy to a "test" serverclass of a small grouping of UF's and monitor before applying everywhere. The same applies for all suggestions offered in this KB Article.

Additional Options

If opening up the forwarders' thruput does not resolve the issue entirely, consider the following:

Specific to inputs.conf/[WinEventLog://Security]

Option#1 - Change the batch size

By default, Splunk grabs events in tiny batches (10 events) and updates the checkpoint file on disk frequently. On busy hosts, disk I/O on the checkpoint file becomes a major choke point. Setting batch_size = 500, delaying disk sync to 600s (checkpointInterval = 600), and setting suppress_checkpoint = true drastically increases the EPS (events per second) throughput of splunk-winevtlog.exe.

batch_size = 500
suppress_checkpoint = true
checkpointInterval = 600

Option#2 - Use renderXML to reduce the number of API calls

By default, Splunk issues an API call for each field within the event. Enabling renderXml will instead issue an API call per event, potentially improving performance: Display Windows Event Log events in XML

renderXml = true 

Option#3 - Filter which fields are suppressed to reduce data payload

This option is effective for raw speed, but make sure it meets audit requirements (if applicable).

These settings strip header fields out of the raw event before sending to reduce byte size and API rendering calls. If their SIEM, CIM models, or compliance dashboards rely on Keywords (e.g., Audit Success/Failure), TaskCategory, or Opcode, dropping these fields will break those detections.

suppress_sourcename = true
suppress_keywords = true
suppress_type = true
suppress_task = true
suppress_opcode = true

Option#4 - Enable the FormatMessage() API in the input stanza

evtFormatMessage() is the default API in most Windows deployments, which has an increased processing overhead when compared to the previous FormatMessage() API. Reverting to the older API may alleviate indexing delay. 

Do not set this if where renderXml = true. The legacy FormatMessage() API is only faster when parsing classic plain-text multi-line logs (renderXml = false), because it bypasses the heavy evtFormatMessage() DLL string lookup. When using renderXml = true, Splunk already reads directly from the modern EVTX API (EvtRender) in structured format without calling message formatting DLLs.

use_old_eventlog_api = true

Increase UF Ingestion Pipelines (server.conf)

By default, a forwarder runs a single parsing and ingestion pipeline. For multi-core machines ingesting heavy event volume, the pipeline set can be increased to allow parallel processing. This controls how many internal threads the forwarder uses to read data from local Windows APIs, package it, and push it onto the network.

On a UF the resource footprint of increasing ingestion pipelines from 1 to 2 is very lightweight compared to an Indexer or Heavy Forwarder, because the UF only does lightweight reading, batching, and network output (no regex transforms, field extraction, or indexing to disk).

If the Windows server has at least 4 CPU cores and 8 GB RAM, setting parallelIngestionPipelines = 2 is usually safe and will significantly help clear a heavy event backlog and prevent future lag. CPU usage may increase while the UF gets caught up but will drop down again once ingestion returns to normal levels.

[general]
parallelIngestionPipelines = 2

Outputs.conf queue size

If adjusting the input batching to 500, consider ensuring the TCP output queue has sufficient memory buffer so the network layer does not block the input worker:

[tcpout:myindexers]
maxQueueSize = 50MB

Check Windows Defender / Antivirus Exclusions

Security endpoint agents (e.g., Defender, CrowdStrike, Carbon Black) can aggressively inspect splunk-winevtlog.exe and lock the $SPLUNK_HOME\var\lib\splunk\modinput\WinEventLog\ directory. Ensure the entire $SPLUNK_HOME directory and splunk-winevtlog.exe are excluded from real-time file system inspection.

Domain Controller Affinity / Network Routing:

If the problem UF's are running on Domain Controllers or handling high Active Directory traffic, ensure no local firewall or network rate-limiting is capping the Splunk TCP port (default 9997) to the receiving layer (HFs or indexers).

Related Links

• Splunk Documentation:
  
  • Monitoring Windows Eventlog data with Splunk Enterprise
  • Advanced Configurations for the Universal Forwarder
  • Spec files: server.conf, inputs.conf, outputs.conf
• Splunk KB Articles
  • Troubleshooting delayed logs from UF or HF
  • UF not sending security logs to Splunk
• Splunk Lantern Article: Performance tuning the forwarding tier