splunk-winevtlog.exe crashing on Windows Forwarders with Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Applicability

Description

Post migration of Windows to 2022, "splunk-winevtlog.exe" crashes occur on the Windows machine logs with the below crash thread:

Faulting application name: splunk-winevtlog.exe, version: 2305.256.25832.56887, time stamp: 0x64e8dfcc
Faulting module name: ntdll.dll, version: 10.0.x.x, time stamp: 0x31881ea2
Exception code: 0xc0000374
Fault offset: 0x0000000000104909
Faulting process id: 0x1304
Faulting application start time: 0x01d9ed2bd5be870c
Faulting application path: C:\Program Files\Splunk\bin\splunk-winevtlog.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 45c2b6fd-2c6e-484d-9602-eb948052101d
Faulting package full name:
Faulting package-relative application ID: 

Investigation

1.  Validate if the Windows forwarder is crashing on Splunk 9.1.x, 9.2.x, 9.3.x, 9.4.0 or 9.4.1 on Windows Server 2022. If the crashes continue, then upgrade Splunk to fixed version 9.4.2 and validate.
   
   
2.  If the crash still continues, try disabling AD object resolution (evt_resolve_ad_obj = 0).
   

   ./inputs.conf
   [WinEventLog://ForwardedEvents]
   evt_resolve_ad_obj = 0

3. Run SFC scan to repair the corrupted files if any.
   

   sfc /scannow

Resolution

This is identified as a known issue and is fixed in Splunk version 9.4.2 onwards. Please upgrade to the fixed version.

If upgrading is not possible the only available workaround is to disable WinEventLog://ForwardedEvents. To disable, either remove the event viewer subscriptions from Windows end, or disable the inputs for WinEventLog from Splunk. 

To disable the input add the following to $SPLUNK_HOME/etc/system/local/inputs.conf and restart Splunk:

[WinEventLog://ForwardedEvents]
disabled = 1