Universal forwarder splunk-winevtlog.exe crashing after upgrading to 9.x

Applicability

Description

As of 9.x UF (Upgraded from 8x) servers are receiving multiple crashes a day of splunk-winevtlog.exe, resulting in EventID 1000 and 1001 

All Windows hosts have the same splunk apps and Winevent log ingestion settings (Splunk_TA_Windows); however while this is occurring on most hosts (~1-50), the highest occurrence rate is on domain controllers (~100-1000 times per day)

Content of eventid 1000:Faulting application name: splunk-winevtlog.exe, version: 2305.1.25759.49261, time stamp: 0x649fc2ec
Faulting module name: splunk-winevtlog.exe, version: 2305.1.25759.49261, time stamp: 0x649fc2ec
Exception code: 0xc0000005
Fault offset: 0x000000000007a406
Faulting process id: 0x6d04
Faulting application start time: 0x01d9c00b433c8525
Faulting application path: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe
Faulting module path: C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe
Report Id: 3f5a4a31-92b6-440a-8552-7e85462c59f7
Faulting package full name:
Faulting package-relative application ID:

With frequent crashing on splunk-winevtlog.exe. It creates events with codes 1000 and 1001.
The errors started occurring after upgrading UFs to version 9.0.5

Version=1
EventType=APPCRASH
EventTime=133330927109385791
ReportType=2
Consent=1
ReportIdentifier=d18dec64-1bb8-11ee-a9cf-0050568fabe6
IntegratorReportIdentifier=35aeba3d-bcef-460f-8184-bae85061c96c
NsAppName=splunk-winevtlog.exe
AppSessionGuid=00005e10-0000-0096-024f-b093c5afd901
TargetAppId=W:000038f841dc2fb638f806ebd10761167bed00000904!000087c38d8c8fd8f397531b8ef3618e13825abe20bb!splunk-winevtlog.exe
TargetAppVer=2023//05//26:23:20:33!11f60a1!splunk-winevtlog.exe
BootId=4294967295
Response.type=4
Sig[0].Name=Application Name
Sig[0].Value=splunk-winevtlog.exe
Sig[1].Name=Application Version
Sig[1].Value=2304.1280.25713.15594
Sig[2].Name=Application Timestamp
Sig[2].Value=64713ec1
Sig[3].Name=Fault Module Name
Sig[3].Value=splunk-winevtlog.exe
Sig[4].Name=Fault Module Version
Sig[4].Value=2304.1280.25713.15594
Sig[5].Name=Fault Module Timestamp
Sig[5].Value=64713ec1
Sig[6].Name=Exception Code
Sig[6].Value=c0000005
Sig[7].Name=Exception Offset
Sig[7].Value=0000000000074667
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=10.0.14393.2.0.0.400.8
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=3081
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=4e1e
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=4e1ee9bcd03e82a35fd76d88471c10ac
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=3cde
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=3cde873c2d394617a158d6d1cc8d3e8e
UI[2]=C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=Monitor windows event logs stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\System32\KERNEL32.DLL
LoadedModule[3]=C:\Windows\System32\KERNELBASE.dll
LoadedModule[4]=C:\ProgramData\Symantec\Symantec Endpoint Protection\14.3.9205.6000.105\Data\Sysfer\x64\sysfer.dll
LoadedModule[5]=C:\Windows\System32\WS2_32.dll
LoadedModule[6]=C:\Program Files\SplunkUniversalForwarder\bin\libxml2.dll
LoadedModule[7]=C:\Windows\SYSTEM32\ACTIVEDS.dll
LoadedModule[8]=C:\Windows\System32\sechost.dll
LoadedModule[9]=C:\Windows\System32\ucrtbase.dll
LoadedModule[10]=C:\Windows\System32\RPCRT4.dll
LoadedModule[11]=C:\Windows\System32\msvcrt.dll
LoadedModule[12]=C:\Program Files\SplunkUniversalForwarder\bin\libxslt.dll
LoadedModule[13]=C:\Windows\System32\ADVAPI32.dll
LoadedModule[14]=C:\Program Files\SplunkUniversalForwarder\bin\SSLEAY32.dll
LoadedModule[15]=C:\Windows\System32\USER32.dll
LoadedModule[16]=C:\Program Files\SplunkUniversalForwarder\bin\libxmlsec.dll
LoadedModule[17]=C:\Windows\System32\win32u.dll
LoadedModule[18]=C:\Program Files\SplunkUniversalForwarder\bin\libxmlsec-openssl.dll
LoadedModule[19]=C:\Windows\System32\GDI32.dll
LoadedModule[20]=C:\Program Files\SplunkUniversalForwarder\bin\archive.dll
LoadedModule[21]=C:\Program Files\SplunkUniversalForwarder\bin\LIBEAY32.dll
LoadedModule[22]=C:\Windows\System32\gdi32full.dll
LoadedModule[23]=C:\Program Files\SplunkUniversalForwarder\bin\VCRUNTIME140.dll
LoadedModule[24]=C:\Windows\System32\ole32.dll
LoadedModule[25]=C:\Windows\System32\combase.dll
LoadedModule[26]=C:\Windows\System32\bcryptPrimitives.dll
LoadedModule[27]=C:\Windows\SYSTEM32\adsldpc.dll
LoadedModule[28]=C:\Windows\System32\OLEAUT32.dll
LoadedModule[29]=C:\Windows\System32\WLDAP32.dll
LoadedModule[30]=C:\Windows\System32\msvcp_win.dll
LoadedModule[31]=C:\Program Files\SplunkUniversalForwarder\bin\MSVCP140.dll
LoadedModule[32]=C:\Windows\SYSTEM32\bcrypt.dll
LoadedModule[33]=C:\Windows\SYSTEM32\CRYPTSP.dll
LoadedModule[34]=C:\Windows\system32\rsaenh.dll
LoadedModule[35]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[36]=C:\Windows\SYSTEM32\DbgHelp.dll
LoadedModule[37]=C:\Windows\SYSTEM32\dbgcore.DLL
LoadedModule[38]=C:\Windows\SYSTEM32\wevtapi.dll
LoadedModule[39]=C:\Windows\SYSTEM32\Netapi32.dll
LoadedModule[40]=C:\Windows\SYSTEM32\LOGONCLI.DLL
LoadedModule[41]=C:\Windows\SYSTEM32\NETUTILS.DLL
LoadedModule[42]=C:\Windows\SYSTEM32\Ntdsapi.dll
LoadedModule[43]=C:\Windows\System32\kernel.appcore.dll
LoadedModule[44]=C:\Windows\system32\mswsock.dll
LoadedModule[45]=C:\Windows\SYSTEM32\DNSAPI.dll
LoadedModule[46]=C:\Windows\System32\NSI.dll
LoadedModule[47]=C:\Windows\SYSTEM32\IPHLPAPI.DLL
LoadedModule[48]=C:\Windows\System32\rasadhlp.dll
LoadedModule[49]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[50]=C:\Windows\SYSTEM32\DSPARSE.dll
LoadedModule[51]=C:\Windows\SYSTEM32\sspicli.dll
LoadedModule[52]=C:\Windows\system32\kerberos.DLL
LoadedModule[53]=C:\Windows\System32\MSASN1.dll
LoadedModule[54]=C:\Windows\SYSTEM32\cryptdll.dll
FriendlyEventName=Stopped working
ConsentKey=APPCRASH
AppName=Monitor windows event logs
AppPath=C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=40459D28659BCB1C3EA4D8C28EDA1771
MetadataHash=1154802164

Investigation

Did not have these errors before upgrading.

This was tested on 9.4 as well.

Resolution

 Disabling the evt_resolve_ad_obj value for an input [WinEventLog://Security]. After disabling the parameter, the crash stopped.

This is a known issue in 9.0.5 and 9.1.0.1; there is JIRA ticket: SPL-240820