Windows Universal forwarders crashing after upgrade to 9.2.1

Applicability

Description

Splunk universal forwarders crashed due to an access violation after the upgrade to the 9.2.1 version on windows 10 servers.

[build 78803f08aabb] 2024-04-09 13:57:41
 Access violation, cannot write at address [0x0000000000000024]
 Exception address: [0x00007FFCABCB34F6]
 Crashing thread: WinHostmonWorker

[build 1fff88043d5f] 2024-04-09 14:00:47
 Access violation, cannot write at address [0x0000000000000024]
 Exception address: [0x00007FFDF24134F6]
 Crashing thread: WinEventLogInputProcessor
    MxCsr:  [0x0000000000001FA0]
    SegDs:  [0x000000000000002B]
    SegEs:  [0x000000000000002B]
    SegFs:  [0x0000000000000053]
    SegGs:  [0x000000000000002B]
    SegSs:  [0x000000000000002B]
    SegCs:  [0x0000000000000033]
    EFlags:  [0x0000000000010213]
    Rsp:  [0x000000D6D81FFAC0]
    Rip:  [0x00007FFDF24134F6] RtlWaitOnAddress + 198/2080
    Dr0:  [0x0000000000000000]
    Dr1:  [0x0000000000000000]
    Dr2:  [0x0000000000000000]
    Dr3:  [0x0000000000000000]
    Dr6:  [0x0000000000000000]
    Dr7:  [0x0000000000000000]
    Rax:  [0x0000000000000000]
    Rcx:  [0x00007FF66D8567F8]
    Rdx:  [0x00000000FFFFFFFA]
    Rbx:  [0x00007FF66D8567F8]
    Rbp:  [0x000000D6D81FFD00]
    Rsi:  [0x0000000000000000]
    Rdi:  [0x0000000000000000]
    R8:  [0x0000000000000000]
    R9:  [0x00007FF66D856800]
    R10:  [0x0000000000000000]
    R11:  [0x000000D6D81FFC60]
    R12:  [0x0000000000000000]
    R13:  [0x000000D6D5509000]
    R14:  [0x0000000000000001]
    R15:  [0x0000000000000000]
    DebugControl:  [0x000002BF38650FA0]
    LastBranchToRip:  [0x0000000000000000]
    LastBranchFromRip:  [0x0000000000000000]
    LastExceptionToRip:  [0x0000000000000000]
    LastExceptionFromRip:  [0x0000000000000000]
 OS: Windows
 Arch: x86-64
 Backtrace:
  [0x00007FFDF24134F6] RtlWaitOnAddress + 198/2080
Args:  [0x0000046B380F0080]  [0x0000000003ED8295]  [0x00000000FFFFFFFA]
  [0x00007FFDF23DFCB4] RtlEnterCriticalSection + 532/800
Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000046B3C080080]
  [0x00007FFDF23DFAE2] RtlEnterCriticalSection + 66/800
Args:  [0x0000040054564500]  [0x0000046AD4050380]  [0x0000000000000000]
  [0x00007FF66C24F672] ?
Args:  [0x0000046B3C080080]  [0x0000046B3C080090]  [0x0000000000000000]
  [0x00007FF66C28BA72] OPENSSL_Applink + 208946
Args:  [0x0000046B380F0080]  [0x0000000000000000]  [0x0000000000000000]
  [0x00007FF66C28B354] OPENSSL_Applink + 207124
Args:  [0x000000D6D81FFD80]  [0x0000000000000000]  [0x0000000000000000]
  [0x00007FF66C28B9EE] OPENSSL_Applink + 208814
Args:  [0x0000046B38050170]  [0x0000046B38050170]  [0x0000000000000000]
  [0x00007FFDEFA81BB2] configthreadlocale + 146/544
Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000000000000000]
  [0x00007FFDF1277344] BaseThreadInitThunk + 20/2768
Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000000000000000]
  [0x00007FFDF24026B1] RtlUserThreadStart + 33/80
Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000000000000000]

Note: WinHostmonWorker is truncated and cannot observe the complete crash log. 

Investigation

1. Turn off the persistent queue. This will eliminate the crash. Please check all the inputs that have persistent queue enabled and change the configuration as shown in the below example:

/opt/splunk/etc/system/local/inputs.conf

[<input_stanza>]
persistentQueueSize =

2. Disable `splunk_internal_metrics` app as it does source type cloning for metrics.log. Most of us are probably not aware that metrics.log is cloned and additionally indexed into the `_metrics` index. If you are not using the `_metrics` index, disable the app.

3. Apply below configurations to reduce the chances of crashes. make the changes under /opt/splunk/etc/system/local/limits.conf

[<input_channels>]
max_inactive=300001
lowater_inactive=300000
inactive_eligibility_age_seconds=120

4. Also Increase the input queue size. under

/opt/splunk/etc/system/local/server.conf

[queue=splunktcpin]
maxSize = <This value is dependent on each environment and might require tuning, a suggested value to start is 10MB>

5. Slow down forwarders by setting `autoBatch=false` on all universal forwarders/heavy forwarders.

/opt/splunk/etc/system/local/outputs.conf

autoBatch=false

Resolution

This is a known issue in version 9.2.1:

1. The crash is caused by a race condition on the WinHostMonitor exit (race condition between the worker thread and main thread)

2. The fix will be available in 9.1.7, 9.2.4, and 9.3.2 patches.