Splunk universal forwarder crash due to ChunkedLBProcessor

Applicability

Description

The universal forwarder was crashing very frequently, with thread ChunkedLBProcessor falling, causing the crash.
 

crash log: Received fatal signal 6 (Aborted) on PID 14351.
Cause: Signal sent by PID 14351 running under UID 100014.
Crashing thread: parsing
Registers: RIP: [0x00007EFF10EDB387] gsignal + 55 (libc.so.6 + 0x36387)
RDI: [0x000000000000380F]
RSI: [0x000000000000384A]
RBP: [0x0000000000000077]
RSP: [0x00007EFF021FD798]
RAX: [0x0000000000000000]
RBX: [0x00005648D42C65F0]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000000]
R9: [0x00007EFF10EF22CD]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x00005648D4206711]
R13: [0x00005648D42C65B8]
R14: [0x00007EFF078B38C8]
R15: [0x00007EFEFF096E00]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace (PIC build): [0x00007EFF10EDB387] gsignal + 55 (libc.so.6 + 0x36387)
[0x00007EFF10EDCA78] abort + 328 (libc.so.6 + 0x37A78)
[0x00005648D141A1EA] ? (splunkd + 0x1A0A1EA)
[0x00005648D1970F12] _ZN18ChunkedLBProcessor10setupStateER15CowPipelineData + 802 (splunkd + 0x1F60F12)
[0x00005648D19714A1] _ZN18ChunkedLBProcessor7executeER15CowPipelineDataP18PipelineDataVector + 193 (splunkd + 0x1F614A1)
[0x00005648D197190E] _ZN18ChunkedLBProcessor12executeMultiER18PipelineDataVectorPS0_ + 78 (splunkd + 0x1F6190E)
[0x00005648D177F4A2] _ZN8Pipeline4mainEv + 1074 (splunkd + 0x1D6F4A2)
[0x00005648D27E286D] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 13 (splunkd + 0x2DD286D)
[0x00005648D27E3762] _ZN6Thread8callMainEPv + 178 (splunkd + 0x2DD3762)
[0x00007EFF118D8EA5] ? (libpthread.so.0 + 0x7EA5)
[0x00007EFF10FA3B0D] clone + 109 (libc.so.6 + 0xFEB0D)Linux / 3.10.0-1160.76.1.el7.x86_64 / #1 SMP Tue Jul 26 14:15:37 UTC 2022 / x86_64Last few lines of stderr (may contain info on assertion failure, but also could be old):
assertion_failure="chan->hasSourceType()" assertion_function="void ChunkedLBProcessor::setupState(CowPipelineData&)" assertion_file="/opt/splunk/src/LBProcessor.cpp:119"2023-01-18 17:22:17.561 +0000 splunkd started (build dd0128b1f8cd) pid=25279assertion_failure="chan->hasSourceType()" assertion_function="void ChunkedLBProcessor::setupState(CowPipelineData&)" assertion_file="/opt/splunk/src/LBProcessor.cpp:119"2023-01-18 21:20:42.492 +0000 splunkd started (build dd0128b1f8cd) pid=14351assertion_failure="chan->hasSourceType()" assertion_function="void ChunkedLBProcessor::setupState(CowPipelineData&)" assertion_file="/opt/splunk/src/LBProcessor.cpp:119"

Investigation

• By running the following search

index=_internal component=ChunkedLBProcessor

WARN  ChunkedLBProcessor [14410 parsing] - The EVENT_BREAKER regex in props.conf does not have a valid capturing group for sourcetype::splunkd. 
Please update your regex with valid capturing groups. Reverting to the default EVENT_BREAKER regex for now.

• The EVENT_BREAKER regex in props.conf does not have a valid capturing group for a sourcetype (e.g. sourcetype::splunkd). 

Resolution

• Make sure to use a valid regex for the EVENT_BREAKER value.
• Remove the application that is containing the invalid value for the EVENT_BREAKER.
• Regex are edited at the [sourcetype] stanza at props.conf

Related Links