Search Head crashing with crashing thread: DispatchReaper

Applicability

Splunk Enterprise

Description

Members of the search head cluster are crashing on Splunk Enterprise version 8.1.x.
Segmentation fault, crashing thread: DispatchReaper

Investigation

Crash.log shows the below:

[build 63079c59e632] 2021-07-16 15:29:41
Received fatal signal 11 (Segmentation fault).
 Cause:
   No memory mapped at address [0x00007F7857000000].
 Crashing thread: DispatchReaper
 Registers:
    RIP:  [0x00007F787DC38D42] ? (libc.so.6 + 0x146D42)
    RDI:  [0x00007F7856FFFFD0]
    RSI:  [0x00007F7856FE7214]
    RBP:  [0x0000000000000029]
    RSP:  [0x00007F78575FAF68]
    RAX:  [0x00007F78575FAFB0]
    RBX:  [0x00007F78575FAF80]
    RCX:  [0x0000000000000010]
    RDX:  [0x0000000000000000]
    R8:  [0x00007F7856FE7214]
    R9:  [0x00007F787DC46280]
    R10:  [0x0000000000000061]
    R11:  [0x00007F787DC7F250]
    R12:  [0x00007F787F2AB8C8]
    R13:  [0x00007F7856E5F190]
    R14:  [0x00007F78575FAF80]
    R15:  [0x00007F78575FB170]
    EFL:  [0x0000000000010282]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000004]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000] OS: Linux
 Arch: x86-64 Backtrace (PIC build):
  [0x00007F787DC38D42] ? (libc.so.6 + 0x146D42)
  [0x000055BA7D14F5EF] _ZN10StrSegment6assignEPKc + 15 (splunkd + 0x22B65EF)
  [0x000055BA7D7A062D] ? (splunkd + 0x290762D)
  [0x000055BA7D7A8CB6] _ZN19SearchJobStatusData12loadFromDiskERK3StrbN17SearchResultsInfo13PopulateFlagsE + 982 (splunkd + 0x290FCB6)
  [0x000055BA7D4C6D69] _ZN8SHPSlave28addDispatchManagerSID_lockedERK3Str + 697 (splunkd + 0x262DD69)
  [0x000055BA7D4C7773] _ZN8SHPSlave17updateRunningSidsERK9StrVectorR3Str + 163 (splunkd + 0x262E773)
  [0x000055BA7D78BF21] _ZN23DispatchDirectoryReaper10doReapWorkEv + 8017 (splunkd + 0x28F2F21)
  [0x000055BA7DC28057] _ZN22PeriodicReapingTimeout4tickEv + 55 (splunkd + 0x2D8F057)
  [0x000055BA7D17235A] _ZN15PeriodicTimeout12when_expiredEP8Interval + 74 (splunkd + 0x22D935A)
  [0x000055BA7D17379C] _ZN11TimeoutHeap18runExpiredTimeoutsER13MonotonicTime + 348 (splunkd + 0x22DA79C)
  [0x000055BA7D0B25B1] _ZN9EventLoop3runEv + 209 (splunkd + 0x22195B1)
  [0x000055BA7C413829] _ZN14DispatchReaper4mainEv + 4857 (splunkd + 0x157A829)
  [0x000055BA7D16BBE7] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22D2BE7)
  [0x00007F787DEC7EA5] ? (libpthread.so.0 + 0x7EA5)
  [0x00007F787DBF08CD] clone + 109 (libc.so.6 + 0xFE8CD)
 Linux / NRR-SPLK-SH01.alrajhi.bank / 3.10.0-1062.el7.x86_64 / #1 SMP Thu Jul 18 20:25:13 UTC 2019 / x86_64
 /etc/redhat-release: Red Hat Enterprise Linux Server release 7.7 (Maipo)
 glibc version: 2.17
 glibc release: stable

Also splunkd.log showing the below:

07-17-2021 00:55:15.648 +0300 WARN  DispatchReaper - Stale artifacts: 
rsa_scheduler__admin__search__RMD56b3eb6a292646e23_at_1587145200_8554_102BC3DD-9680-4902-B37C-16AE003756E0 
rsa_scheduler__admin__search__RMD56f25944b6ab27f91_at_1586799000_245_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__admin_U0EtTmV0d29ya1Byb3RlY3Rpb24__RMD5b7e70e10f3a64971_at_1587138300_33445_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__admin_U0EtVGhyZWF0SW50ZWxsaWdlbmNl__RMD5411db2741096efb0_at_1587138000_33391_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__admin__search__RMD564017d15766964bf_at_1587092400_24935_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__admin_U0EtVGhyZWF0SW50ZWxsaWdlbmNl__RMD51dbc70e0e18846c9_at_1587144600_11510_F3111C82-CFED-43FD-A1B1-5D77A52ABB2C 
rsa_scheduler__admin_U0EtVGhyZWF0SW50ZWxsaWdlbmNl__RMD501805f07ae0aff07_at_1587145260_11533_F3111C82-CFED-43FD-A1B1-5D77A52ABB2C 
rsa_scheduler__admin_U0EtTmV0d29ya1Byb3RlY3Rpb24__RMD5eba227fdebf843bd_at_1587142200_9180_9437D7B3-36DF-4591-814E-C0E5BA6F0364 
rsa_scheduler__admin_VEEtQ3VzdG9tTG9va3Vwcw__RMD5267cf48f0376739e_at_1587070800_6953_F3111C82-CFED-43FD-A1B1-5D77A52ABB2C
rsa_scheduler__admin__search__RMD51fa418051fe1da4b_at_1587096000_25587_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__admin__search__RMD5468491d498ed9ebd_at_1587096000_25618_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__con0570__SplunkEnterpriseSecuritySuite__RMD5d7507bc20ed247df_at_1587078000_7335_F3111C82-CFED-43FD-A1B1-5D77A52ABB2C 
rsa_scheduler__admin__search__RMD5d1de0a980be7b34a_at_1587092400_6467_102BC3DD-9680-4902-B37C-16AE003756E0 
rsa_scheduler__admin_U0EtTmV0d29ya1Byb3RlY3Rpb24__RMD5801804c5775ceeab_at_1587136800_33193_3C7FD1EE-5684-49BC-A534-DA54F648D1EE 
rsa_scheduler__admin__search__RMD5add620941ab2b9b0_at_1587144000_8487_102BC3DD-9680-4902-B37C-16AE003756E0 
rsa_scheduler__admin_U0EtVGhyZWF0SW50ZWxsaWdlbmNl__RMD5bce4b82e8925789e_at_1587145140_9342_9437D7B3-36DF-4591-814E-C0E5BA6F0364 
rsa_scheduler__admin__search__RMD56f8dac1c23d252be_at_1587142800_11397_F3111C82-CFED-43FD-A1B1-5D77A52ABB2C 
rsa_scheduler__admin__SplunkEnterpriseSecuritySuite__RMD5a6d54b4f1c421eaa_at_1587144900_8544_102BC3DD-9680-4902-B37C-16AE003756E0 
rsa_scheduler__admin__search__RMD56f8dac1c23d252be_at_1587139200_8249_102BC3DD-9680-4902-B37C-16AE003756E0 
rsa_scheduler__admin__SplunkEnterpriseSecuritySuite__RMD5a6d54b4f1c421eaa_at_1587145200_11531_F3111C82-CFED-43FD-A1B1-5D77A52ABB2C07-17-2021 00:55:15.647 +0300 WARN  DispatchReaper - Found 74 expired and unreaped artifacts (SHC managed). Displaying first 20.

 

Resolution

This is a known issue in 8.1.x and the below workaround should resolve the issue.

Push the below configuration from the deployer (if you have one), you need to set enable_jobs_data_lite in server.conf under the [shclustering] stanza to false.
 

[shclustering]
enable_jobs_data_lite = false

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release