Indexer Crash

Applicability

Splunk Enterprise

Description

The indexer crashes multiple times and the peer status goes down.
 

Investigation

1. Navigated to $SPLUNK_HOME/opt/splunk/log and observed a lot of crash logs.

2. The crashing threads were BundleLookupIndexingExecutorWorker-0 and StreamSearch

3.  These were the errors observed below :-
 

Received fatal signal 11 (Segmentation fault) on PID 36414.
 Cause:
   No memory mapped at address [0x00007F5B61200000].
 Crashing thread: StreamSearch
 Registers:
    RIP:  [0x0000557D8E68D605] _ZN17KeyValueExtractor6_parseERK10StrSegmentPNS_13scratch_spaceEP3Str + 901 (" + 0x3789605)
    RDI:  [0x00007F5B611FE136]
    RSI:  [0x0000000000000008]
    RBP:  [0x0000000000000001]
    RSP:  [0x00007F5B615F9F60]
    RAX:  [0x0000000000000001]
    RBX:  [0x0000000000000048]
    RCX:  [0x0000000000000048]
    RDX:  [0x0000000000004848]
    R8:  [0x00007F5B61200000]
    R9:  [0x00007F5B611FE014]
    R10:  [0x0000000000000001]
    R11:  [0x0000000000000008]
    R12:  [0x00007F5B611FFFFF]
    R13:  [0x00007F5B61200000]
    R14:  [0x00007F5B611FFFFE]
    R15:  [0x0000557D8EDA6AE0]
    EFL:  [0x0000000000010246]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000004]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]
 OS: Linux
 Arch: x86-64

Received fatal signal 6 (Aborted) on PID 58924.
 Cause:
   Signal sent by PID 58924 running under UID 1002.
 Crashing thread: BundleLookupIndexingExecutorWorker-0
 Registers:
    RIP:  [0x00007F5B6BA4B387] gsignal + 55 (libc.so.6 + 0x36387)
    RDI:  [0x000000000000E62C]
    RSI:  [0x000000000000993D]
    RBP:  [0x00007F5B6BB9FF60]
    RSP:  [0x00007EBD836CFCB8]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F5B6D081000]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0x0000000000000008]
    R9:  [0x00007F5B6B800080]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000202]
    R12:  [0x0000557D8E903A79]
    R13:  [0x0000557D8EB3FD50]
    R14:  [0x00007EBD836D0AE8]
    R15:  [0x00007EBD836D00E0]
    EFL:  [0x0000000000000202]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]
 OS: Linux
 Arch: x86-64

Resolution

The workaround is to change the enable_splunkd_kv_lookup_indexing attribute to false.

The enable_splunkd_kv_lookup_indexing flag offloads the problematic indexing of lookups either from the main splunkd or search process.
The crash thread is related to the indexing of a specified bundle lookup which is failing to take place, and it results in crashes occurring.

enable_splunkd_kv_lookup_indexing = <boolean>
* This setting determines whether KV Store lookup indexing is performed
  during bundle replication.
* When set to true, KVStore lookup indexing occurs on the main splunkd process,
  asynchronous to searches.
* When set to false, KV Store lookup indexing is triggered by the search
  process, potentially slowing search performance.
* Default: true

Workaround - 

-  From an app-specific level:-

1. Navigate to $SPLUNK_HOME/opt/splunk/etc/slave-apps/<app-name>/local/limits.conf

2. Append the following:- 

[lookup]
enable_splunkd_kv_lookup_indexing = false

3. Restart the instance.

- On a global level:-

1. Navigate to $SPLUNK_HOME/opt/splunk/etc/system/local/limits.conf

2. Add the stanza:-

[lookup]
enable_splunkd_kv_lookup_indexing = false

3. Restart the instance.

Related Links

https://docs.splunk.com/Documentation/Splunk/9.0.3/Admin/Limitsconf

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release