New Search head is getting crashed while adding it to the Search head cluster(SHC).

Applicability

Splunk Enterprise Splunk Apps and Addon

Description

Here the new search head which has just been added to the search head cluster starts getting crashed with the following crashing thread;
Crashing thread: WebuiStartup

The user experiences the  same thing every time when they restart the Search head instance, and they can see the following files being generated:

The creation of a core.xxx file in /tmpA crash file was created in /opt/splunk/var/log/splunk, named crash-2022-12-29-10:52:51.log.
splunkd stderr.log, found in /opt/splunk/var/log/splunk, has been updated with new lines.

The same problem is occurring in three different "new search heads".

The user could see error messages for splunk_app_stream in the splunkd.log at $SPLUNK_HOME/var/log/splunk:

2023-01-24 14:09:16,983 ERROR [63cfe5518e7f341fc524d0] __init__:522 -
Socket error communicating with splunkd (error=[Errno 111] Connection
refused), path = /services/splunk_app_stream/vocabularies/

2023-01-24 14:09:17,173 ERROR [63cfe5518e7f341fc524d0] custom:189 - /opt/splunk/etc/apps/splunk_app_stream/appserver/controllers/streamforwardergroups.py cannot be executed by Python: the JSON object must be str, bytes or bytearray, not dict

 

Investigation

Follow the following steps to investigate the issue:
1) When an individual adds a new Search head member to a search head cluster, during the initialization the search head contacts the deployer to get all the cluster configuration (apps/addons) so there is no way to remove an app/addon on just on one search head.

2)The option to fully remove it from the deployer is not feasible for the user because they can not afford that situation in their environment.
Examine the diag file and discovered one crash log on December 29 at 10:58. From crash.log, at $SPLUNK_HOME/var/log/splunk ;

Crash logs:[build da25d08d5d3e] 2023-01-24 14:09:01
Received fatal signal 6 (Aborted) on PID 112864.
 Cause:
   Signal sent by PID 112864 running under UID 94241.
 Crashing thread: WebuiStartup
 Registers:
    RIP:  [0x00007F644D1C737F] gsignal + 271 (libc.so.6 + 0x3737F)
    RDI:  [0x0000000000000002]
    RSI:  [0x00007F6424DFE2E0]
    RBP:  [0x00007F644D31A5F8]
    RSP:  [0x00007F6424DFE2E0]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x00007F644D1C737F]
    RDX:  [0x0000000000000000]
    R8:  [0x0000000000000000]
    R9:  [0x00007F6424DFE2E0]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000246]
    R12:  [0x0000561F688238D0]
    R13:  [0x0000561F686ECAAB]
    R14:  [0x0000000000000760]
    R15:  [0x00007F6424843698]
    EFL:  [0x0000000000000246]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x002B000000000033]
    OLDMASK:  [0x0000000000000000] OS: Linux
 Arch: x86-64 Backtrace (PIC build):
  [0x00007F644D1C737F] gsignal + 271 (libc.so.6 + 0x3737F)
  [0x00007F644D1B1DB5] abort + 295 (libc.so.6 + 0x21DB5)
  [0x00007F644D1B1C89] ? (libc.so.6 + 0x21C89)
  [0x00007F644D1BFA76] ? (libc.so.6 + 0x2FA76)
  [0x0000561F6729AA69] ? (splunkd + 0x2609A69)
  [0x0000561F6624CA70] ? (splunkd + 0x15BBA70)
  [0x0000561F67421CEC] _ZN18AppServerProcessesC1ERK13PropertiesMapRK8PathnameRKSt6vectorI10PortNumberSaIS7_EES7_immbPKb + 380 (splunkd + 0x2790CEC)
  [0x0000561F6741A45C] _ZN12UiHttpServer4initERK17PropertiesMapTypeRKSt6vectorI10PortNumberSaIS4_EES4_b + 7324 (splunkd + 0x278945C)
  [0x0000561F6741B525] _ZN17RealWebuiCallback16_initUihs_LockedERK17PropertiesMapTypeRKSt6vectorI10PortNumberSaIS4_EES4_b + 101 (splunkd + 0x278A525)
  [0x0000561F6741BB71] _ZN17RealWebuiCallback6_startEb + 1457 (splunkd + 0x278AB71)
    [0x0000561F664B6C9D] _ZN18WebuiStartupThread4mainEv + 77 (splunkd + 0x1825C9D)
  [0x0000561F67355EB7] _ZN6Thread8callMainEPv + 135 (splunkd + 0x26C4EB7)
  [0x00007F644D55D14A] ? (libpthread.so.0 + 0x814A)
  [0x00007F644D28CDC3] clone + 67 (libc.so.6 + 0xFCDC3)
 Linux / rbaussplkprd20.bas.roche.com / 4.18.0-305.45.1.el8_4.x86_64 / #1 SMP Wed Apr 6 13:48:37 EDT 2022 / x86_64
 Libc abort message: splunkd: /opt/splunk/src/util/EventLoop.cpp:1888: virtual PolledReadPipe::~PolledReadPipe(): Assertion `!isActive()' failed. /etc/redhat-release: Red Hat Enterprise Linux release 8.4 (Ootpa)
 glibc version: 2.28
 glibc release: stable
Last errno: 11
Threads running: 91
Runtime: 315.514595s
argv: [splunkd -p 8089 restart]
Regex JIT enabledRE2 regex engine enabledusing CLOCK_MONOTONIC
Thread: "WebuiStartup", did_join=0, ready_to_run=Y, main_thread=N, token=140068092114688
MutexByte: MutexByte-waiting={none}x86 CPUID registers:
         0: 00000016 756E6547 6C65746E 49656E69
         1: 00050657 44400800 7FFEFBFF BFEBFBFF
         2: 76036301 00F0B6FF 00000000 00C30000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000040 00000040 00000003 00002020
         6: 00000AF7 00000002 00000009 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300804 00000000 00000000 00000603
         B: 00000000 00000000 00000047 00000044
         C: 00000000 00000000 00000000 00000000
         D: 00000000 00000000 00000000 00000000
         E: 00000000 00000000 00000000 00000000
         F: 00000000 00000000 00000000 00000000
        10: 00000000 00000000 00000000 00000000
        11: 00000000 00000000 00000000 00000000
        12: 00000000 00000000 00000000 00000000
        13: 00000000 00000000 00000000 00000000
        14: 00000000 00000000 00000000 00000000
        15: 00000002 000000D0 00000000 00000000
        16: 00000A28 00000F3C 00000064 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000121 2C100800
  80000002: 65746E49 2952286C 6F655820 2952286E
  80000003: 6C6F4720 32362064 43203034 40205550
  80000004: 362E3220 7A484730 00000000 00000000
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 0000302E 00000000 00000000 00000000
terminating...

3) This is a known issue for splunk_app_stream version 8.1.0 which can be found in the following location $SPLUNK_HOME/etc/apps/splunk_app_stream/default/app.conf.

4) Root cause:
The issue that causes the crash for the WebuiStartup thread is that the "EventLoop pipe" is not cleaned up properly and the app, namely stream here, is still attached to the pipe. When the Search head is restarted the next time and try to use the read pipe the crashing thread still remains active, hence we are getting blocked "EventLoop pipe", you can find the following message in web_service.log at the given location $SPLUNK_HOME/var/log/splunk/web_service. log

1Libc abort message: splunkd: /opt/splunk/src/util/EventLoop.cpp:1888: virtual PolledReadPipe::~PolledReadPipe(): Assertion `!isActive()' failed.

The issue with the cleanup is with the splunk_app_stream app. For some reason, it doesn’t cleanly release the event loop pipe. So when the SH WebuiStartup thread is running, the pipe is still active due to the splunk_app_stream app causing the crash and the failure of the UI to startup. Stream may is still attached to the pipe due to this error:

12023-01-24 14:09:17,173 ERROR [63cfe5518e7f341fc524d0] custom:189 - /opt/splunk/etc/apps/splunk_app_stream/appserver/controllers/streamforwardergroups.py cannot be executed by Python: the JSON object must be str, bytes or bytearray, not dict

Resolution

1)The fix will be available in a future version release.
2) The workaround to avoid the crash of search head instance is to disable the splunk_app_stream app in app.conf prior to adding nodes to the Search head cluster.
Visit the following location $SPLUNK_HOME/etc/apps/splunk_app_stream/default/app.conf and disable the app :

[install]
state = disabled

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release