Following the upgrade to 9.3.1, random crashes occur while executing searches in custom dashboards. The crash logs from the Search Heads indicate fatal signal 11 (Segmentation fault) errors, which indicates a potential memory access violation during the internal processing of search requests. The crashes occurred across multiple search heads, and logs showed that the TcpChannelThread was involved.
Please refer to the below crashing thread:
[build 0b8d769cb912] 2024-09-27 08:51:42
Received fatal signal 11 (Segmentation fault) on PID 3928429.
Cause:
No memory mapped at address [0x0000000000001270].
Crashing thread: TcpChannelThread
Registers:
RIP: [0x00005EFD8B4984F2] _ZN22HandleJobsDataProvider14doLaunchSearchEP24ISplunkDispatchReferenceRK13HttpArgumentsR6StrSetRNS_32jobHandlePreparatoryParameters_tER8Pathname + 4466 (splunkd + 0x3B244F2)
RDI: [0x0000761C037F4958]
RSI: [0x00007621500A5250]
RBP: [0x0000761C037F3940]
RSP: [0x0000761C037F2510]
RAX: [0x0000761C037F3A58]
RBX: [0x0000000000000000]
RCX: [0x000076212DB5D100]
RDX: [0x0000762133C487D0]
R8: [0x0000000000000048]
R9: [0x000076214F000080]
R10: [0x00007621256CD800]
R11: [0x000000000000010A]
R12: [0x000000000000000A]
R13: [0x0000000000000001]
R14: [0x0000000000000000]
R15: [0x0000761BBA469000]
EFL: [0x0000000000010202]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000004]
CSGSFS: [0x002B000000000033]
OLDMASK: [0x0000000000000000] OS: Linux
Arch: x86-64 Backtrace (PIC build):
[0x00005EFD8B4984F2] _ZN22HandleJobsDataProvider14doLaunchSearchEP24ISplunkDispatchReferenceRK13HttpArgumentsR6StrSetRNS_32jobHandlePreparatoryParameters_tER8Pathname + 4466 (splunkd + 0x3B244F2)
[0x00005EFD8B4A1C44] _ZN28HandleOnPremJobsDataProvider12launchSearchEP24ISplunkDispatchReferenceRK13HttpArgumentsR6StrSetRN22HandleJobsDataProvider32jobHandlePreparatoryParameters_tER8Pathname + 36 (splunkd + 0x3B2DC44)
[0x00005EFD8B4945EE] _ZN22HandleJobsDataProvider10handleJobsERK21UserTimezoneSpecifier14RestAPIVersionRK10HttpMethodRK3StrRK10StrSegmentbRK13HttpArgumentsRKNS_27AdhocSearchRequestParameterE + 1710 (splunkd + 0x3B205EE)
[0x00005EFD8B4A1BB2] _ZN28HandleOnPremJobsDataProvider18handleWithTimezoneERK21UserTimezoneSpecifier + 370 (splunkd + 0x3B2DBB2)
[0x00005EFD8B4CE51C] _ZN38DispatchSearchDataProviderWithTimezone21handleWithoutTimezoneEv + 284 (splunkd + 0x3B5A51C)
[0x00005EFD8B45626A] _ZN26DispatchSearchDataProvider2goEv + 42 (splunkd + 0x3AE226A)
[0x00005EFD8A33A25B] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 91 (splunkd + 0x29C625B)
[0x00005EFD8A328B13] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 35 (splunkd + 0x29B4B13)
[0x00005EFD8CA818D0] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 240 (splunkd + 0x510D8D0)
[0x00005EFD89A4BB51] _ZN16TcpChannelThread4mainEv + 385 (splunkd + 0x20D7B51)
[0x00005EFD8CB5C3AE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x51E83AE)
[0x00005EFD8CB5C4BB] _ZN6Thread8callMainEPv + 139 (splunkd + 0x51E84BB)
[0x000076214F294AC3] ? (libc.so.6 + 0x6CAC3)
[0x000076214F326850] ? (libc.so.6 + 0xFE850)
Linux / splunk-prod-sh-02 / 6.5.0-1025-azure / #26~22.04.1-Ubuntu SMP Thu Jul 11 22:33:04 UTC 2024 / x86_64
/etc/debian_version: bookworm/sid
MAP: 5efd87974000-5efd8df9b000 r-xp 00000000 08:01 7077892 /opt/splunk/bin/splunkd
MAP: 5efd8df9b000-5efd8e0d6000 r--p 06626000 08:01 7077892 /opt/splunk/bin/splunkd
MAP: 5efd8e0d6000-5efd8e0fb000 rw-p 06761000 08:01 7077892 /opt/splunk/bin/splunkd
.
.
.
.
MAP: 7ffd5c30e000-7ffd5c310000 r-xp 00000000 00:00 0 [vdso]
MAP: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
Last errno: 2
Threads running: 219
Runtime: 1881.634242s
argv: [splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd]
Regex JIT enabledRE2 regex engine enabledWARN DispatchManager [3704868 TcpChannelThread] - enforceQuotas: username="<user>", search_id="<sid>" - QUEUED reason="The maximum number of concurrent historical searches for this user based on their role quota has been reached.", concurrency_limit="6"
OR:
WARN DispatchManager [2471390 TcpChannelThread] - enforceQuotas: username="<user>", search_id="<sid>" - QUEUED reason="The maximum disk usage quota for this user has been reached.", concurrency_limit="2000"
4. Find this last message before the crash, which might help with identifying the dashboard with the issue.
Search in audit.log for that sid
cat $SPLUNK_HOME/car/log/splunk/audit.log* | grep "action=search, info=granted" | grep <sid>
Look in the audit.log event, the app="<app name>", provenance="UI:Dashboard:<dashboardname>"
5. Validate if the below-mentioned errors are present in search.log with respect to DiskQuota issues. :
WARN DispatchManager [331495 TcpChannelThread] - Queued job id="_cGFsbGF2aS5wYXRpbEBkeGMuY29t_cGFsbGF2aS5wYXRpbEBkeGMuY29t_YmFua2lkX29wcw__RMD53fd5d4de96591a8e_1727430017.6064_6BC717FC-37FB-4C43-9742-59BDB523E8E0", reason="The maximum number of concurrent historical searches on this instance has been reached.", provenance ="UI:Dashboard:bankid_operational_status"
WARN DispatchManager [500182 TcpChannelThread] - Queued job id="_cGFsbGF2aS5wYXRpbEBkeGMuY29t_cGFsbGF2aS5wYXRpbEBkeGMuY29t_YmFua2lkX29wcw__RMD5d13b70f11f2f942c_1727430017.6066_6BC717FC-37FB-4C43-9742-59BDB523E8E0", reason="The maximum number of concurrent historical searches on this instance has been reached.", provenance ="UI:Dashboard:bankid_operational_status"
INFO loader [504259 MainThread] - Splunkd starting (build 0b8d769cb912).WARN DispatchManager [282698 TcpChannelThread] - enforceQuotas: username="name.lastname@email.com", search_id="_cGFsbGF2aS5wYXRpbEBkeGMuY29t_cGFsbGF2aS5wYXRpbEBkeGMuY29t_YmFua2lkX29wcw__RMD5a063b4b536010477_1727428183.835_6BC717FC-37FB-4C43-9742-59BDB523E8E0" - QUEUED reason="The maximum disk usage quota for this user has been reached.", concurrency_limit="1000"
WARN DispatchManager [282698 TcpChannelThread] - Queued job id="_cGFsbGF2aS5wYXRpbEBkeGMuY29t_cGFsbGF2aS5wYXRpbEBkeGMuY29t_YmFua2lkX29wcw__RMD5a063b4b536010477_1727428183.835_6BC717FC-37FB-4C43-9742-59BDB523E8E0", reason="The maximum disk usage quota for this user has been reached.", concurrency_limit="1000", concurrency_limit="1000", provenance ="UI:Dashboard:bankid_operational_status"
INFO loader [283764 MainThread] - Splunkd starting (build 0b8d769cb912).
6. Check if the crashes can be reproduced by opening multiple dashboards and refreshing them multiple time.
1. Open multiple custom dashboards on search heads
2. Refresh the dashboards multiple times.
After which. Splunk crashes intermittently, causing Fatal Signal 11.
7. Validate if those apps are running searches under dashboards which can be the cause of the issue.
find $SPLUNK_HOME/etc/<appname> -name <dashboardname>.xml -exec fgrep "search id" {} \; -print
If results found, then search if that <search id: tag> is a savedsearch using:
./splunk cmd btool savedsearches list --debug | grep <savedsearchname>Work Around:
To avoid the crashes, the <search id> tag in the dashboard XML file can NOT be a savedsearch name. Change <search id> to a different name.
For example in the $SPLUNK_HOME/etc/apps/<appname>/<local|default>/data/ui/views/<dashboardname>.xml you find
<search id="savedsearch name">
Change to:
<search id="savedsearch name123">
Use the following search to find other occurrences of <search id:tag> that is a savedsearch name:
| rest splunk_server=local servicesNS/-/-/data/ui/views/ | spath input=eai:data output={@id} path=form.search{@id} | search {@id} != null | mvexpand {@id} | rename {@id} AS title | table title, id, label | join title [| rest services/saved/searches | rename eai:acl.app AS savedsearch_from_app |table title, savedsearch_from_app] | table *
This issue is fixed in 9.3.4 version.
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |