The problematic indexer has many crash files in the %SPLUNK_HOME%\var\log\splunk directory.
Splunk crash logs are typically named in the format crash-<timestamp>.log, where <timestamp> represents the date and time when the crash occurred. These logs are generated when there is an unexpected crash in Splunk's processes, and they provide diagnostic information to help identify the cause of the crash.
Crashing threads are below:
BucketSummaryActorThread BatchSearch StreamSearch TSCollectThread
In further investigation, it was found in etc/system/local/splunkd.log the following errors.
WARN ulimit - This configuration of vm.overcommit is known to cause serious runtime problems with Splunk. Typical symptoms include sporadic crashes due to memory allocation failures. Please fix by setting the value for vm.overcommit to "1" (always overcommit) or preferably "0" (heuristic overcommit) via sysctl, kernel boot parameters, or other method recommended by your Linux distribution.
In this case, changes need to be made at the OS level to address this issue.
1, add the below parameter in /etc/sysctl.conf file.
vm.overcommit = 0
2.
echo 0 > /proc/sys/vm/overcommit_memory.
Restart the Splunk service. Commands are in this documentation.
How this setting impacts Splunk performance:
Setting this parameter to anything other than the default of 0 can cause Splunk Enterprise to crash frequently with logs that do not reliably provide information on the root cause of the crash. In particular, setting the overcommit_memory Parameter 2 can result in crashes that provide no information, making troubleshooting and problem diagnosis difficult. Confirm that you have set the vm.overcommit_memory parameter to its default of 0 unless you run an app on the same machine that requires changing this parameter.
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
28-Sep-2026
|
Initial Release |