Kernel logs show the OOM-killer targeting splunkd. Resolving by enforcing OS memory limits (systemd MemoryMax/MemoryHigh) and disabling the optional Prometheus exporter has stabilized the node.
Follow these steps to confirm if OOM errors are causing indexer downtime.
Search system logs to identify OOM killer invocations.
journalctl -u splunk.service --since "YYYY-MM-DD" grep -iE "drop|error|retransmit|oom" /var/log/syslog grep -i "oom" /var/log/messages
Look for entries similar to:
kernel: Out of memory: Killed process 907875 (splunkd) total-vm:19629888kB, anon-rss:15990996kB kernel: oom-kill:constraint=CONSTRAINT_NONE
Verify memory usage and limits by checking current memory limits applied to the splunk.service.
systemctl show splunk.service -p MemoryMax -p MemoryHigh -p MemorySwapMax -p Slice
For cgroup v2:
cat /sys/fs/cgroup/system.slice/splunk.service/memory.max
For cgroup v1:
cat /sys/fs/cgroup/memory/system.slice/splunk.service/memory.limit_in_bytes
If MemoryLimit is used instead of MemoryMax, it indicates an outdated configuration for Splunk 9.4.x.
Check prometheus configuration to verify if the Prometheus exporter is enabled, contributing to memory growth.
$SPLUNK_HOME/bin/splunk btool server list --debug | grep prometheus
Output showing enabled Prometheus is empty.
Monitor Cluster Status to check the indexer's status within the cluster.
$SPLUNK_HOME/bin/splunk show cluster-status --verbose
Review resource utilization and monitor memory usage and compare with other indexers to identify discrepancies.
top htop free -m ps aux --sort -rss | grep splunk
Compare configurations (server.conf, limits.conf, indexes.conf) across cluster nodes to ensure consistency.
Apply these steps to prevent OOM-related indexer crashes. Restart Splunk after each change using:
$SPLUNK_HOME/bin/splunk restart
Configure Memory Limits in splunk.service
Set appropriate memory limits in the systemd service file to prevent excessive memory usage.
Use MemoryMax for Splunk 9.4.x (replace MemoryLimit if present).
For persistent changes:
sudo systemctl set-property splunk.service MemoryMax=<INTEGER>G MemoryHigh=<INTEGER>G
For temporary changes (reverts on reboot):
sudo systemctl set-property --runtime splunk.service MemoryMax=<INTEGER>G MemoryHigh=<INTEGER>G
Adjust values based on system memory (depending on your system, 80-90% of total RAM).
Reload systemd after changes:
sudo systemctl daemon-reload
Disable the Prometheus exporter to mitigate the known memory leak issue.
Update the [prometheus] stanza in:
$SPLUNK_HOME/etc/system/local/server.conf
[prometheus] disabled = true
Validate the change:
$SPLUNK_HOME/bin/splunk btool server list --debug prometheus
Verify system settings to ensure transparent huge pages are disabled to optimize Splunk performance:
cat /sys/kernel/mm/transparent_hugepage/enabled cat /sys/kernel/mm/transparent_hugepage/defrag
Both should returnĀ never. If not, update per Transparent Huge Memory Pages and Splunk Performance docs.
Verify ulimit settings meet Splunk recommendations:
ulimit -n ulimit -u
Ensure nofile=64000 and nproc=65536 or higher, as per System Requirements for Splunk Enterprise.
init.d does not support MemoryMax settings.Workload Management with cgroups v1
Run Splunk Enterprise as a systemd Service
Transparent Huge Memory Pages and Splunk Performance
System Requirements for Splunk Enterprise
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
01-Oct-2026
|
Initial Release |