Splunk can be killed by an OS's out of memory manager. This can happen if Splunk or some other process is using all of the system memory. Most of the time, this is the result of search scheduling or search inefficiency. It can also be an indicator that the cluster needs to be scaled up, or more memory added to keep up with the load. It is best to start by ensuring searches are written efficiently and not over-scheduled.
Search a Linux system's logs for out-of-memory conditions using the following grep commands:
grep -i oom /var/log/messagesgrep -i "out of memory" /var/log/messages
Change "messages" to "syslog" in the command above for debian-based systems.
Splunk may have signs of memory constraints as well. Internal logs may contain evidence of memory constraints:
index=_internal sourcetype=scheduler
ERROR SavedSplunker - savedsearch_id="<some_saved_search>", message="Your search has been terminated. This is most likely due to an out of memory condition. Either edit your search to reduce memory requirements or contact your Splunk administrator.". No actions executed
If there is a startup event in splunkd.log, search for "My GUID":
index=_internal host=<hostname> sourcetype=splunkd "My GUID"
verify the source to see if there are any shutdown events in the log event before the event containing "My GUID". Shutdown events will look similar to the following events:
INFO Shutdown [235804 Shutdown] - shutting down level="ShutdownLevel_WorkloadManager" INFO loader [235549 MainThread] - All pipelines finished.
Splunk will have search concurrency issues if there are too many searches running at the same time. This search will identify them:
index=_internal sourcetype=scheduler status="skipped" | stats count by savedsearch_id reason
There may be other reasons why searches are skipped, but in this case, look for search concurrency messages.
Search Concurrency
There is a setting in limits.conf called max_searches_per_cpu. This setting controls the maximum number of concurrent historical searches for each CPU. It is calculated by adding base_max_searches + (#of cores * max_searches_per_cpu). max_searches_per_cpu is 1 by default, and base_max_searches is 6 by default. A 12-core system should be able to run 18 simultaneous searches by default. If the system in question is running more than the number of cores + 6 searches at a time, search concurrency issues can happen.
Expensive Searches
Searches that consume large amounts of memory can also cause OOM conditions in Splunk. The monitoring console has a dashboard that can be used to see expensive searches. The dashboard lives in the monitoring console under Search > Search Activity: Instance and is called "Top 20 Memory-Consuming Searches". Searches can be ad-hoc (run by a user) or saved searches.
Indexing Performance
Not only can indexers be affected by OOM conditions and other performance issues from search efficiency and/or over-scheduling, but they can also cause search heads to perform poorly on searches if the indexers are experiencing performance issues. Check indexers to ensure that data processing queues are not saturated. This can be done from the monitoring console by clicking on Indexing > Performance > Indexing Performance: Instance. There is a dashboard named "Median Fill Ratio of Data Processing Queues" which will provide the needed information.
Below SPL identifies any saved searches that are scheduled to run at the start of each minute, which can lead to too many searches running simultaneously and increased hardware resource usage.
It looks for all scheduled reports/alerts that are turned on and run every 5, 10, 15, or 30 minutes (or once an hour). It ignores certain types (like real-time or summary indexing). Then it proposes new cron schedules that spread the runs out across different minutes, so they don’t all start at the same time and overload the system.
|rest splunk_server=local /servicesNS/-/-/saved/searches
|search is_scheduled=1 disabled=0
|fields search title author cron_schedule eai:acl.app eai:acl.sharing dispatch.earliest_time dispatch.latest_time action.summary_index search id
| search NOT (dispatch.earliest_time=rt* OR dispatch.latest_time=rt* OR action.summary_index=1 OR search=*timechart* OR search=bin OR search=span OR search=*bucket*)
|eval cron_type=case
(match(cron_schedule,"\*/5 \* \* \* \*"),5,
match(cron_schedule,"\*/10 \* \* \* \*"),10,
match(cron_schedule,"\*/15 \* \* \* \*"),15,
match(cron_schedule,"\*/30 \* \* \* \*"),30,
match(cron_schedule,"^0 \* \* \* \*"),0,0=0,-1)
| where cron_type >= 0
|sort cron_type
| eval reset_count=case(cron_type=5,4, cron_type=10,9, cron_type=15,14, cron_type=30,29, cron_type=0,59)
| streamstats count reset_after="count=reset_count" by cron_type
| eval lower_bound=count
| eval upper_bound=59
| eval new_cron=case(cron_type=0,count . " * * * *", cron_type!=0,lower_bound . "-" . upper_bound . "/" . cron_type . " * * * *")
|fields title cron_schedule new_cron author count reset_count
In order to resolve out-of-memory conditions due to excessive searching, the system will require more memory to complete searches as they are, more search heads will need to be added to the cluster, or the searches can be reviewed and edited for efficiency. Before scaling any resources, search efficiency and scheduling should be reviewed. If this is a Cloud search head, adjusting search schedules and editing SPL to be more efficient will be the path to take. Please see the steps below to improve search performance:
About search optimization
Write better searches
Troubleshoot high memory usage
Resolve data quality issues
Configure event line breaking
limits.conf - Concurrency
Community Posts:
How to improve my search to identify queries which consume a large amount of memory?
Are there set guidelines for Splunk search best practices, and are there any other resources on this topic?
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
28-Sep-2026
|
Initial Release |