There are many entities AND many services/KPIs (8 KPIs per template/service). As a result, each run of the KPI base searches are returning tens of thousands of results, which may exceed the limit when ITSI transports the processed search results to the ITSI summary index
To avoid the issue : reduce the number of results, by distributing the services to multiple shared base searches.
The ID in the search name is the id of the shared base search.
index=_internal source=*scheduler.log* shared indicator | stats min(result_count) AS min_result_count avg(result_count) AS avg_result_count max(result_count) AS max_result_count min(run_time) avg(run_time) max(run_time) by savedsearch_name | sort -max_result_count |
Or
increase the limit (do not go beyond 100000)
[scheduler] * The maximum number of results to load when triggering an alert action. * Default: 50000 max_action_results = 50000
3. You will need to calculate and adjust the above number yourself, which is based on the following formula:
limit = (number of KPIs * number of entities for each service) + (number of services) * 2NOTE: As a result of increasing this limit, ITSI will be using more memory as more memory will be allocated to support the # of results
Top 3 Best Practices for Configuring Splunk ITSI in a Large-Scale Environment
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
28-Sep-2026
|
Initial Release |