Search Delays resolved by tuning limits.conf

Applicability

Description

Issue

Large amount of searches seem to be delaying, and not getting completed.

This is likely related to search concurrency, and the limits.conf percentage settings under the scheduler stanza.

Solution

To solve it, in addition to making sure searches are scheduled efficiently (i.e. Staggered, and only run for required times and time ranges).

Look at scheduling to make sure it is staggered effectively.

Finally, tune concurrency percentages (depending on what it is set at).

Investigation

Check the current settings
Check to see what the current percentages are set to, and consider tuning, based on the below guidelines.

(Will need to open a ticket if using Splunk Cloud, to determine the settings.)
 

Concurrency Example

Adding this example to help understand the search concurrency settings.

The SHC Peers (4 total in this example - counting the Captain), have 80 CPUs each, so here is how it breaks down:

/opt/splunk/etc/system/default/limits.conf
[scheduler]
auto_summary_perc = 50
max_searches_perc = 50

And this section in the same file:

[search]
base_max_searches = 6
max_searches_per_cpu = 1

This translates into the following:   80 cpus x 1 + 6 = 86 searches (4 SHC example)

43 scheduled searches per SHC peer

43 adhoc searches per SHC Peer

Since this is set, the Captain does NOT run Scheduled searches:

/opt/splunk/etc/system/default/server.conf
[shclustering]
adhoc_searchhead = false
captain_is_adhoc_searchhead = true

TOTAL

43 x 3 scheduled searches (43 per SHC peer)

129 Total Scheduled searches
 

43 x 3 adhoc searches (43 per SHC Peer)

129 Total Adhoc searches

86 x 3 = 258 Total searches

References
limits.conf

Adhoc cluster member

server.conf

Additional

FYI, don't normally change "max_searches_per_cpu" in the cloud.

Also, changing "auto_summary_perc" and "max_searches_perc" becomes a "steal from one to give to the other" scenario in terms of performance.
( end up taking away from Scheduled or from Adhoc) the 86 total is still the limit in this example.

Resolution

Solution

These are often increased as follows (in limits.conf on the SH/SHC), but be careful. (Read the above links)

/opt/splunk/etc/system/default/limits.conf
[scheduler]
auto_summary_perc = 85
max_searches_perc = 70

(It "can" even be 100/75 respectively - not recommended)In the same file, this is also sometimes set to "2".

[search]
max_searches_per_cpu = 2