Splunk Indexers | SmartStore CacheManager Failures and Partial Search Results | Failed to reserve free space

Applicability

Description

Splunk Instances utilizing SmartStore may exhibit severe performance degradation. This is characterized by persistent WARN CacheManager messages in splunkd.log, indicating an inability to allocate necessary disk space or perform timely data evictions within the local cache. The consequence is searches returning "partial results" and potential instability across the indexer cluster, including unfulfilled replication and searchable factors. Specific errors include:

08-06-2025 05:27:56.996 +0000 WARN CacheManager [1517243 cachemanagerDownloadExecutorWorker-2572] - Last run failed to evict requested bytes. Performing eviction in urgent mode for path=/cache/splunk/abc_efg_xyz/db/db_1732659729_1729793239_17216_191F6C86-7304-4E70-A5E4-xxxxx66095B0
08-06-2025 05:27:56.996 +0000 WARN CacheManager [1517243 cachemanagerDownloadExecutorWorker-2572] - Unable to evict enough data. Evicted size=0 instead of size=7106838528

Search UI may hit below Warnings during this issue occurrence.

The search process with
search_id="remote_shd1_1755061796.141041" may have returned partial results. Try running your search again. If you see this error repeatedly, review search.log for details or contact your Splunk administrator.

Investigation

1. Initial Symptoms: The primary observation was the recurrence of SmartStore bucket eviction warnings and searches returning "partial results" . This spike is noticed after a 9.3.3 upgrade but later confirmed no any connection with the upgrade .

2. Log Analysis: Examination of splunkd.log confirmed a surge in cache manager warnings, detailing issues with space reservation and data eviction. Specific errors like:

08-06-2025 05:27:56.996 +0000 WARN CacheManager [1517243 cachemanagerDownloadExecutorWorker-2572] - Last run failed to evict requested bytes. Performing eviction in urgent mode for path=/cache/splunk/abc_efg_xyz/db/db_1732659729_1729793239_17216_191F6C86-7304-4E70-A5E4-xxxxx66095B0
08-06-2025 05:27:56.996 +0000 WARN CacheManager [1517243 cachemanagerDownloadExecutorWorker-2572] - Unable to evict enough data. Evicted size=0 instead of size=7106838528

3. Followed SmartStore Troubleshooting manual to review the Smartstore health and confirm remote storage connectivity is good.  

4. SmartStore Cache Exhaustion: The local cache filesystem on the indexers was frequently reaching its max_cache_size, leading to "splunk cache was full and space could not be reserved" errors. When the cache was full, indexers could not reserve space for new data or evict older data, resulting in "Failed to reserve free space" and "Unable to evict enough data" warnings. This directly prevented searches from localizing necessary data, leading to partial results.

5. Unoptimized Searches: A key contributing factor for these smartstore errors  was the presence of poorly optimized searches, particularly  scheduled searches. Searches, configured with a broad index=* or without and index name and a high lookback period( for example 30 days lookback running every 15 mins schedule), scanning billions of events and causing massive bucket downloads, overwhelming the indexers' local cache and I/O capabilities. 

May consider running below query to identify bad searches with huge number of event scan count : (this is a modified version of Top 20 Memory consuming searches under Monitoring console)

index=_internal host=<search_head>
sourcetype=splunk_resource_usage data.search_props.sid::* data.search_props.mode!=RT
| eval process = 'data.process'
| eval args = 'data.args'
| eval pid = 'data.pid'
| eval ppid = 'data.ppid'
| eval elapsed = 'data.elapsed'
| eval mem_used = 'data.mem_used'
| eval mem = 'data.mem'
| eval pct_memory = 'data.pct_memory'
| eval normalized_pct_cpu = 'data.normalized_pct_cpu'
| eval pct_cpu = 'data.pct_cpu'
| eval sid = 'data.search_props.sid'
| eval app = 'data.search_props.app'
| eval label = 'data.search_props.label'
| eval type = 'data.search_props.type'
| eval mode = 'data.search_props.mode'
| eval user = 'data.search_props.user'
| eval role = 'data.search_props.role'
| eval scan_count = 'data.search_props.delta_scan_count'
| eval label = if(isnotnull('data.search_props.label'), 'data.search_props.label', "")
| eval provenance = if(isnotnull('data.search_props.provenance'), 'data.search_props.provenance', "unknown")
| eval search_head = case(isnotnull('data.search_props.search_head') AND 'data.search_props.role' == "peer", 'data.search_props.search_head', isnull('data.search_props.search_head') AND 'data.search_props.role' == "head", "_self", isnull('data.search_props.search_head') AND 'data.search_props.role' == "peer", "_unknown")
| eval workload_pool = if(isnotnull('data.workload_pool'), 'data.workload_pool', "UNDEFINED")
| stats max(scan_count) as scan_count max(elapsed) as runtime max(mem_used) as mem_used earliest(_time) as _time  by sid, label, provenance, type, mode, app, role, search_head, user
| eval mem_used = round(mem_used, 2)
| sort 200 - scan_count 
| fields - day, hour, minute, second
| eval _time = strftime(_time,"%+")
| rename sid as SID, label as "Search Name", provenance AS Provenance, type as Type, mode as Mode, app as App, role as Role, search_head as "Search Head", user as User,scan_count as Scanned_Events, mem_used as "Memory Usage (MB)", _time as Started, runtime as Runtime

Similar investigation can be done under "Search Head UI --> Activity --> Jobs" tab. May sort with the event count to identify potential bad searches. 

6. Infrastructure Assessment: Noticed  High I/O wait times on few indexers cache FS, indicating potential underlying storage performance bottlenecks.

Resolution

Search Optimization: Few highly resource-intensive scheduled searches, were identified as a primary contributor and subsequently disabled and tweaked. Also noticed few dashboard panels with  index=* or an index name is absent . Workload management rules were implemented and refined to control and prioritize search execution, preventing system overload.

Role mapping : If necessary, update the role mappings in authorize.conf with the appropriate srchIndexesDefault setting. Adding multiple indexes to srchIndexesDefault will allow users to search across a broader range of indexes whenever a search is executed without specifying the "index=" stanza.

srchIndexesDefault = <semicolon-separated list>
* A list of indexes to search when no index is specified.
* These indexes can be wild-carded ("*"), with the exception that "*" does not
  match internal indexes.
* To match internal indexes, start with an underscore ("_"). All internal indexes are
  represented by "_*".
* The wildcard character "*" is limited to match either all the non-internal
  indexes or all the internal indexes, but not both at once.
* No default.

SmartStore Cache Tuning: 

Review cachemanager settings to confirm a good max_cache_size  and eviction_padding  applied to safeguard the cache file system. Note that the default eviction_padding setting [Default: 5120 (~5GB)] may not be suitable for large (terabyte-sized) cache filesystems, which can result in cache filesystem usage consistently reaching 99%. It is recommended to configure an appropriate max_cache_size to keep cache filesystem usage below 95%  and so as to avoid Write Amplification. 

Also may consider increasing hotlist_bloom_filter_recency_hours setting from 15 days to a higher value. (For example 30days)

The hotlist_bloom_filter_recency_hours setting protects certain small metadata files, such as the bloomfilter file, from eviction. By inspecting such metadata files, the cache manager can sometimes eliminate the need to fetch larger bucket files, such as the rawdata journal and the tsidx files, from remote storage when handling search requests.

hotlist_bloom_filter_recency_hours = <unsigned integer>
* When a bucket's non-journal and non-tsidx files (such as bloomfilter files)
  are older than this value, those files become eligible for eviction. Bloomfilter
  and associated files younger than this value are evicted only if there are
  no older files eligible for eviction.
* The recency of a bloomfilter file is based on its bucket's recency and is
  calculated in the same manner described for 'hotlist_recency_secs'.
* This setting works in concert with 'hotlist_recency_secs' which is designed to be
  configured for a shorter age. If 'hotlist_recency_secs' leads to the eviction of a
  bucket, the bloomfilter and associated files will continue to remain in the cache
  until they reach the age configured by this setting. Thus, the bucket will remain
  in cache, but without its journal and tsidx files.
* This setting can be overridden on a per-index basis in indexes.conf.
* Default: 360 (15 days)

Workload management: Workload management rules can be implemented and refined to control and prioritize search execution, preventing system overload

Related Links

Cachemanager settings

SmartStore Troubleshooting manual

Write Amplification

Server.conf

authorize.conf/srchIndexesDefault