SmartStore Localization Timeouts and Partial Search Results

Applicability

Description

When multiple searches request buckets simultaneously, the Cache Manager must localize (download) those buckets from S3/remote storage.

If the number of requests exceeds the available worker threads or download slots, requests are queued. If a request remains in the queue longer than the search's allowed wait time, the search returns partial results. 

Investigation

Check splunkd.log for the following indicators:

• Timeout Errors: Action=EXPIRE, loctoken=..., status=failed, last_error='waitFor, timed out trying to localize files...'

• High Thread Usage: Look for high worker IDs (e.g., cachemanagerDownloadExecutorWorker-27).

• Heavy Eviction: Frequent logs showing large amounts of data being evicted to make room for new downloads.

Run the following command to check current limits: splunk btool server list cachemanager --debug

Resolution

Increase the Cache Manager concurrency limits in server.conf. A 2:1 ratio between threads and download slots is recommended to provide a buffer for metadata and eviction tasks.

Recommended Settings:
[cachemanager]
# Increase management threads
max_cache_manager_thread_pool_size = 40
# Increase simultaneous download slots
max_concurrent_downloads = 20

Note: Adjust these values based on available CPU cores and network bandwidth. If performance remains stable but errors persist, scale these values higher in the same 2:1 ratio. 

Related Links

10.2.0-configuration-file-reference/server.conf

implement-smartstore-to-reduce-local-storage-requirements

configure-smartstore

troubleshoot-smartstore