The day before indexers were added to the cluster, all searches were running nominally with 0.00% skipped searches. The day after the indexers were added, skipped searches were seen for the entire day and users reported slow search performance. The new indexers were added successfully and were all in a healthy state.
While testing searches and reviewing the search.log files in Job Inspector, it was noticed that one of the indexers was stuck waiting to download a bucket.
Details on how to open the Job Inspector and search.log can be found in the links below:
View search job properties and What the Search Job Inspector shows you .
Restarting this indexer cleared the issue. However, the issue was then seen on another indexer. An Indexer Cluster Rolling Restart was carried out, which allowed all indexers to successfully complete their bucket downloads.
The issue with the indexer was viewed in the search.log found in the Search Job Inspector. The error seen was:
ERROR S2BucketCache [3517093 StreamSearch] - waitFor, timed out trying to localize files from remote storage, cache_id="bid|infra~111~484E6253-3624-41C4-843D-7C09C219EA2D|" download_status=queued files={"file_types":["sources_data"]} local_files={"file_types":[]} probe_count=5907ERROR DatabaseDirectoryManager [3517093 StreamSearch] - Bucket still not local after waiting, bid=infra~111~484E6253-3624-41C4-843D-7C09C219EA2D FailureSource=none
Perform a rolling restart of an indexer cluster
Manage indexes in Splunk Cloud Platform
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |