SmartStore enabled indexes causing indexers to run out of local drive space

Applicability

Description

The CacheManager maximizes search efficiency through intelligent management of the local cache. It favors retaining in the cache copies of buckets and files that have a high likelihood of participating in future searches. When the cache fills up, the CacheManager removes, or "evicts," copies of buckets that are least likely to participate in future searches.

Since CacheManager eviction only removes the cached copies of buckets, the eviction process does not result in data loss. The manager copies continue to reside in remote storage.

If the SmartStore CacheManager is not configured correctly, the indexer CacheManager may allow the local data partition on the indexer to fill up, which can result in indexing and search performance issues.

Investigation

For SmartStore deployment, the indexer's storage partition space is managed by the CacheManager settings that reside in the server.conf stanza [cachemanager].

1. The max_cache_size setting specifies the maximum occupied space, in megabytes, for the disk partition that contains the cache.
2. The eviction_padding setting controls the amount of additional space, in megabytes, that the CacheManager protects.
   
   

Splunk Indexer will begin evicting warm buckets locally from the indexer when either the space usage exceeds the max_cache_size value or the available disk space on the indexer drops below eviction_padding.

If max_cache_size is set to 0 (as it is by default), the CacheManager will instead begin evicting cache data from remote-storage-enabled indexers when free space on a partition falls below eviction_padding.

server.conf
[cachemanager]
eviction_padding = 5120 (default)
max_cache_size = 0 (default
 

Addition Note: For smarsttore enabled  indexes.conf, the following settings are required:
 
     remotePath
     maxDataSize - for SmartStore-enabled indexes, best practice is to use "auto", which allows for 750MB buckets.
 
 The following settings apply the retention policies for SmartStored enabled indexes:
 
     maxGlobalDataSizeMB
     maxGlobalRawDataSizeMB
     frozenTimePeriodInSecs
 
 It should be noted, the following drive space related settings are ignored entirely for remote indexes:
 
     coldPath and thawedPath (but must still be specified)
     coldPath.maxDataSizeMB
     homePath.maxdataSizeMB
     maxWarmDBCount
     maxTotalDataSizeMB 
     warmToColdScript
    maxVolumeDataSizeMB

Resolution

This configuration for smartstore  is not specific to an individual index but for the whole storage partition on indexer.

For locally attached storage like NVMe, recommendation is to set 10% free space on partition.

[cachemanager]
eviction_padding = <10% of disk Space on Partition (in megabytes)>
max_cache_size=0

These recommendation is for AWS but should be the same for other S3 providers like Azure or GCP.
In some cases strict setting like 5%  was utalized after testing. As 10% on i3en.12XL is 3TB, and 50x3TB is quite a bit unusable disk space .

In a clustered environment, these settings can be configured from the Cluster Manager app, which sends the server.conf configurations to your indexers (the default _cluster app is designed for this purpose, but this can also be accomplished using a custom app). These apps are located on the Cluster Manager in $SPLUNK_HOME/etc/manager-apps (or, for older versions of Splunk, $SPLUNK_HOME/etc/master-apps) and pushed to the indexers from the Cluster Manager's UI or via the CLI.

Please note, in indexer clustering, apps deployed to indexers by the Cluster Manager are located in $SPLUNK_HOME/etc/peer-apps (or, for older versions of Splunk, $SPLUNK_HOME/etc/slave-apps, and such settings take precedence over any configurations set in $SPLUNK_HOME/etc/system/local. Working with a clustered environment but cannot deploy apps from the Cluster Manager because of space limitations, please first check the Cluster Manager for excess buckets. If there are any, evict all excess buckets to gain additional drive space. Additional space may gained by cleaning indexers' dispatch directory. 

Note: 

Related Links

• server.conf specification
• indexes.conf specification
• Splunk Documentation: Configure S3 remote store for SmartStore
• Splunk Documentation: Configure the SmartStore cache manager