Cluster Master keeps making requests against peers to freeze already frozen buckets on SmartStore enable indexes

Available Languages

Updated:September 29, 2026
Document ID:SPLK046151

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise

Description

[Setup]

  • Indexers = 2.
    • Indexer Clustered.
    • SmartStore enabled (AWS S3).
  • Search Head = 1.
  • All instances are running Splunk Enterprise 8.0.6.
  • From reproduce setup, the following settings are in place, which are identical to the users setup and for reproduce purpose, the frozenTimePeriodInSecs is set to 600.
$ vi $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf
    [default]
    remotePath = volume:remote_store/$_index_name
    repFactor = auto    # Configure the remote volume    [volume:remote_store]
    storageType = remote    path = s3://vol01
    remote.s3.endpoint = http://minio01:9000
    remote.s3.access_key = minio
    remote.s3.secret_key = minio123
    # Minio requires the following settings to work fine with Splunk
    remote.s3.signature_version = v2
    remote.s3.use_delimiter = false    [_audit]
    frozenTimePeriodInSecs = 2592000    [_internal]
    frozenTimePeriodInSecs = 2592000    [_introspection]
    #frozenTimePeriodInSecs = 2592000
    frozenTimePeriodInSecs = 600    [_telemetry]
    frozenTimePeriodInSecs = 31622400    [_thefishbucket]
    frozenTimePeriodInSecs = 2592000

Investigation

  • Cluster Master reaches out to all its peers to gather a list for bucket information every 900 seconds. For the first round, CM has found buckets that needs to be frozen thus asks search peers to run the freeze process.
INFO  CMMasterRemoteStorageThread - Will initiate retrieving the list 
of buckets to be frozen for remote storage retention for idx=_introspection with 
frozenTimePeriodInSecs=600 with maxGlobalRawDataSizeMB=0 and maxGlobalDataSizeMB=0INFO  CMMasterRemoteStorageThread - retrieving the list of buckets to 
be frozen for remote storage retention for idx=_introspection with search=| dbinspect 
index=_introspection cached=true timeformat=%s| search state=warm OR state=cold| search modTime != 1
| stats max(endEpoch) AS endEpoch BY bucketId, startEpoch| search endEpoch<1606469163| fields 
bucketId, endEpoch, startEpochINFO  CMMasterRemoteStorageThread - 
freezing bid="_introspection~0~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751" of idx="_introspection" 
bucket_earliest=1606467862 bucket_latest=1606467895 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606469163 
bucket_age_seconds=33WARN  CMMaster - 
bid=_introspection~0~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not foundWARN  CMMasterRemoteStorageThread - Unable to find any peers for 
bid=_introspection~0~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 to run remote storage retentionINFO  CMMasterRemoteStorageThread - 
freezing bid="_introspection~0~684769E4-14B0-4090-98BD-19A507E78735" of idx="_introspection" 
bucket_earliest=1606467861 bucket_latest=1606467894 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606469163 
bucket_age_seconds=33WARN  CMMaster - 
bid=_introspection~0~684769E4-14B0-4090-98BD-19A507E78735 not foundWARN  CMMasterRemoteStorageThread - Unable to find any peers for 
bid=_introspection~0~684769E4-14B0-4090-98BD-19A507E78735 to run remote storage retentionINFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~1~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751" of idx="_introspection" 
bucket_earliest=1606467899 bucket_latest=1606467902 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606469163 
bucket_age_seconds=3WARN  CMMaster - 
bid=_introspection~1~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not foundWARN  CMMasterRemoteStorageThread - Unable to find any peers for 
bid=_introspection~1~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 to run remote storage retentionINFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~1~684769E4-14B0-4090-98BD-19A507E78735" of idx="_introspection" 
bucket_earliest=1606467899 bucket_latest=1606467902 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606469163 
bucket_age_seconds=3WARN  CMMaster - 
bid=_introspection~1~684769E4-14B0-4090-98BD-19A507E78735 not foundWARN  CMMasterRemoteStorageThread - Unable to find any peers for 
bid=_introspection~1~684769E4-14B0-4090-98BD-19A507E78735 to run remote storage retentionINFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751" of idx="_introspection" 
bucket_earliest=1606467866 bucket_latest=1606469115 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606469163 
bucket_age_seconds=1249INFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~2~684769E4-14B0-4090-98BD-19A507E78735" of idx="_introspection" 
bucket_earliest=1606467866 bucket_latest=1606469085 bucket_disk_size_MB=0 
bucket_raw_size_MB=139683461967168 reason="bucket age exceeded frozenTimePeriodInSecs" 
frozenTimePeriodInSecs=1606469163 bucket_age_seconds=1219INFO  CMMasterRemoteStorageThread - freezing 
buckets=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751,
_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 as part of remote storage retention on 
hp=172.20.0.4:8089
  • On indexer-node-01, we can see the two buckets (*bid=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751* and *bid=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735*) were successfully evicted around "11-27-2020 09:36:03.929".
INFO  BucketMover - Will freeze bkt=_introspection~2~4FD9262B-A3A4-4E44
-BD0C-F9E4417AD751 path='/opt/splunk/var/lib/splunk/_introspection/db/db_1606469115_1606467866_2_
4FD9262B-A3A4-4E44-BD0C-F9E4417AD751'INFO  BucketMover - Will freeze bkt=_introspection~2~684769E4-14B0-4090
-98BD-19A507E78735 path='/opt/splunk/var/lib/splunk/_introspection/db/db_1606469085_1606467866_2_
684769E4-14B0-4090-98BD-19A507E78735'INFO  DatabaseDirectoryManager - 
cache_id="bid|_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751|" found to be on remote storageINFO  BucketMover - RemoteStorageAsyncFreezer trying to freeze 
bid=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751, freezeInitiatedByAnotherPeer=falseINFO  DatabaseDirectoryManager - 
cache_id="bid|_introspection~2~684769E4-14B0-4090-98BD-19A507E78735|" found to be on remote storageINFO  BucketMover - RemoteStorageAsyncFreezer trying to freeze 
bid=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735, freezeInitiatedByAnotherPeer=falseINFO  CacheManager - will remove cache_id="bid|_introspection~2~4F9262B
-A3A4-4E44-BD0C-F9E4417AD751|" removeRemote=1INFO  CacheManager - will remove cache_id="bid|_introspection~2~68769E4
-14B0-4090-98BD-19A507E78735|" removeRemote=1INFO  CacheManager - Evicted cache_id="bid|_introspection~2~4FD9262B
-A3A4-4E44-BD0C-F9E4417AD751|" file_types="{"file_types":["dma_metadata","strings_data",
"sourcetypes_data","sources_data","hosts_data","lex","tsidx","bloomfilter","journal_gz","deletes",
"other"]}" freed_space=2500240 reason=manual earliest_time=1606467866 latest_time=1606469115INFO  CacheManager - Evicted cache_id="bid|_introspection~2~684769E4-
14B0-4090-98BD-19A507E78735|" file_types="{"file_types":["dma_metadata","strings_data",
"sourcetypes_data","sources_data","hosts_data","lex","tsidx","bloomfilter","journal_gz","deletes",
"other"]}" freed_space=0 reason=manual earliest_time=1606467866 latest_time=1606469085INFO  DatabaseDirectoryManager - idx=_introspection Writing a bucket 
manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=2 .  
Reason='Updating manifest: bucketUpdates=2'INFO  DatabaseDirectoryManager - Finished writing bucket manifest in 
hotWarmPath=/opt/splunk/var/lib/splunk/_introspection/dbINFO  BucketMover - RemoteStorageAsyncFreezer freeze completed 
succesfully for bid=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751INFO  BucketMover - RemoteStorageAsyncFreezer freeze completed 
succesfully for bid=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735
  • However, by next and every following round of remote_storage_retention_period, CM still tries asking search peers to freeze the the above two buckets.
egrep "_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751|_introspection~2~684769E4-14B0-4090-98BD-19A507E78735" $SPLUNK_HOME/var/log/splunk/splunkd.log
11-27-2020 09:51:04.625 +0000 INFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751" of idx="_introspection" 
bucket_earliest=1606467866 bucket_latest=1606469115 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606470064 
bucket_age_seconds=1249INFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~2~684769E4-14B0-4090-98BD-19A507E78735" of idx="_introspection" 
bucket_earliest=1606467866 bucket_latest=1606469085 bucket_disk_size_MB=0 
bucket_raw_size_MB=139683225620896 reason="bucket age exceeded frozenTimePeriodInSecs" 
frozenTimePeriodInSecs=1606470064 bucket_age_seconds=1219INFO  CMMasterRemoteStorageThread -  freezing 
buckets=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751,_introspection~2~684769E4-14B0-
4090-98BD-19A507E78735 as part of remote storage retention on hp=10.10.0.4:8089WARN  CMMasterRemoteStorageThread - unable to freeze 
buckets=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751,_introspection~2~684769E4-14B0-
4090-98BD-19A507E7873  as part of remote storage retention on hp=10.10.0.4:8089
... snip ...INFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751" of idx="_introspection" 
bucket_earliest=1606467866 bucket_latest=1606469115 bucket_disk_size_MB=0 bucket_raw_size_MB=0 
reason="bucket age exceeded frozenTimePeriodInSecs" frozenTimePeriodInSecs=1606474569 
bucket_age_seconds=1249INFO  CMMasterRemoteStorageThread - freezing 
bid="_introspection~2~684769E4-14B0-4090-98BD-19A507E78735" of idx="_introspection" 
bucket_earliest=1606467866 bucket_latest=1606469085 bucket_disk_size_MB=0 
bucket_raw_size_MB=139683288592784 reason="bucket age exceeded frozenTimePeriodInSecs" 
frozenTimePeriodInSecs=1606474569 bucket_age_seconds=1219INFO  CMMasterRemoteStorageThread -  freezing 
buckets=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751,_introspection~2~684769E4-14B0-4090
-98BD-19A507E78735 as part of remote storage retention on hp=10.10.0.4:8089WARN  CMMasterRemoteStorageThread - unable to freeze 
buckets=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751,_introspection~2~684769E4-14B0-4090
-98BD-19A507E7873  as part of remote storage retention on hp=10.10.0.4:8089
  • And on search peer side, since the above two buckets has been frozen, following entries were logged.
$ egrep "won't freeze" var/log/splunk/splunkd.log
ERROR BucketMover - 
Bucket=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~4FD9262B-A3A4-4E44-BD0C-F9E4417AD751 not found; won't freezeERROR BucketMover - 
Bucket=_introspection~2~684769E4-14B0-4090-98BD-19A507E78735 not found; won't freeze
  • From search peers, following rfs command returns no corresponding bucket information.
$ /opt/splunk/bin/splunk cmd splunkd rfs ls index:_introspection | egrep "2~"
  (nothing returns)

 

Resolution

Issue was due to the default value of repFactor is 0 for the indexes _introspection and _metrics. To workaround the issue set the value to auto in a local copy of indexes.conf. A code fix will be included in 8.1.0 and 8.0.8.

indexes.conf:
-----------------
[default]
repFactor = auto
[_introspection]
repFactor = auto
[_metrics]
repFactor = auto

Revision History

Revision Publish Date Comments
1.0
29-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products