Data retention Is not working as expected.

Applicability

Description

Scenarios:

1. The Data Retention was set to 380 days with frozenTimePeriodInSecs in indexes.conf, but when looking through DMC, the user found that the data age was even older than 380 days, so the user wanted to confirm retention of 380 days and the reason for seeing events older than 380 days for some of the indexes.

2. Data is not searchable for the time period specified in frozenTimePeriodInSecs for a specific index, data is freezing earlier than configured.

3. Different scenario where the data is rolling earlier than the retention period.

Investigation

1. This search/query can be used to check if the events on the index have more than 380 days, run the following query from the search head. It will display all events stored on the index in question.
 

| dbinspect index= <index in question>
| convert ctime(endEpoch), ctime(startEpoch) 
| table splunk_server path state startEpoch endEpoch sizeOnDiskMB 
| sort startEpoch

It can also be checked from the Indexer's CLI - go to $SPLUNK_HOME/var/lib/splunk/<...>/db and run the following command:
 

ls -d1 ?b_*| gawk -F'_' '{print $0 "\n\tcontains data from "strftime("%c", $3) " to " strftime("%c", $2)}'

As well, confirm if there are no ".rbsentinel buckets" that might be causing the issue, and for that reason, events with more than 380 days are showing in the DMC. The .rbsentinel buckets are stored in $SPLUNK_HOME/var/lib/splunk/<...>/db. This type of buckets always have a size of zero, and do not affect the storage or performance of the instance.

2. Identify the index you are looking to search against. In this example, it will be index = Main

If you have been searching Main for the past 12 months and your data is only pulling back for the last 4 months. This shows that your expected retention period frozenTimePeriodInSecs = 31557600 is not being adhered to. 

We can look to start searching across internal indexes to find out why data is rolling. 

This search below should assist with finding the freeze action by the bucket mover on indexes in your environment. 

index=_internal sourcetype=splunkd component=BucketMover "Will attempt to freeze" | rex field=_raw "/logs/splunk_logs/(?P<index_name>[^\/]+)/(db|colddb)/db_(?P<earliest_epoch>[\d]+)_(?P<latest_epoch>[\d]+)_(?P<bucket_number>[\d]+)\' (?P<reason>.*)"  | convert ctime(earliest_epoch) as Earliest_Data  | convert ctime(latest_epoch) as Latest_Data  | convert ctime(_time) as Log_TimeStamp | table Log_TimeStamp,index_name,bucket_number,Earliest_Data,Latest_Data,reason | sort - Log_TimeStamp

The fields listed in the search above will show the below 
 

Log_TimeStamp = _time of the log entry
index_name = index that's being frozen
bucket_number = bucket that's being frozen
Earliest_Data = Earliest date of events that was in that bucket
Latest_Data = Latest date of events that was in that bucket
Reason = the reason it was rolled to frozen.

This search below will look to find any specific buckets being rolled due to data size. 

index=_internal sourcetype=splunkd (source=*splunkd.log) "BucketMover - will attempt to freeze" NOT "because frozenTimePeriodInSecs=" 
| rex field=bkt "(rb_|db_)(?P<newestDataInBucket>\d+)_(?P<oldestDataInBucket>\d+)"
| eval newestDataInBucket=strftime(newestDataInBucket, "%+"), oldestDataInBucket = strftime(oldestDataInBucket, "%+") 
| table message, oldestDataInBucket, newestDataInBucket

splunkd.log:02-07-2023 07:14:26.862 -0600 INFO BucketMover [12792 IndexerService] - will attempt to freeze bkt="/opt/splunk/var/lib/splunk/colddb/db_1661365476_1622171436_1289'" maxTotalDataSize=590558003200 bytes, diskSize=590559264768 bytes"splunkd.log:02-07-2023 07:14:27.456 -0600 INFO BucketMover [21224 FilesystemOpExecutorWorker-0] - AsyncFreezer freeze succeeded for bkt='/opt/splunk/var/lib/splunk/colddb/db_1661365476_1622171436_1289'

The messages above in splunkd.log we can see that the reason we are rolling is because of maxTotalDataSize=590558003200 bytes and the actual data on diskSize=590559264768 bytes.

• DiskSpace = 590.559264768 GB
• maxTotalDataSize = 590.5580032 GB

As the Disk Size is larger than maxTotalDataSize the Cluster Manager will look to roll the oldest data off in the index to adhere to maxTotalDataSize.

3.The first thing to check is if the volume limit is being reached before the time limit. For that, the Splunk internal logs can be checked, specifically the BucketMover processor, which indicates which retention limit was hit. From there, the current configurations can be checked with a Splunk CLI btool command or checking the logs from the indexer.

If the Splunk internal logs indicate the size limit was reached the next settings need to be checked:

indexes.conf on the indexers:
coldPath.maxDataSizeMB = <value>
maxTotalDataSizeMB = <value>
frozenTimePeriodInSecs = <value>

In this case, the smallest setting between coldPath.maxDataSizeMB and maxTotalDataSizeMB gets applied, so, make sure maxDataSizeMB is not smaller than maxTotalDataSizeMB.

Resolution

1. Based on the warm bucket (db_1572505379_1568369613_1) and end Epoch date 10/31/2019, The events don't have 380 days. The buckets had 357 days, and therefore they hadn't rolled to frozen due to retention policy has not being reached.
Bear in mind, that all events have to expire the retention policy to be rolled to the frozen bucket or deleted.
 

frozenTimePeriodInSecs = <nonnegative integer>
* The number of seconds after which indexed data rolls to frozen.
* If you do not specify a 'coldToFrozenScript', data is deleted when rolled to
frozen.
* NOTE: Every event in a bucket must be older than 'frozenTimePeriodInSecs'
seconds before the bucket rolls to frozen.
* The highest legal value is 4294967295.
* Default: 188697600 (6 years)

In other words, the whole bucket will apply the retention policy until all the events reach the time.

2.Increasing maxTotalDataSizeMB for this specific index can help by allowing the index to grow to a larger size before rolling off data to frozen. Refer to the following document for more information.

Re-assess data ingestion for this specific index. Refer to the following document to estimate the index's storage requirement.

3.Then, if there is no specific reason to use coldPath.maxDataSizeMB, that setting needs to be removed and the indexes only need to have maxTotalDataSizeMB with the right value so that the data does not hit the size limit before the time limit.

Therefore, just remove coldPath.maxDataSizeMB, set maxTotalDataSizeMB with the right value and deploy the change to the indexer or indexers. 

Note: In scenarios where a volume is involved, the setting maxVolumeDataSizeMB needs to be reviewed as well. If this is reached data on the volume would be frozen.