Hot/warm volume is not following data retention policies in Splunk Enterprise

Applicability

Splunk Enterprise 8.1.0 Splunk Enterprise 8.1.3 Splunk Enterprise 8.1.4 Splunk Enterprise 8.2.1 Volume Management

Description

Volume Management maxVolumeDataSizeMB not be adhered to due to a known issue.

Investigation

These 3 Items need to be checked in order to validate this known issue. 

• Verifying maxVolumeDataSizeMB is set correctly for a specific volume that is showing issues. 
  • This can be done by doing a btool on indexes.conf to show volume settings and their locations where this is set. 

$SPLUNK_HOME/bin/splunk btool inputs list --debug

• If maxVolumeDataSizeMB is correctly set and is being exceeded on the partition by seeing the size on disk for the particular volume mount location.
  • For Linux you can use :

df -h

 Show mounts for the filesystem

du -sh

To show the human-readable output of the disk size. 

• The version matches
  • Splunk Enterprise 8.1.0 
  • Splunk Enterprise 8.1.3 
  • Splunk Enterprise 8.1.4
  • Splunk Enterprise 8.2.1

Upgrade as well as run through steps to recreate bucket manifests for the largest indexes affected to trim disk space.

Resolution

Upgrade to a known fixed version 

8.1.5 ,8.1.6, 8.2.2

After upgrading to a fixed version,  still not seeing the volume manager can confine the hot/warm volume usage to the maxVolumeDataSizeMB setting.

Based on research, it was found that the buckets – that were rolled to warm after the 8.1.5 upgrade – had the right values for the bucket size in the .bucketManifest files, however, the buckets that were already in a warm state (before the upgrade done) may still be having incorrect value for the bucket size (which is size_on_disk field within .bucketManifest file for warm buckets). Such older warm buckets would still have an impact on the volume manager trimming calculation so it may *not* still see the trimming of hot/warm volume is done to the level defined in maxVolumeDataSizeMB. 

To get a total solution, refresh the .bucketManifest files for all the indexes. *The best time to do the refresh is when going through the upgrade. Then going through the following steps again is not needed. (which will need a restart of indexers).* 

To re-generate the .bucketManifest for  all the indexes

1. On each indexer:

2. Stop Splunk

3. Go to the index whose .bucketManifest , want to re-generate 

   cd $SPLUNK_HOME/var/lib/splunk/my_index_name/db
   

4. Back up the .bucketManifest file to /tmp/.bucketManifest_my_index_name. – this is just in case , see any issues and , want to revert, have the file saved.

5. Remove .bucketManifest

6. Repeat steps 3-5 for all the indexes.

7. Restart Splunk

8. See the newly generated .bucketManifest file inside all indexes.

Having already gone through an upgrade to a fixed version (like 8.1.5) and have not refreshed bucket Manifest as pointed out above, re-evaluate the need to run the above steps or not. If the hot/warm volume is slowly and steadily getting more free space (that is complying with the value of maxVolumeDataSizeMB or getting very close to it), then it is also ok to NOT go through the above steps. As time progresses, newer and newer warm buckets will show up (that will have the correct size_on_disk value for the bucket). And slowly the older warm buckets will age out. That will drive the volume manager to trim at the right mark.

Related Links

8.2.1/ReleaseNotes/Knownissues#Indexer_and_indexer_clustering_issues

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release