Index Archiving: Unexpected bucket names observed in coldToFrozenDir after archiving

Applicability

Description

An administrator configures archiving for Splunk indexes (such as _audit, _internal, or custom indexes) by setting coldToFrozenDir and lowering frozenTimePeriodInSecs in indexes.conf.

Before archiving, an inventory of bucket directory names, file sizes, and MD5 checksums of rawdata/journal.zst is captured from the warm path:

find /mnt/data/splunk_data/[index]/db/ -name journal.zst -exec ls -l {} \;
find /mnt/data/splunk_data/[index]/db/ -name journal.zst -exec md5sum {} \;

After the archive operation completes, the administrator inspects the archive destination:

find /mnt/data/splunk_data/[index]/archiveData/ -name journal.zst -exec ls -l {} \;
find /mnt/data/splunk_data/[index]/archiveData/ -name journal.zst -exec md5sum {} \;

Symptoms:

• Many bucket directory names present in archiveData/ do not exist in the pre-archive inventory collected from db/.
• For buckets whose names do match between pre- and post-archive lists, the MD5 checksums of journal.zst match exactly.
• This discrepancy raises concerns that bucket directory names were modified or that unexpected data was archived during the process.

Investigation

1. Check effective index storage paths:

   Run btool on the indexer to verify the configured homePath (warm/db) and coldPath (cold/colddb) locations:

   splunk btool indexes list [index_name] --debug | grep -E "(homePath|coldPath|coldToFrozenDir)"

2. Verify the pre-archive inventory scope:

   Confirm whether the pre-archive listing was collected only from the db/ directory or whether colddb/ was also inventoried.

3. Inspect splunkd.log for freezing events:

   Review BucketMover log entries on the indexer during the archive window for one of the unmatched bucket directory names:

   index=_internal host=[indexer] sourcetype=splunkd component=BucketMover "will attempt to freeze" "[bucket_name]"

   Example log showing the bucket originating from colddb/:

   INFO BucketMover - will attempt to freeze: candidate='/mnt/data/splunk_data/audit/colddb/db_1759776150_1757879872_396_CDC7D20B-17C9-4AFC-9D9D-264DC585F108' because frozenTimePeriodInSecs=10368000 is exceeded...
   INFO BucketMover - AsyncFreezer freeze succeeded for bkt='/mnt/data/splunk_data/audit/colddb/db_1759776150_1757879872_396_CDC7D20B-17C9-4AFC-9D9D-264DC585F108'

   If the source candidate path shows colddb/, the bucket was already residing in the cold tier before being frozen.

Resolution

Cause

Bucket freezing via coldToFrozenDir moves expired buckets from both the warm directory (homePath / db) and the cold directory (coldPath / colddb). Splunk preserves the exact bucket directory name and raw data when archiving buckets. If the pre-archive inventory is collected only from the db/ directory, buckets that had already aged to colddb/ are omitted from the initial list, creating an apparent bucket name mismatch in archiveData/.

Remediation

1. No correction is required: The bucket names in archiveData/ are the original, unaltered bucket directory names. Do not rename any directories in archiveData/.
2. Data integrity is intact: For all buckets archived, journal.zst contents and checksums remain untouched.

Prevention & Best Practices

1. For future archive operations, collect pre-archive inventories and checksums from both db/ and colddb/:

   find /path/to/[index]/db/ -name journal.zst -exec md5sum {} \; > pre_archive_db.txt
   find /path/to/[index]/colddb/ -name journal.zst -exec md5sum {} \; > pre_archive_colddb.txt

2. Confirm active storage paths with splunk btool indexes list [index] --debug prior to running inventory scripts.

Related Links

• Splunk Documentation: Archive indexed data
• Splunk Documentation: indexes.conf reference
• Splunk Documentation: How the indexer stores indexes