Error while trying to search bucket - Please check/repair bucket

Applicability

Splunk Enterprise 8.1.1

Description

Errors occurred while a search was executing. Therefore, search results might need to be completed.
 

[subsearch] Error while trying to search bucket=pan_logs~4539~A7949D79-CCAD-4A1C-8637-0EC3257FEBEC (error='Failed to read compression bits from bucket=pan_logs~4539~A7949D79-CCAD-4A1C-8637-0EC3257FEBEC - exception thrown: JournalSliceDirectory: Cannot seek to rawdata offset 0, path="/opt/splunk/var/lib/Splunk/pan_logs/db/db_1631366845_1631312836_4539_A7949D79-CCAD-4A1C-8637-0EC3257FEBEC/rawdata" Please check/repair bucket path='/opt/splunk/var/lib/splunk/pan_logs/db/db_1631366845_1631312836_4539_A7949D79-CCAD-4A1C-8637-0EC3257FEBEC' with 'fsck' as it could be corrupted.'). Results may be incomplete!

Investigation

Locate the corrupted bucket. 

Resolution

Step 1 - Download the bucket

$SPLUNK_HOME/bin/splunk cmd splunkd rfs -- getR bucket:<bid>/receipt.json /tmp

Please put it on the test index, this can be done on any standalone test instance of Splunk

Step 2 - After the bucket is restored on the Splunk instance

restart the Splunk.

Step 3 - After restarting, search for the bucket using

index=<<index_NAME_WHERE_BUCKET_IS_RESTORED>

Step 4 - After the bucket is fixed
re-upload to the remote store using the steps below

In the current Production.
Put the Cluster Master in Maintenance Mode.
Restore these fixed buckets from standby indexer to PRODUCTION_INDEXER, restore the bucket in the corresponding correct index
Remove the bucket from local and remote
NOTE: Please change "_internal~266~761A77A2-6676-4BF9-83CD-1CB243ED61BF" with your own bucket name.

curl -k -u admin: password https://localhost:8089/services/cluster/master/buckets/_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71/remove_all -X POST

Step 5 - Performed on the CM
Works only in a Cluster environment
Only deletes local buckets

Delete bucket from the remote store (CLI )

$SPLUNK_HOME/bin/splunk cmd splunkd rfs -- rmV bucket:_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71

Start the indexer.

Step 6 - Queue these restored buckets for upload to remote - run curl like below from the indexer where buckets were restored. Below is just an example you need to replace with your own buckets example:

curl -k -u admin:changeme -X POST https://localhost:10041/services/admin/cacheman/_bulk_register -d cache_id="bid|taktaklog~1~F7770FEB-F5A6-4846-A0BB-DDC05126BBF6|"curl -k -u admin:changeme -X POST https://localhost:10041/services/admin/cacheman/_bulk_register -d cache_id="bid|taktaklog~1~F7770FEB-F5A6-4846-A0BB-DDC05126BBF6|"

Here is an example to upload multiple buckets

curl --netrc-file nnfo -k -X POST https://localhost:8089/services/admin/cacheman/_bulk_register -d cache_id="bid|nccihub~17~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~22~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~28~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~34~024011E7-E61E-45CE-82DE-732038D5C276|" -d cache_id="bid|nccihub~12~E646664A-D351-41E4-BBE7-5B02A08C44C9|" -d cache_id="bid|nccihub~17~F876C294-3E3E-488A-8344-16727AC34C52|" -d cache_id="bid|nccihub~17~E646664A-D351-41E4-BBE7-5B02A08C44C9|"

Step 7 - restart the indexer, and the bucket should get uploaded you should be able to check these using the command

$SPLUNK_HOME/bin/splunk cmd splunkd rfs -- ls bucket:_internal~266~761A77A2-6676-4BF9-83CD-1CB243ED61BF

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release