Rbsentinel files getting created in the indexes (hot/cold) although the index log retention logic is set

Applicability

Description

Issue reported
Why are rbsentinel files getting created in the indexes (hot/cold) although the index log retention logic is set

Error

Root Cause(s):The percentage of small buckets (81%) created over the last hour is high and 
exceeded the red thresholds (50%) for index=_introspection, and possibly more indexes, on 
this indexer. At the time this alert fired, total buckets created=11, small buckets=9

Investigation

Check on line what could be the reason for this, then have come across these points:

>Bucket retention settings w.r.t homepath, cold path (I have disabled homepath.maxdb by the way)
>Too many .rbsentinel files in hot/cold db of an index.
>Timestamp issues (Splunk uses buckets as an index directory to index the data. Now, when an event comes to Splunk for indexing, the new hot bucket will be created for that event or the event is indexed to the existing one of the hot buckets as per the event's timestamp and constraints on the index(refer to maxHotBuckets, maxHotSpanSecs and quarantinePastSecs parameters of indexes.conf). So, if the event is having the older/strange timestamp than acceptable for the existing buckets, then Splunk will create a new hot bucket to index that event. Hence, the number of hot buckets created may reach or cross the threshold.)

Post deleting the .rbsentinel files the issue got fixed

Resolution

Solution
The .rbsentinel files are used on Splunk for win32/win64 as part of the locking strategy, to ensure exclusive access to buckets for certain types of modifications. Windows doesn't allow locking calls on directories with the same semantics as desired.
 
They are mostly useless on a unix-based system, but in the interests of the index data format being able to traverse platforms are produced nonetheless.
 
When using Linux as their OS platform they can be deleted.
 

Related Links