Error messages on SH GUI

Applicability

Description

Splunk Architecture: 
48 Universal Forwarder, single Indexer+search head.

Messages on UI: INFO messages like "HotBucketRoller - finished moving hot to warm bid=xxxxxxxxx"

Investigation

The percentage of small buckets (55%) created over the last hour is high and exceeded the red thresholds (50%) for index=_internal, and possibly more indexes, on this indexer.

Here is the root cause behind small buckets,

Splunk uses buckets as an index directory to index the data. Now, when an event comes to Splunk for indexing, the new hot bucket will be created for that event or the event is indexed to the existing one of the hot buckets as per the event's timestamp and constraints on the index(refer to maxHotBuckets, maxHotSpanSecs and quarantinePastSecs parameters of indexes.conf).

So, if the event is having the older/strange timestamp than acceptable for the existing buckets, then Splunk will create a new hot bucket to index that event. Hence, the number of hot buckets created may reach or cross the threshold.

In the screenshot, I can see the INFO messages like "HotBucketRoller - finished moving hot to warm bid=xxxxxxxxx"

Resolution

This will not cause any performance problems. But, to fix these messages you can increase the threshold as below:-

>> Create the local folder, set the config file

$SPLUNK_HOME$/etc/system/local/health.conf,[feature:buckets]
indicator:percent_small_buckets_created_last_24h:red = 100
indicator:percent_small_buckets_created_last_24h:yellow = 50

Also, increase hot buckets for internal index:
 

$SPLUNK_HOME$/etc/system/local/indexes.conf,[_internal]
maxHotBuckets = 5

To make the above changes effective, restart the Splunk service.

Related Links

Splunk Documentation
indexes.conf
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf