Splunk Instance health is showing RED for the Bucket component

Applicability

Splunk Enterprise

Description

The percentage of small of buckets created (100) over the last hour is very high and exceeded the red thresholds (50) for index=<index name>, and possibly more indexes, on this indexer.

ERROR messages in the Health check tab:

Buckets 
Root Cause(s): 
The percentage of small of buckets created (100) over the last hour is very high and exceeded the red thresholds (50) for index=mbam, and possibly more indexes, on this indexer 

Last 50 related messages: 

07-02-2019 02:16:05.989 -0700 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4984~99D2CFA5-1005-45DA-8C68-0E8A5FB0C95C idx=_internal 
from=hot_v1_4984 to=db_1562087584_1562043339_4984 size=1048645632 caller=size_exceeded _maxHotBucketSize=1048576000 (1000MB), bucketSize=1048641536 (1000MB) 
07-02-2019 00:22:57.931 -0700 INFO HotBucketRoller - finished moving hot to warm bid=_introspection~2046~99D2CFA5-1005-45DA-8C68-0E8A5FB0C95C idx=_introspection 
from=hot_v1_2046 to=db_1562027109_1562015374_2046 size=569110528 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots 
07-02-2019 00:00:22.067 -0700 INFO HotBucketRoller - finished moving hot to warm bid=cloudflare~324~99D2CFA5-1005-45DA-8C68-0E8A5FB0C95C idx=cloudflare 
from=hot_v1_324 to=db_1561885206_1561791602_324 size=394207232 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots 
07-01-2019 23:03:37.981 -0700 INFO HotBucketRoller - finished moving hot to warm bid=_internal~4983~99D2CFA5-1005-45DA-8C68-0E8A5FB0C95C idx=_internal 
from=hot_v1_4983 to=db_1562076172_1562032683_4983 size=1054523392 caller=size_exceeded _maxHotBucketSize=1048576000 (1000MB), bucketSize=1054404608 (1005MB) 
07-01-2019 22:43:06.030 -0700 INFO HotBucketRoller - finished moving hot to warm bid=clearpass~362~99D2CFA5-1005-45DA-8C68-0E8A5FB0C95C idx=clearpass 
from=hot_v1_362 to=db_1561787045_1561538116_362 size=393191424 caller=lru maxHotBuckets=3, count=4 hot buckets,evicting_count=1 LRU hots

 

Investigation

The message above is saying that 100% of the buckets created for the last 24 hours is less than 10% of maxDataSize. This will not cause any performance problem. To fix this warning, you can increase the threshold from 50 to 150 as below.

Default configuration is in the location  $SPLUNK_HOME/etc/system/default/health.conf

[feature:buckets]
display_name = Buckets
indicator:percent_small_buckets_created_last_24h:description = "This indicator tracks the percentage of small buckets created over the last 24 hours. 
small bucket is defined as less than 10 % of the ‘maxDataSize’ setting in indexes.conf."
indicator:percent_small_buckets_created_last_24h:red = 50
indicator:percent_small_buckets_created_last_24h:yellow = 30

 

Also from the above same health warning message is shown when the number of hot buckets created to index the data in particular hour/day crosses or reaches the threshold limit defined in system/health.conf.
 

The reason behind the creation of too many hot buckets
 

Splunk uses buckets as an index directory to index the data. Now, when an event comes to Splunk for indexing, the new hot bucket will be created for that event or the event is indexed to the existing one of the hot buckets as per the event's timestamp and constraints on the index.

Refer to maxHotBuckets, maxHotSpanSecs and quarantinePastSecs parameters of indexes.conf
 

So, if the event is having the older/strange timestamp than acceptable for the existing buckets, then Splunk will create a new hot bucket to index that event. Hence, the number of hot buckets created may reach or cross the threshold.
 

When DATETIME_CONFIG parameter of props.conf is not set explicitly, Splunk will try to find the timestamp in an event itself.


So, while extracting the timestamp from the events, if Splunk finds the value that matches the valid timestamp format, it would parse the same value as event time and would index it using that timestamp only, which may cause Splunk to create new hot bucket as the timestamp is not properly extracted.

Consider adding timestamp attributes in props.conf to make Splunk extract timestamp properly. Configure timestamp recognition
 

TIme Stamp Issue:

Also, I have found the below messages in the splunkd.log regarding the timestamp also fix them by applying the correct settings depends on the log format.
 

06-02-2019 22:37:36.976 -0700 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (30) characters of event. 
Defaulting to timestamp of previous event (Mon Jun 3 02:37:35 2019). Context: source=/data/syslogng/asa/richmond-vpn-inside/20190603.log|host=richmond-vpn-inside
|cisco:asa|48952806-21-2019 09:48:48.467 -0700 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (44) characters of event. 
Defaulting to timestamp of previous event (Fri Jun 21 06:48:45 2019). Context: source=/data/syslogng/pan/vcadcl-fw-mgmt/20190621.log|host=vcadcl-fw-mgmt|pan_log|86736606-20-2019 11:29:10.742 -0700 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. 
Defaulting to timestamp of previous event (Thu Jun 20 11:29:10 2019). Context: source=/scratch/data/syslogng/vlsj-securelog/sj-anliu1/20190620.log|host=sj-anliu1
|linux_secure|526909106-14-2019 12:46:38.301 -0700 WARN DateParserVerbose - Accepted time format has changed ((?i)\s([01]\d)([0123]\d\d)\s), possibly indicating a problem 
in extracting timestamps. Context: source=C:\Windows\debug\netlogon.log|host=DC2SJGLOBAL|MSAD:NT6:Netlogon|4001606-02-2019 16:17:18.106 -0700 WARN DateParserVerbose - Accepted time format has changed ((?i)(?<!\w|\d[:\.\-])(?i)(?<![\d\w])(jan|\x{3127}\x{6708}|feb|\x{4E8C}\x{6708}
|mar|\x{4E09}\x{6708}|apr|\x{56DB}\x{6708}|may|\x{4E94}\x{6708}|jun|\x{516D}\x{6708}|jul|\x{4E03}\x{6708}|aug|\x{516B}\x{6708}|sep|\x{4E5D}\x{6708}|oct|\x{5341}\x{6708}
|nov|\x{5341}\x{3127}\x{6708}|dec|\x{5341}\x{4E8C}\x{6708})[a-z,\.;]*([/\- ]) {0,2}(?i)(0?[1-9]|[12]\d|3[01])(?!:) {0,2}(?:\d\d:\d\d:\d\d(?:[\.\,]\d+)? {0,2}(?i)((?:(?:
UT|UTC|GMT(?![+-])|CET|CEST|CETDST|MET|MEST|METDST|MEZ|MESZ|EET|EEST|EETDST|WET|WEST|WETDST|MSK|MSD|IST|JST|KST|HKT|AST|ADT|EST|EDT|CST|CDT|MST|MDT|PST|PDT|CAST|CADT|EAST
|EADT|WAST|WADT|Z)|(?:GMT)?[+-]\d\d?:?(?:\d\d)?)(?!\w))?)?((?:\2|,) {0,2}(?i)(20\d\d|19\d\d|[901]\d(?!\d)))?(?!/|\w|\.\d)), possibly indicating a problem 
in extracting timestamps. Context: source=C:\Windows\System32\DHCP\DhcpSrvLog-Mon.log|host=srv07-seoul|DhcpSrvLog|286986

 

Resolution

1. Navigate to $SPLUNK_HOME/etc/system/local/health.conf and append the below stanza
 

[feature:buckets]
indicator:percent_small_buckets_created_last_24h:red = 150

2. Restart the Splunk for changes to take effect.

If it a timestamp or DATETIME_CONFIG issue, write a proper regex to identify the date and time from the event correctly and rectify the issue.

Related Links

Splunk Documentation
Configure timestamp recognition 


Splunk Answers
Health Status : The percentage of small of buckets created 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release