In DMC, WARN TimeInvertedIndex - bucket=XXXX Already running 1 splunk-optimize's, max=1

Applicability

Description

In DMC , the below alert will be triggered when the indexer does not have more processes for the internal index due to many tsidx files and the index processor will pause data flow.

throttled - The index processor has paused data flow. Too many tsidx files in idx=_internal bucket="XXXXXXX" , waiting for the splunk-optimize indexing helper to catch up merging them. Ensure reasonable disk space is available, and that I/O write throughput is not compromised.

Investigation

Upon investigating, the below error message was noticed in indexer logs:

WARN TimeInvertedIndex [31997 indexerPipe] - bucket=XXXX Already running 1 splunk-optimize's, max=1
WARN TimeInvertedIndex [31997 indexerPipe] - bucket=XXXX Already running 1 splunk-optimize's, max=1
WARN TimeInvertedIndex [31997 indexerPipe] - bucket=XXXX Already running 1 splunk-optimize's, max=1

This error indicates that Splunk has tried to optimize the hot bucket but getting limited due to too many tsidx files. The Index throttling more often (throttleCheckPeriod=5), with only 1 Splunk-optimize running over a bucket (maxConcurrentOptimizes=1), spawning several child processes over it (maxRunningProcessGroups=32) and checking every second if any other child process can be launched (processTrackerServiceInterval=0).

This error normally occurs when Splunk indexes a lot of information in a short time . It fixes itself temporarily in about 30 to 60 minutes.

Resolution

Follow the below steps to fix the error permanently:

i. First, run the below command in indexers to optimize the bucket.

$SPLUNK_HOME/bin/splunk-optimize -d|--directory <bucket_directory_from_alert>

ii. If the errors still persist , then make below changes in the $SPLUNKHOME/etc/system/local/indexes.conf, in the _internal index stanza of the indexers.

[_internal]
maxRunningProcessGroups=32
processTrackerServiceInterval=0
throttleCheckPeriod=5
maxConcurrentOptimizes = 12
maxMemMB = 25

In the case of clustered environment , push the configuration changes from the cluster master ,apply it to all the indexers, and proceed with the rolling restart.

Related Links

Indexesconf