The number of splunk optimize processes is at 100% of the maximum. As a result, the index processor has paused data flow

Applicability

Description

The following message prompted on the UI:

The number of splunk optimize processes is at 100% of the maximum. As a result, the index processor has paused data flow.

Investigation

The maximum concurrent optimizer was reached (1), so the alert was triggered immediately. 

According to Indexesconf the maximum number of concurrent optimizes are defined by the maxConcurrentOptimizes.

Also, the [default] stanza defines this value for the rest of the indexes. 

The documentation recommends increasing (but never reducing) this value in specific scenarios like the following: 
" * There are always many small tsidx files in the hot bucket.
  * After rolling, there are many tsidx files in warm or cold buckets."

Running the following btool command showed that the value of maxConcurrentOptimizes for the [default] stanza in indexes.conf was reduced to 

<splunk btool indexes list --debug |grep local>

Resolution

• Increase those values to the default (6) value or use a higher value if necessary (i.e. maxConcurrentOptimizes = 25).  Search peer has the following message idx internal Throttling
• Deploy and restart the servers.

Related Links

Search peer has the following message: idx=_internal Throttling indexer, too many tsidx files in bucket='dir", is splunk optimizer running? 
Indexes.conf