ALERT: "DMC Alert - Abnormal State of Indexer Processor for index=XXX"

Applicability

Description

DMC Alert - Abnormal State of Indexer Processor for index=XXX, with the following message:throttled - The index processor has paused data flow. Too many tsidx files in idx=XXX 
bucket="XXX", waiting for the splunk-optimize indexing helper to catch up merging them. 
Ensure reasonable disk space is available, and that I/O write throughput is not compromised.

Investigation

Checking the splunkd.log you find multiple log messages for "The index processor has paused data flow. Too many tsidx files" for example using the search:

Index=_internal sourcetype=splunkd "Too many tsidx files"

Returns several log messages like below:

04-22-2024 15:36:20.478 +0800 INFO  IndexWriter [3445260 indexerPipe_0] - The index processor has paused data flow. Too many tsidx files in idx=<index name> bucket="/opt/splunk/var/lib/splunk/<index name>/db/hot_v1_1254" , waiting for the splunk-optimize indexing helper to catch up merging them. Ensure reasonable disk space is available, and that I/O write throughput is not compromised.

These can be an indication that Splunk needs more child processes to complete the work properly.
 

Resolution

By increasing the maxRunningProcessGroups attribute and reducing the processTrackerServiceInterval in the [default] stanza of $SPLUNK_HOME/etc/system/local/indexes.conf it will allow more splunk-optimize processes concurrently and run them as soon as possible
 

[default]
maxRunningProcessGroups= XX
processTrackerServiceInterval = 0

The default value of maxRunningProcessGroups is 8, so please make sure to increase the value from there.

Increasing the maxRunningProcessGroups can increase CPU usage

According to documentation:

maxRunningProcessGroups = <positive integer>
* splunkd runs helper child processes like "splunk-optimize",
  "recover-metadata", etc. This setting limits how many child processes
  can run at any given time.
* This maximum applies to all of splunkd, not per index. If you have N
  indexes, there will be at most 'maxRunningProcessGroups' child processes,
  not N * 'maxRunningProcessGroups' processes.
* Must maintain maxRunningProcessGroupsLowPriority < maxRunningProcessGroups
* This is an advanced setting; do NOT set unless instructed by Splunk
  Support.
* Highest legal value is 4294967295.
* Default: 8

Related Links