Processing queues are full: "The monitor input cannot produce data because splunkd's processing queues are full"

Applicability

Description

Spikes in processing queues were observed on Heavy Forwarder or Universal Forwarder instances and the following messages were occasionally logged in the splunkd log:

The monitor input cannot produce data because splunkd's processing queues are full. This will 
be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.

Investigation

The above error was reported on a Forwarder which could not send data to the indexing layer. A search similar to the one below was used to observe the percent fill pattern of the outbound queue on the Forwarder:

index=_internal host=<host_name> (sourcetype=metrics OR sourcetype=splunkd) group=queue name=tcpout_*
| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size)
| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)
| eval fill_perc=round((curr/max)*100,2)
| timechart max(fill_perc) by name useother=false limit=15 
| fields _time, tcpout_*

Spikes in processing queues were observed where the percent fill reached 100%, this correlated with spikes in data thruput which could be reviewed using the following search:

index=_internal host=<host_name> sourcetype=metrics OR source=*/metrics.log* group=per_index_thruput
| timechart per_second(kb) useother=false limit=15

Where <host_name> was set to the hostname of the forwarder. Please note, for Universal Forwarders introspection logging must be enabled before metrics data used in the above searches is generated:

Configure_PIF

Resolution

Please note that this article is written with an appropriately spec'd environment in mind.

Generally, it is not recommended making changes to any queue size (maxQueueSize, parsingQueue, aggQueue etc.) on either the Forwarder or Indexing layer as increasing queue sizes can impact your environment negatively and hinder troubleshooting other root causes of blocked processing queues.

Increasing queue size attributes will only increase the amount of data a queue can hold; this means that the queues themselves may appear less full but this does not mean the data is being indexed any quicker. The queues may look less full but there might still be a delay to data being indexed, and depending on the situation the queues might fill up again.

In this instance, the environment was experiencing temporary bursts of very large data; the maxQueueSize attribute was updated from auto to a static number in the outputs.conf.  It is recommend engaging Splunk Professional Services or Splunk Support before making changes to this attribute:

maxQueueSize = [<integer>|<integer>[KB|MB|GB]|auto]
* The maximum size of the forwarder output queue.
* The size can be limited based on the number of entries, or on the total
  memory used by the items in the queue.
* If specified as a lone integer (for example, "maxQueueSize=100"),
  the 'maxQueueSize' setting indicates the maximum count of queued items.
* If specified as an integer followed by KB, MB, or GB
  (for example, maxQueueSize=100MB), the 'maxQueueSize' setting indicates
  the maximum random access memory (RAM) size of all the items in the queue.
* If set to "auto", this setting configures a value for the output queue
  depending on the value of the 'useACK' setting:
  * If 'useACK' is set to "false", the output queue uses 500KB.
  * If 'useACK' is set to "true", the output queue uses 7MB.
* If you enable indexer acknowledgment by configuring the 'useACK'
  setting to "true", the forwarder creates a wait queue where it temporarily
  stores data blocks while it waits for indexers to acknowledge the receipt
  of data it previously sent.
  * The forwarder sets the wait queue size to triple the value of what
    you set for 'maxQueueSize.'
  * For example, if you set "maxQueueSize=1024KB" and "useACK=true",
    then the output queue is 1024KB and the wait queue is 3072KB.
  * Although the wait queue and the output queue sizes are both controlled
    by this setting, they are separate.
  * The wait queue only exists if 'useACK' is set to "true".
* Limiting the queue sizes by quantity is historical. However,
  if you configure queues based on quantity, keep the following in mind:
  * Queued items can be events or blocks of data.
    * Non-parsing forwarders, such as universal forwarders, send
      blocks, which can be up to 64KB.
    * Parsing forwarders, such as heavy forwarders, send events, which
      are the size of the events. Some events are as small as
      a few hundred bytes. In unusual cases (data dependent), you might
      arrange to produce events that are multiple megabytes.
* Default: auto
  * if 'useACK' is set to "true" and this setting is set to "auto", then
    the output queue is 7MB and the wait queue is 21MB.

Large bursts of data can also be managed by persistent queues, as discussed here: Use persistent queues

Assuming the environment is properly spec'd, other common causes of blocked queues/pipelines are:

- Bad data load balancing from the forwarder layer to the Indexing layer. In this situation you may notice pipelines full on some indexers, whilst on other indexers are empty. If you have parallelIngestionPipelines greater than 1, you might even see a pipeline full and the other one underutilized on the same indexer host

- Network issues or bandwidth limitations between the Forwarding layer and the Indexing layer. In this case, you may see the pipelines on the Forwarding layer full but the queues on the Indexing layer empty. Please note, the throughput on Universal Forwarder is limited by default to 256 Kbps as per maxKbps attribute in the limits.conf:

Limits.conf

- Bad regex in props and transforms. In this case, you may see the "Typing" queue full but the queues upstream in the pipeline line (i.e. Indexing queue on an Indexer or the Output queue on a Heavy Forwarder) empty

- Slow disk IO on the Indexing peers. In this case, you may see the "Indexing" queue full and all queues downstream in the pipeline line (Parsing, Aggregation, Typing) also full

- Data replication issues between Indexing peers. In this case, you may see the "Indexing" queue full and all queues downstream in the pipeline line (Parsing, Aggregation, Typing) also full

There may be other causes of full queues; a good place to start troubleshooting would be to follow the below troubleshooting guide:
How to Troubleshoot Blocked Ingestion Pipeline Queues with Indexers and Forwarders

Related Links

Splunk Documentation:
Outputs.conf
Capacity planning

General queue troubleshooting slides