ERROR HttpInputDataHandler - Parsing error : Server is busy. ERROR HttpInputDataHandler - Failed processing http input, reply=9

Applicability

Description

Many 503 errors presented in $SPLUNK_HOME/var/log/splunk/splunkd.log:

ERROR HttpInputDataHandler - Failed processing http input, token name=XXXX, channel=n/a, source_IP=XX.XXX.XX.XX, reply=9,"

Investigation

1) To check the trend of errors, Please run the below search on the search head :

index=_internal source=*splunkd.log* log_level=ERROR OR log_level=DEBUG component=HttpInputDataHandler reply=9
|timechart counts

2) Noted a trend of  "server is busy" errors on a specific day, where a large amount of data was sent via HEC when compared with other days.

Resolution

Note: Fine-tuning the queue size in server.conf and the [input_channels] stanza in limits.conf is no longer valid for Splunk Cloud 9.2.2406.x and above. The queue size is now elastic, and manual adjustments are no longer required.

Some tuning may be in order for HTTP Event Collector performance:

$SPLUNK_HOME/etc/system/local/server.conf 

[general] 
parallelIngestionPipelines = 2 # Note: Never set more than 3 on indexing tier for 8.x and older releases. 9.x onwards any number of parallelIngestionPipelines can be set on indexing tier. However on Search Heads/Heavy Forwarders it can be set up to number of available CPU cores.# To avoid 503 response back, have enough queue buffer available for spike in ingestion. Be conservative in increasing sizes which should be tested and increased in small increments.[queue=indexQueue]
maxSize = 100MB[queue=aggQueue]
maxSize = 100MB#Must for HEC
[queue=parsingQueue]
maxSize = 100MB#Must for HEC
[queue=httpInputQ]
maxSize = 100MB[queue=rulesetQueue]
maxSize = 100MB[queue=typingQueue]
maxSize = 100MB

Usually, this setting in $SPLUNK_HOME/etc/system/local/limits.conf will be suggested if the data ingestion is slow on indexers.

[input_channels]
max_inactive = 10000 
#( ideally 2 times max(new_channels))[input_channels] 
lowater_inactive = 9000 
#( max_inactive -1000 recommended if max_inactive > 10000)

If you have enough CPU resources (32 cores) on your indexers/forwarders, you can increase dedicated threads for HEC inputs in
$SPLUNK_HOME/etc/system/local/inputs.conf

[http] 
dedicatedIoThreads = 8busyKeepAliveIdleTimeout = 300 
#(useful when HEC clients are using connection pools and want to keep connections idle. Set it 2 times expected idle time of connection.sslServerHandshakeTimeout = 300 
#(useful when HEC clients are using connection pools and want to keep connections idle. Set it 2 times expected idle time of connection. Supported from 8.2.0)

* If useACK is disabled on indexers, $SPLUNK_HOME/etc/system/local/indexes.conf:

[<PER_INDEX>] 
maxTimeUnreplicatedNoAcks=60 
#( needed if useACK=false and indexer cluster environment)

Reference: About HTTP Event Collector Indexer Acknowledgment

Related Links

Low-ingestion thruput without queues blocked