Data gaps/discrepancies b/w Forwarders and indexers

Applicability

Splunk Enterprise Splunk Cloud

Description

Error Messages:

WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
INFO TailReader - ...continuing.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
INFO TailReader - ...continuing.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
INFO TailReader - ...continuing.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.
INFO TcpOutputProc - Connected to idx=x.x.x.x:9997, pset=0, reuse=0.

 

Investigation

Based on the diag investigation, we could see due to the data quality issues Splunk indexers fill indexing, typing, and other queues. The following LineBreakingProcessor, AggregatorMiningProcessor, and Time stamping issues were found on splunkd.logs

WARN LineBreakingProcessor - Truncating line because limit of 1000000 bytes has been exceeded with a line length >= 1003520 - data_source="lsof", data_host=" osbplsoadr66.amer.dell.com (http://osbplsoadr66.amer.dell.com/) ", data_sourcetype="lsof"

This message you are seeing means events are not seeing the line breaker you've specified even 1000000 bytes into the message. so you need to increase this value to something above 1003520.

TRUNCATE = <non-negative integer>
* Change the default maximum line length (in bytes).
* Although this is in bytes, line length is rounded down when this would
otherwise land mid-character for multi-byte characters.
* Set to 0 if you never want truncation (very long lines are, however, often a sign of
garbage data).
* Defaults to 10000 bytes.

WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded - data_source="I:\Program Files\Tableau\Tableau Server\data\tabsvc\logs\backgrounder\jprotocolserver_backgrounder_6-6.log", data_host="TABPWAPPPRD06", data_sourcetype="backgrounder"

When your event exceeds the default props.conf configuration MAX_EVENTS you can see the following warning messages in the splunkd.log file. This will cause the event to be cut short potentially losing useful information within the data. see props.conf MAX_EVENTS=256

MAX_EVENTS = <integer>
* Specifies the maximum number of input lines to add to any event.
* Splunk software breaks after the specified number of lines are read.
* Defaults to 256 (lines).

If your events are larger than the limit set in MAX_EVENTS, you can increase limits. In your case, a few of the event source type has the MAX_EVENTS as 256 i.e, default. You may need to adjust this value as per your log.

Please check the below link regarding the data quality issues.
https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Resolvedataqualityissues
 

Resolution

In order to resolve this issue, you may need to adjust your props.conf settings on indexers.
Also, a best practice is to fine-tune the below things on indexers.

Socket and Thread configuration

WARN HttpListener - Socket error from x.x.x.x:xxxx while accessing /services/streams/search: Broken pipe

https://docs.splunk.com/Documentation/Splunk/8.2.5/Troubleshooting/HTTPthreadlimitissues#Override_automatic_socket_and_thread_configuration

$SPLUNK_HOME\etc\system\local, create or edit server.conf or push this configuration changes through cluster master to all the indexers.

[httpServer]
maxThreads=150000
maxSockets=100000

Also, please make the below changes on indexers through CM.

Increase queue size as below:

server.conf add[queue]
maxSize = 500MB
[queue=AQ]
maxSize = 100MB
[queue=aggQueue]
maxSize = 100MB
[queue=parsingQueue]
maxSize = 100MB

Please do a rolling restart in order to take the effect.

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release