UF stops sending data: TcpOutputProc - The TCP output processor has paused the data flow

Applicability

Splunk Cloud Splunk Enterprise Splunk Universal Forwarder

Description

Universal Forwarder suddenly stop sending data.
Restart fixes the issue, but it repeats again after few days.

Investigation

The errors from the splunkd.log in the UF:

07-27-2022 08:53:06.997 +1000 ERROR TailReader [5964 tailreader0] - Ignoring path="D:\Batch\Output\controlm_prod_alerts.out" due to: Bug: tried to check/configure STData processing but have no pending metadata.07-27-2022 08:53:06.997 +1000 ERROR TailReader [5964 tailreader0] - failed to compute crc for D:\Batch\Output\controlm_prod_alerts.out (method: 0, hint: The system cannot find the file specified.).07-27-2022 08:53:06.982 +1000 WARN WatchedFile [5964 tailreader0] - encountered error computing crc, hint: [seekptr=502,start_read=246,read_sz=256] (The system cannot find the file specified.)07-27-2022 09:34:36.227 +1000 INFO  AutoLoadBalancedConnectionStrategy[15560 TcpOutEloop] - Connection to x.x.x.x:9996 closed. context=read sock_error=10054. ssl_error=error:00000000:lib(0):func(0):reason(0)
07-26-2022 11:02:02.314 +1000 WARN  loader [4472 MainThread] - Failed to open file="C:\ProgramData\Splunk Enterprise\SplunkWeb.URL" error="The system cannot find the path specified.""TcpOutputProc - The TCP output processor has paused the data flow..." or "TcpOutputProc - Applying quarantine) in splunkd..."

When the UF stops , check the queue blockage using the below query:

index=_internal sourcetype=splunkd source=*metrics.log group=queue blocked=true

Also check the queue blockage under /opt/splunk/var/log/splunk/metrics.log which will display as shown below: 

07-26-2022 19:51:03.455 +0000 INFO Metrics - group=queue, name=typingqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=891, largest_size=894, smallest_size=0

 Increase the queue size in server.conf if a queue blockage is observed:
 

[queue]maxSize = [<integer>|<integer>[KB|MB|GB]]
* Specifies default capacity of a queue.
* If specified as a lone integer (for example, maxSize=1000), maxSize
  indicates the maximum number of events allowed in the queue.
* If specified as an integer followed by KB, MB, or GB (for example,
  maxSize=100MB), it indicates the maximum RAM allocated for queue.
* Default: 500KB

queue maxSize | server.conf
 

• Validate if indexes are correctly configured to have an inputs.conf for splunktcp port. 
•  Validate if the SSL certificate is readable and valid/expired on the forwarder's output.conf using the below command:

openssl x509 -noout  -enddate 

certificate is valid as shown below:

notafter=May 17 23:59:59 2023 GMT

 

 

 

Resolution

There is a wait parameter (time_before_close) in inputs.conf.
This parameter is more to say Splunk to wait until the other systems finish writing the file.

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3

 In Inputs.conf, increase the value of time_before_close=60

"time_before_close=60" 

time_before_close | inputs.conf

 

Related Links

How to troubleshoot why a universal forwarder lost data

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release