Heavy Forwarder Unable to Send Data to Indexer/Deployment Server.

Applicability

Splunk Enterprise

Description

Two Heavy Forwarders Unable to send Data to Indexer server

The issue is most likely in the Heavy forwarder side as the indexer works fine for other Heavy Forwarders.

 

Investigation

Used diag for troubleshooting, both Heavy Forwarders presented the following

Health Check

=========Splunk to Splunk forwarding connection=========
Description:
When a Forwarder is unable to send data to the indexers we typically see the following messages in splunkd.log: "TcpOutputProc - The TCP output processor has paused the 
data flow..." or "TcpOutputProc - Applying quarantine) in splunkd..." This symptom is usually the result of a port being blocked, the indexer being down, or queue 
congestion somewhere along the event processing pipeline.Message: Matching event detectedSuggested Action:
1. Validate if you correctly configured your indexes to have a inputs.conf for splunktcp port. 
2. Validate if using custom ssl if the ssl certificate is readable or has not yet expired on the forwarder outputs.conf. 
3. Validate if this issue isn't related to a Queue blockage on either indexer or forwarder side as pipeline blockage can derail the tcp output. If related to queue 
blockage look into configuring queues in server.conf, as well as validating data being forwarder. For a quick remedy you can restart your splunk instance, but this 
is not recommended as it may not resolve the underlying issue.

WARNINGS AND ERRORS:
 

WARN  TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 370 seconds. 
This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. 
It is probably not accepting data.WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='certificate expired'.ERROR X509Verify - X509 certificate (CN=splunk-deployment.osyture.org,OU=California Community College Technology Center,O=California Community Colleges 
Chancellor's Office,street=3536 Butte Campus Dr,L=Oroville,ST=CA,postalCode=95965,C=US) failed validation; error=10, reason="certificate has expired"

Resolution

Ran the command bellow which showed the certificate expired

openssl x509 -enddate -noout -in /opt/splunk/etc/apps/ssl_app/auth/splunk-forwarder.pem 

Then the client proceeded to get a new certificate for the Heavy Forwarders which fixed the issue.


 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release