TCP output processor has paused the data flow due to "can't resolve a valid IP address for host"

Applicability

Splunk Enterprise

Description

Oftentimes, a forwarder sending data to an indexing tier that is clustered is configured to reach out to the Cluster Manager for its list of peers rather than the peers being hard-coded into the outputs.conf file itself.

When this occurs, if the cluster manager becomes unavailable or unreachable, the forwarder will send data to the last known receiver (as long as it is available) until either it is able to communicate with the cluster manager again or it reaches the connectionTimeout value in outputs.conf (by default, 20 seconds).

If the forwarder is restarted, it will attempt to send to the last known receiver until the timeout is reached again (see this community article for more details).

Investigation

1. Review logs on the impacted forwarder. Look for these messages (messages vary only by host_src (indexer) and blocked_seconds): 

   06-02-2023 15:52:44.203 -0400 WARN  TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=CM_HOSTNAME inside output group MY_OUTPUTGROUP from host_src=AN_INDEXER_HOSTNAME has been blocked for blocked_seconds=100. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.06-02-2023 14:06:37.612 -0400 ERROR TcpOutputProc - Processing server from outputs.conf: can't resolve a valid IP address for host="CM_HOSTNAME"

2. Example searches might look something like this, which will display a list of indexers (host_src) 

   index=_internal sourcetype=splunkd "TCP output processor has paused the data flow" | stats count by host_src AS Indexer
   

   and this, which should return just the Cluster Manager's hostname within the event message:

   index=_internal sourcetype=splunkd "Processing server from outputs.conf: can't resolve a valid IP address for" | stats count by event_message
   

3. Use ./splunk btool list outputs --debug to find the configuration a specific instance is using. By default, $SPLUNK_HOME/etc/system/local configs take precedence.
4. If the server defined in the defaultGroup is presented as a hostname, verify the hostname and port are correct. For example:

   etc/apps/my_outputs_app/local/outputs.conf [tcpout]
   etc/apps/my_outputs_app/local/outputs.conf defaultGroup = MY_OUTPUTGROUPetc/apps/my_outputs_app/local/outputs.conf [tcpout:MY_OUTPUTGROUP]
   etc/apps/my_outputs_app/local/outputs.conf clientCert = <PATH/TO/CERT.pem>
   etc/apps/my_outputs_app/local/outputs.conf disabled = 0
   etc/apps/my_outputs_app/local/outputs.conf server = CM_HOSTNAME:49152
   etc/apps/my_outputs_app/local/outputs.conf sslPassword = <HASHED PASSWORD>

Resolution

1. If the hostname and port are correct in outputs.conf, for verification purposes, change the hostname to instead reference the CM's IP address and restart Splunk on this forwarder.
2. If forwarding resumes and does not reach connectionTimeout, then this is a to a routing, DNS or firewall issue (not related to Splunk).
3. work with appropriate administrators (network, OS/system or security) to resolve the issue.
4. If a workaround is needed, update outputs.conf on all impacted forwarders (if this is feasible) to reference the CM's IP address.

NOTE: For maintenance purposes using the hostname is a best practice as this ensures continuous forwarding if a host's IP address changes, or if the host machine itself is replaced.

 

Related Links

See this community article for information regarding why a forwarder may resume sending data for a short period once it is restarted.

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release