Search Head Cluster Members Can't Sent Internal Logs to Index Cluster

Applicability

Description

This was happening on two out of three Search Head Cluster Members. They were still able to search the indexers but the output from two of them was not received in the indexers 

These error messages were displayed:

01-30-2025 08:22:59.038 -0800 WARN TcpOutputProc [34430 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest= inside output group primary_indexers from host_src=xxxxxx has been blocked for blocked_seconds=516210 . This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.  

Investigation

It is possible to run the following commands in the captain to verify health of Search Head Cluster:

/opt/splunk/bin/splunk show shcluster-status --verbose
/opt/splunk/bin/splunk show kvstore-status --verbose

In this case it is important to confirm that all the Search Heads are up and that their kvstore are ready.

Also it could be checked in the Search Head Cluster captain under distributed system-->Search Head Clustering and make sure  all the search heads are listed

Additionally this command could be run in the Cluster Manager to review the help of the Indexer Cluster:

/opt/splunk/bin/splunk show cluster-status --verbose

It would also very important to review the configurations to determine the state of the listening ports.

Resolution

After reviewing the configuration of the indexers from the diag files It was found a non-ssl port was configured
on the indexers that some clients started using for indexer discovery but the clients were only configured for ssl. 
Removing that configuration from the indexers fixed the issue