Search Affinity not working as expected for scheduled searches

Applicability

Description

Search Head Cluster members were querying both configured sites on indexer clusters even with search affinity enabled.

Investigation

This issue can be visualized by going to the Monitoring Console and checking the instance that ships the searches as well as the instance that runs them.

1. Go to Settings > Monitoring Console
2. Click the Search drop-down and select Search Activity: Instance
3. Look for the scheduled searches and check the fields that refer who shipped the search and which indexer ran it

This can also be checked by running a search that correlates the Search Heads from site X and the Indexers from site Y, for example:

index=_internal <sh-name/IP from site X> AND <idx-name/IP from site Y> 

It was found that the SH from site X was having streaming issues with its own site indexers:

From site X indexer:

09-04-2024 15:03:40.184 +0000 WARN BucketReplicator [2659226 TcpOutEloop] - Failed to replicate Streaming bucket bid=xxxxxx-xxx~x~xxxxxxxx-xxx-xxxxxxxxx-xxxxxxxxxxx to guid=xxxxxxxx-xxxx-xxx-xxx-xxxxxxxxxxx host=xxx.xxx.xxx.xxx s2sport=xxxx. Read timed out after 180 secs. Setting: server.conf/[clustering]/rep_max_rcv_timeout09-04-2024 05:54:13.104 +0000 WARN BucketReplicator [2659226 TcpOutEloop] - Failed to replicate Streaming bucket bid=xxxxxx-xxx~x~xxxxxxxx-xxx-xxxxxxxxx-xxxxxxxxxxx to guid=xxxxxxxx-xxxx-xxx-xxx-xxxxxxxxxxx host=xxx.xxx.xxx.xxx s2sport=xxxx. Connection closed.09-04-2024 15:29:24.210 +0000 WARN BucketReplicator [2659226 TcpOutEloop] - Failed to replicate Streaming bucket bid=xxxxxx-xxx~x~xxxxxxxx-xxx-xxxxxxxxx-xxxxxxxxxxx to guid=xxxxxxxx-xxxx-xxx-xxx-xxxxxxxxxxx host=xxx.xxx.xxx.xxx s2sport=xxxx. Connection failed09-04-2024 15:30:09.816 +0000 WARN BucketReplicator [2659226 TcpOutEloop] - Replication connection to ip=host=xxx.xxx.xxx.xxx:xxxx timed out

Resolution

Increased the values of the following configurations from their default to 300, in the [clustering]stanza in server.conf in the affected indexers:

[clustering]
cxn_timeout = 300
rcv_timeout = 300
send_timeout = 300

Related Links

server.conf - Splunk Documentation