Search Head Captain skipping 100% searches

Applicability

Description

The search Head Captain suddenly started to skip 100% of searches throwing errors about user and disk quota enforcement limit breached and also there were issues related to job delegation and artifact replication. 

Investigation

1. In the Search Head cluster of 4 Search Heads, only the Search Head captain was skipping the searches and as a result, the cluster started to skip all the searches and the skip search ratio went up to 99.7%.

2. On checking the Search Head Captain GUI-----> Search Activity, we could see that around 55k+ searches were getting skipped with the reason "The maximum number of concurrent historical scheduled searches on this cluster has been reached" and "The maximum number of concurrent historical scheduled searches on this cluster has been reached".

3. But on checking the Seach Head Captain logs, we could see logs related to user search limits and disk limits being enforced for some users. 

02-13-2024 15:52:25.774 +0000 WARN DispatchManager [98192 TcpChannelThread] - enforceQuotas: username=" xyz.sxyz@tesco.com (mailto:xyz.sxyz@tesco.com) ", search_id="_YXJyYW4uc3RvYmJzQHRlc2NvLmNvbQ_YXJyYW4uc3RvYmJzQHRlc2NvLmNvbQ_U0hDX3Rlc2NvX2FwcF9wYXltZW50X3NlcnZpY2U__search11_1707839545.35690_2AA58C5B-B081-4E9B-8A9E-722516EBAD97" - QUEUED reason="The maximum number of concurrent historical searches for this user based on their role quota has been reached.", concurrency_limit="5"02-13-2024 15:07:49.299 +0000 WARN DispatchManager [20114 DispatchReaper] - enforceQuotas: username=" xyz.sxyz1@tesco.com (mailto:xyz.sxyz1@tesco.com) ", search_id="_c2FyaXQubXVraGVyamVlQHRlc2NvLmNvbQ_c2FyaXQubXVraGVyamVlQHRlc2NvLmNvbQ_U0hDX3Rlc2NvX2FwcF9pbnN0b3JlLWN1c3RvbWVyLXBpY2tpbmc__search12_1707836735.32447_2AA58C5B-B081-4E9B-8A9E-722516EBAD97" - QUEUED reason="The maximum disk usage quota for this user has been reached.", concurrency_limit="2000"

4. As limits enforcement issues for a few users cannot cause 100% skip searches given that other users did not breach their limits. On checking the logs further, we could see that the Search Head Captain has job delegation and artifact replication issues with its slave members.

2-13-2024 15:52:23.503 +0000 ERROR HttpClientRequest [3074 SHPPushExecutorWorker-30] - HTTP client error=Read Timeout while accessing server=https://X.X.X.X:8089 for request=https://X.X.X.X:8089/servicesNS/prachi.saxena%40tesco.com/SHC_tesco_app_Retail_Demand_Forecast/shcluster/member/delegatejob/Splunk%20Alert%
20-%20BetterPromotions/sched_dispatch?output_mode=json.02-13-2024 15:42:42.537 +0000 ERROR SHCMasterArtifactHandler [82138 TcpChannelThread] - failed on handle async replicate request sid=scheduler_c2ViYXN0aWFuLmJhY2hvcmN6eWtAdGVzY28uY29t_U0hDX3Rlc2NvX2FwcF9jb2xsZWFndWUtdHJhbnNmb3JtYXRpb24__RMD5c5f1c35eeb5362b6_at_1707837660_16448_015EFD18-0D54-4725-9A1E-19A7F7E94FD0 err='srcPeer="euwprodsplkopsshazb1", srcGuid="015EFD18-0D54-4725-9A1E-19A7F7E94FD0" cannot be valid source for artifactId=scheduler_c2ViYXN0aWFuLmJhY2hvcmN6eWtAdGVzY28uY29t_U0hDX3Rlc2NvX2FwcF9jb2xsZWFndWUtdHJhbnNmb3JtYXRpb24__RMD5c5f1c35eeb5362b6_at_1707837660_16448_015EFD18-0D54-4725-9A1E-19A7F7E94FD0 targetPeer="euwprodsplkopsshaza2", targetGuid="17F078ED-6BD1-4E96-83C0-7C1B1B6BBD1C" reason="active replication count >= max_peer_rep_load"'02-13-2024 15:45:39.131 +0000 ERROR SHCRepJob [66210 SHPPushExecutorWorker-406] - failed to delegate job job=SHPDelegateSearchJob peer="euwprodsplkopsshaza2", guid="17F078ED-6BD1-4E96-83C0-7C1B1B6BBD1C" saved_search=system;; err= http_status_code=502

5. Given the job delegation and artifact replication were running into timeouts even though the cluster had no connectivity issue and also the network bandwidth was fine, tuned the timeouts of the cluster,

distsearch.conf[distributedSearch]
sendTimeout=120
receiveTimeout=1200limits.conf[search]
remote_timeline_connection_timeout=30
remote_timeline_send_timeout=30
remote_timeline_receive_timeout=30

Post making the above changes, the job delegation and replication failure logs were reduced by a lot and also the skip searches reduced y a bit.

Resolution

Configuring the below on the Search Head cluster helped to bring down the cluster skip search to 0.

$SPLUNK_HOME/system/local/server.conf
[shclustering]
allow_concurrent_dispatch_savedsearch = 1