Splunk Alert: DMC Alert - Search Peer Not Responding (deployment server)

Applicability

Description

If you are receiving alert emails from Splunk stating that your DS is not responding and the mail body shows that your DS is down.

Also, it states that;

The alert condition for 'DMC Alert - Search Peer Not Responding' was triggered.Alert: DMC Alert - Search Peer Not Responding.Instance                           Status
<hostname of your DS>               DOWN

Investigation

You can check the status of the DS by logging into the CLI and running the below command;

./splunk status

If it shows that the status is down, well you found the reason behind the issue and you can simply start the services by running the below command;

./splunk start

OR

./splunk restart

If the status shows as running with the list of PIDs involved, but still you are receiving the alert then you can try restarting the DS again by using the restart command provided above.

If still, this doesn't help then you can check the logs and the peer status by running the below search query on your DS;

| rest splunk_server=local /services/search/distributed/peers/
| where status!="Up"
| fields peerName, status
| rename peerName as Instance, status as Status

The search query of alert "[DMC Alert - Search Peer Not Responding]" hits the rest point on "splunk_server=local" where "local" refers to the search head.

The search query should show results that indicate that the DMC is timing out when the search peer is not available to reach and hence throwing the inconsistent results to the search query.

Resolution

If restarting doesn't help and the rest api search results indicate that the peers and DMC is not able to communicate effectively then you can make the below changes on your DMC and indexers to fix the issue.

On the DMC please add the below parameters under; $SPLUNK_HOME/etc/system/local/distsearch.conf

[distributedSearch]
statusTimeout = 120
connectionTimeout = 120
serverTimeout = 120
sendTimeout = 120
receiveTimeout= 120
authTokenConnectionTimeout = 120
authTokenSendTimeout = 120
authTokenReceiveTimeout = 120

Followed by restarting the DMC for the changes to take effect.

If still, you receive the alert mails then, you need to make the below changes on your indexers as well.

On indexers, please add the below parameters under; $SPLUNK_HOME/etc/system/local/distsearch.conf

[replicationSettings]
connectionTimeout = 120
sendRcvTimeout = 120

Followed by restarting the Indexers for the changes to take effect.

This should stop the false positive alerts that were generating and spamming your mails.

Related Links