Troubleshooting email alert notifications

Applicability

Description

Investigation

Before sending an email notification, confirm that Email Settings have been configured as stated in Configure email notification documentation.

Make sure the email server is not marking the email as SPAM.

Try sending an email notification from within a search. An admin role must be assigned to send an email notification within a search to a mail server that requires SMTP authentication.  To send an email notification within a search to a mail server that does not require SMTP auth requires the list_settings capability., to allow users not belonging to any of these roles to send email notifications using the sendemail command in their search, they must have the list_settings capability.  Add the following to the search in order to send email notifications from within a search:

| sendemail to="my_email@gmail.com" format=raw sendresults=1 footer="Sent from Splunk." from="SplunkAlerts" subject="Splunk Alert" 
message="The following Splunk Alert has been fired:"

Resolution

To determine if the scheduled search is running and with a status of "success", run the following search:

index=_internal source=*scheduler.log savedsearch_name="my_saved_search_name" | stats count by status

Check python.log for similar errors like the following to determine if there was an issue while sending the email:

$SPLUNK_HOME/var/log/splunk/python.log (Linux)

%SPLUNK_HOME%\var\log\splunk\python.log (Windows)

ERROR    sendemail:137 - Sending email. subject="Splunk Alert", results_link="https://localhost:8000/app/search/", recipients="[u'my_email@gmail.com']", server="localhost"ERROR    sendemail:458 - [Errno 99] Cannot assign requested address while sending mail to: my_email@gmail.com

If the manual attempt to send out an email with the sendemail command is successful, the issue could be located on the SMTP server's side. Check if the load it is receiving is too much for the server. This can cause it to reject some connectivity attempts. There could also be a limit on the number of emails that can be received at the same time by the same sender.

If the one of the previous scenarios is what's happening, either make some adjustments to the SMTP server, or change the scheduling of the saved searches so that they do not send emails at the same time.