Dashboard Studio Email Export Fails

Applicability

Description

Dashboard Studio dashboards are unable to successfully send test emails using the “Schedule export → Send test email” workflow. The system returns the following error:

In contrast, classic Splunk dashboards and direct SPL | sendemail commands configured with the same SMTP settings are able to send emails successfully. This behavior suggests a discrepancy in how Dashboard Studio's test email feature processes email configurations compared to other Splunk email functionalities. The error specifically points to a missing or invalid username, often defaulting to 'splunk'. An additional error related to allowedDomainList may also be observed: "The email domains of the recipients are not among those on the allowed domain list." 

Investigation

Verify Global Email Settings: Navigate to Settings > Server settings > Email settings in the Splunk UI. Confirm the following:

Email sender: A full email address (e.g., noreply@domain.com) is configured.
Mail host/Port/TLS: Correct SMTP server and port (e.g., email-smtp.us-east-1.amazonaws.com:587) and TLS settings are applied.

Use SMTP AUTH: Enabled, with a valid username (e.g., noreply@domain.com) and password/method.
Allowed domain list: Ensure the recipient's domain (e.g., trccompanies.com) is present.
Confirm Effective Configuration on Disk: On the Search Head responsible for sending emails, execute the following command from the $SPLUNK_HOME/bin directory:

./splunk btool alert_actions list email --debug

Examine the output for the [email] stanza to confirm from, mailserver, use_ssl, use_tls, auth_username, and allowedDomainList values match the intended configuration. This helps identify any conflicting configuration files overriding settings.

Test Email Sending with SPL: Run a simple search on the Search Head to isolate the mail configuration:

| makeresults
| eval to="recipient@domain.com"
| sendemail to=$to$ subject="Splunk test" message="Testing email from Splunk." sendresults=false

If this command fails with a 554 error, the issue is likely with the core SMTP configuration. If it succeeds, but Dashboard Studio still fails, proceed to the next step.

Check Runtime Email Settings: To view the runtime values Splunk is currently using for email, execute:\

| rest /services/server/settings/email
| table mailserver use_tls use_ssl from auth_username allowedDomainList

Verify that mailserver, use_tls, use_ssl, from, auth_username, and allowedDomainList match the successful SPL test and global settings. A mismatch here indicates a configuration override.

Review Logs: Examine splunkd.log, python.log, and scheduler.log around the time of a failed Dashboard Studio test email attempt for sendemail entries or 554 errors. Look for messages indicating "User name is missing: 'splunk'."

Allowed Domain List Impact: If ERROR sendemail:657 - The email domains of the recipients are not among those on the allowed domain list. is observed, temporarily comment out or remove the allowedDomainList setting in $SPLUNK_HOME/etc/system/local/alert_actions.conf, restart Splunk, and re-test. Re-adding the correct allowedDomainList and restarting has resolved scheduled exports in some cases.

Resolution

The issue can be resolved by directly modifying the sendemail.py script to ensure the correct 'from' email address is used, preventing a fallback to the default 'splunk' username.

1. Navigate to the Script: Access the file located at $SPLUNK_HOME/etc/apps/search/bin/sendemail.py.

2. Locate the sender Definition: Within the script, find the following lines of code:

   python

   # takes header, html, text
   def buildHeaders(argvals, ssContent, email, sid, serverInfoContent):sender = ssContent.get("action.email.from", "splunk")
   

3. Modify the sender Value: Change the sender line to explicitly include the desired 'from' email address. Replace splunk@<domain>.com with the appropriate 'from' email address for the Splunk setup.

   # takes header, html, text
   def buildHeaders(argvals, ssContent, email, sid, serverInfoContent):sender = ssContent.get("action.email.from", "splunk@<domain>.com")
   

   Note: Replace the sender to the cx sender email and before making any changes, create a backup of the sendemail.py file.

4. Restart Splunk: After modifying the script, restart the Splunk server for the changes to take effect.

   $SPLUNK_HOME/bin/splunk restart
   

   Following these steps, the issue with Dashboard Studio's email export functionality should be resolved.

PD: If the "Send test email" function still does not work, it is recommended to save the dashboard and await the scheduled email to confirm successful delivery.