ERROR: "Alert action sendmn not found"

Applicability

Description

Search Head is showing the following errors every time a search is executed.

sendmodalert - Invoking modular alert action=sendmn for search " app="search" type="saved"ERROR sendmodalert - Error in 'sendalert' command: Alert action "sendmn" not found.ERROR SearchScheduler - Error in 'sendalert' command: Alert action "sendmn" not found., 
search='sendalert sendmn

Investigation

1. Identify the Saved Search: Use splunk btool savedsearches list --debug to find which saved searches are configured with action.sendmn = 1. This will help pinpoint the specific alert(s) causing the error.
2. Check alert_actions.conf for "sendmn" Definition:
   • Run splunk btool alert_actions list --debug to inspect the merged alert_actions.conf configuration. Look for a stanza named [sendmn].
   • If [sendmn] is missing, it confirms that the alert action's definition is not available to Splunk.
   • If it exists, verify that is_custom = 1 is set and that the path to the script (if it's a custom script) is correct and accessible.
3. Verify File Locations and Permissions:
   • Custom alert actions are typically defined within an app's directory structure, for example, $SPLUNK_HOME/etc/apps/<app_name>/default/alert_actions.conf or $SPLUNK_HOME/etc/apps/<app_name>/local/alert_actions.conf.
   • Ensure that the alert_actions.conf file containing the [sendmn] stanza is in the correct app directory and that Splunk has the necessary read permissions.
   • For custom scripts associated with the alert action, ensure they are in the bin directory of the app (e.g., $SPLUNK_HOME/etc/apps/<app_name>/bin/) and have execute permissions.
4. Review Splunk Logs: Check splunkd.log for additional errors related to sendmodalert or sendalert that might provide more context, such as issues with script execution or file access.

Use btool to troubleshoot configurations

Resolution

1. Remove or Correct action.sendmn = 1 from Saved Searches:
   • If "sendmn" is an unintended or deprecated alert action, the simplest solution is to edit the savedsearches.conf file(s) and remove or comment out the action.sendmn = 1 entry for the affected alerts.
   • Alternatively, if "sendmn" was a placeholder or typo for another existing alert action, correct the entry in savedsearches.conf to reference the correct action name.
2. Define the Missing Alert Action in alert_actions.conf:
   • If "sendmn" is a legitimate custom alert action that is simply missing its definition, you need to create or correct its stanza in the appropriate alert_actions.conf file.
   • Place the alert_actions.conf file in the local directory of the app where the alert action should reside (e.g., $SPLUNK_HOME/etc/apps/<app_name>/local/alert_actions.conf).
   • A basic custom alert action stanza would look like this:
     ini

      

     [sendmn]
     is_custom = 1
     label = Send MN Notification
     description = Sends a notification to the MN system.
     command = <path_to_script>/sendmn_script.py
     

   • Ensure the command attribute points to the actual script that the alert action should execute.
   • After modifying configuration files, a Splunk restart might be required for changes to take effect, especially for new alert actions.

Related Links

Alertactions.conf