"Send to SOAR" Alert Action Failures After Splunk Upgrades

Applicability

Description

After upgrading Splunk (e.g., to 9.4.1, 10.2+), alerts configured with the "Send to SOAR" action fail to forward events. The splunkd.log typically reports ERROR sendmodalert with exit code 1 and various ImportError or ModuleNotFoundError messages.

Investigation

Search the SPL below in the Search and Reporting App:

index=_internal *sendtophantom* sourcetype=splunkd

Common error patterns include:

ERROR SearchScheduler [43394 AlertNotifierWorker-0] - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert sendtophantom results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__nobody__search__RMD5bd837a3953028025_at_1744921500_54235/per_result_alert/tmp_0.csv.gz" results_link=":443/app/search/search?q=%7Cloadjob%20scheduler__nobody__search__RMD5bd837a3953028025_at_1744921500_54235%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now" param.action_name=sendtophantom | stats count'

ERROR sendmodalert [43394 AlertNotifierWorker-0] - action=sendtophantom STDERR - ImportError: cannot import name 'splitattr' from 'urllib.request' (/opt/splunk/lib/python3.9/urllib/request.py).

ModuleNotFoundError: No module named 'imp' (Python 3.13 transition)

Upon investigation, it was discovered that the directories aob_py2 and aob_py in the $SPLUNK_HOME/etc/apps/phantom/bin/ta_addonphantom were remnants from a previous version of the Phantom app. These directories were attempting to use an outdated Python version, causing compatibility issues with Splunk's new Python 3.9 version.

Root Cause
These errors are caused by orphaned legacy library artifacts persisting in the app directory after an upgrade. Splunk app upgrades perform file-level overwrites rather than directory-level wipes; consequently, obsolete directories from previous versions (such as aob_py, aob_py2, or future/) remain on the file system.

When the Splunk platform is upgraded to a newer Python runtime (e.g., 3.9 or 3.13), the app's app.conf or internal logic may attempt to load these legacy dependencies. Because these legacy files are often hardcoded to use modules that have been deprecated or removed (like imp), the alert action script crashes during initialization.

Resolution

To resolve these compatibility issues, the orphaned legacy directories must be manually removed to force the app to use the current, platform-supported Python libraries.

The resolution involved removing the outdated directories from the Phantom app:

1. Remove the aob_py2 and aob_py directories from $SPLUNK_HOME/etc/apps/phantom/bin/ta_addonphantom.

2. Restart the Splunk service.

This ensured that the app is now fully compatible with Python 3.9, resolving the issues with "Send to SOAR" alerts not being processed.

Locate and remove legacy directories within the Phantom app path. Depending on your version and upgrade history, these may include:

• • $SPLUNK_HOME/etc/apps/phantom/bin/ta_addonphantom/aob_py/
  • $SPLUNK_HOME/etc/apps/phantom/bin/ta_addonphantom/aob_py2/
  • $SPLUNK_HOME/etc/apps/phantom/bin/ta_addonphantom/aob_py3/future/

 (Note: It is recommended to move these to a backup location, such as /tmp/, rather than immediate deletion for safety.)