SOAR: "Send to SOAR" alert actions are failing with HTTP 403s and "Could not verify connection to SOAR" errors

Applicability

Splunk SOAR Splunk SOAR Cloud Splunk App for SOAR Export

Description

We are observing that there are multiple scheduled searches on the SHC configured with "Send to SOAR" alert action (action name is reported as sendtophantom in logs) sending events to a SOAR instance and these events are no longer being received impacting monitoring operations tied to SOAR. 

Investigation

1) Run the search below against internal logs in the Search Head to narrow down errors:

index=_internal host=<SH_hostnames> ERROR sendmodaction

2) The following two log entries may consistently appear in relation to alert action failures:

2022-11-29 23:09:08,941 ERROR pid=45217 tid=MainThread file=cim_actions.py:message:431 | sendmodaction - worker="{SH_FQDN_REDACTED}" signature="Could not post failed data to KV store. Error [HTTP 403] Error talking to Splunk: POST /servicesNS/nobody/phantom/storage/collections/data/phantom_retry: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/servicesNS/nobody/phantom/storage/collections/data/phantom_retry" action_name="sendtophantom" search_name="{NAME_REDACTED}" sid="{SID_REDACTED}" rid="0" app="{NAME_REDACTED}" user="{NAME_REDACTED}" digest_mode="0" action_mode="saved" action_status="failure"

2022-11-29 23:09:08,893 ERROR pid=45217 tid=MainThread file=cim_actions.py:message:431 | sendmodaction - worker="{SH_FQDN_REDACTED}" signature="Could not verify connection to SOAR. Will be posting to KV Store to retry later" action_name="sendtophantom" search_name="{NAME_REDACTED}" sid="{SID_REDACTED}" rid="0" app="{NAME_REDACTED}" user="{NAME_REDACTED}" digest_mode="0" action_mode="saved" action_status="failure"

3) Then run the search below to see the errors above reported over time:

index=_internal host=<SH_hostnames> "Could not verify connection to SOAR" | timechart count by host

4) The search results above may reveal that these errors are not reported by all SHC members over time and only by a single or few SHC members.

5) Splunk SOAR Cloud instance has an explicit IP allowlist configured and IP address(es) of the SHC member(s) reporting those errors may have not been added to the allowlist, which will result in SHC member(s) not able to reach SOAR instance.

 

6) The SOAR user auth-token configured in the SASE app might not include all the IP addresses of the SHC members in its list of allowed IPs.

Resolution

• Add the IP(s) of the SHC member(s) reporting errors and verify all the SHC member IPs are correctly allowlisted on Splunk Cloud SOAR instance for events to come in.
• Splunk SOAR Cloud users will have to open a support case to allowlist IPs in SOAR if these symptoms are observed.
• If the IPs are missing from the SOAR user configuration, add them to the Allowed IPs field with the necessary IP address.

Revision History

Revision	Publish Date	Comments
1.0	30-Sep-2026	Initial Release