ES to SOAR Pairing Breaks After Upgrading to Enterprise Security 8.6.1 on Splunk 10.4.x

Applicability

Splunk Enterprise Security 8.6.1 Splunk Enterprise 10.4.x, including 10.4.0 and 10.4.2 Splunk SOAR 8.5.0, 8.5.0.248, or 8.6.0 On-Premises deployments, including standalone Search Head and Search Head Cluster

Description

After upgrading Enterprise Security to version 8.6.1 on Splunk Enterprise version 10.4.x, the ES to SOAR integration can show one or more of the following symptoms:
Pairing completes successfully, but the integration stops working within minutes or within 24 hours.
The ES Configure page displays the following error.

SOAR returned an error: No Splunk credentials were available in SOAR

The SOAR wsgi.log file reports an expired token.

enterprise_security_token_factory.py:59 WARNING: Stored EnterpriseSecurityToken is invalid: Signature has expired
enterprise_security_user_manager.py:80 WARNING: Failed to fetch central Splunk credentials: Signature has expired

Initial pairing fails while creating the ES SOAR integration user.

Internal server error occurred during pairing process: REST Exception occurred while creating ES SOAR integration User: [HTTP 403] Client is not authorized to perform requested action; https://XXX.XXX.XXX.XXX:8089/services/authentication/users

Unpairing from the ES interface fails with HTTP 403.

Exception in initiate_unpairing: [HTTP 403] Client is not authorized to perform
requested action; https://XXX.XXX.XXX.XXX:8089/services/authentication/users/es_soar_integration_user

Re-pairing temporarily restores the integration, but the integration stops working during the next token refresh cycle.

Cause

The create_ingest_soar_tokens modular input runs every two hours to refresh the credentials used by Splunk SOAR. On affected deployments, the input remains enabled but fails with HTTP 403 authorization errors.

The following message can also appear in mc_es_soar_pairing.log. 

Role=es_soar_integration is not grantable by user splunk-system-user

This indicates that a platform level role grantability restriction may prevent splunk-system-user from assigning or refreshing the es_soar_integration role.

The issue has been observed with the combination of Enterprise Security 8.6.1 and Splunk Enterprise 10.4.x. Internal testing with Enterprise Security 8.6.1 on Splunk Enterprise 10.2.x did not reproduce the issue.

Investigation

1. Review the token creation log

On the ES Search Head, review the following file.

$SPLUNK_HOME/var/log/splunk/mc_create_ingest_soar_tokens.log

Search for HTTP 403 errors similar to the following.

[HTTP 403] Client is not authorized to perform requested action;
https://XXX.XXX.XXX.XXX:8089/services/authorization/tokens?output_mode=json

The log can also show that the modular input detected a stale execution time. 

Checking mod input last run time: <timestamp>, time since last run: <seconds>, refresh_time: 7200
No successful run, or the last successful modinput run was more than 2 hours ago
Running the Modinput for SOAR Tokens ingestion


2. Review the pairing log

Review the following file.
$SPLUNK_HOME/var/log/splunk/mc_es_soar_pairing.log

Search for the following message.

Role=es_soar_integration is not grantable by user splunk-system-user

 



3. Verify the modular input status

Run the following search on the ES Search Head.

| rest /servicesNS/nobody/missioncontrol/data/inputs/mc_create_ingest_soar_tokens
splunk_server=local
| table title, disabled, eai:acl.app

Confirm that the input is enabled.

This issue differs from the separate scenario in which the modular inputs become disabled. In this scenario, the input remains enabled but fails during token creation or refresh.

4. Review checkpoint information

Review the following file.

$SPLUNK_HOME/etc/apps/missioncontrol/local/mc_checkpoints.conf

Under the following stanza, compare the last_run timestamp with the current time.

[create_ingest_soar_tokens://main]

A significant gap can indicate that the modular input has not completed successfully.

5. Confirm the Splunk Enterprise version

Run the following search on the ES Search Head.

| rest /services/server/info splunk_server=local
| table splunk_server, version, build

Confirm whether the deployment is running Splunk Enterprise 10.4.x.

Resolution

This issue is being investigated under BLUERIDGE-25170. A permanent resolution has not been confirmed.

Temporary Workaround

Re-pairing can temporarily restore the ES to SOAR integration. The integration can stop working again during the next token refresh cycle.

On the ES Search Head, complete the following steps. 

1. Navigate to Configure, Splunk SOAR.
2. If the pairing page displays an error, remove the following configuration file.
$SPLUNK_HOME/etc/apps/missioncontrol/local/essoar.conf
3. Restart the Splunk service on the ES Search Head.
4. Navigate to Configure, Splunk SOAR.
5. Start a new pairing operation using an appropriate local administrator account and the soar_local_admin credentials.
6. Monitor mc_create_ingest_soar_tokens.log.
7. Confirm that token ingestion completes successfully after pairing.

This workaround may need to be repeated each time the pairing breaks.

Modular Input Refresh, Partial Mitigation

Cycling the modular input can force a new token ingestion attempt. This procedure does not resolve the underlying HTTP 403 authorization failure, but it can help confirm whether the input is able to run.

1. Navigate to Settings, Data Inputs on the ES Search Head.
2. Locate the input named Create and Ingest Splunk tokens required by SOAR as credentials.
3. Disable the input.
4. Edit the input and temporarily set the refresh interval to 60 seconds.
5. Save the change and enable the input.
6. Monitor mc_create_ingest_soar_tokens.log for a successful run or an HTTP 403 error. 
7. If the input runs successfully, restore the refresh interval to 7200 seconds.

If the HTTP 403 error continues, re pairing remains the available temporary workaround.

Permanent Fix

A permanent fix has not been confirmed.

Validation

The issue is likely related to this known problem when all of the following conditions are present.

1. Enterprise Security version 8.6.1 is installed.
2. Splunk Enterprise version 10.4.x is installed. 
3. The ES to SOAR integration was paired successfully, or pairing is being attempted.
4. The create_ingest_soar_tokens modular input is enabled.
5. mc_create_ingest_soar_tokens.log contains HTTP 403 errors.
6. mc_es_soar_pairing.log contains the following message.

Role=es_soar_integration is not grantable by user splunk-system-user

7. The integration stops working during or shortly after the token refresh cycle.

Related Links

Troubleshoot pairing Splunk Enterprise Security with Splunk SOAR

Revision History

Revision	Publish Date	Comments
1.0	01-Oct-2026	Initial Release