403 Forbidden when saving a dashboard in Observability Cloud, enabling SSO permission fallback

Applicability

Description

Users may encounter intermittent "403 Forbidden" errors when attempting to perform platform actions, in this case attempting to save dashboards, when the connection to the SSO providing Splunk Cloud platform is temporarily unstable or unreachable. This occurs because the application cannot successfully fetch the necessary user roles and metadata to validate the user's permissions.

Investigation

Investigation of the issue revealed that a connection timeout between the Observability platform and the Splunk Cloud domain triggers an SplunkRBACException. When the configuration flag for fallback functionality is not present, the system is unable to retrieve cached role data to compensate for the connection failure. Consequently, the directory access validator fails to verify the user role which results in a 403 access denial. 

Resolution

To ensure business continuity during temporary connectivity issues with the Splunk platform, administrators should enable the "Login and Permission Fallback" feature:

1. Access Settings: Navigate to the General Settings within the Observability Cloud platform.
2. Configure Fallback: Proceed with the documented steps for setting up login and permission fallback.
   1. In Splunk Observability Cloud, go to Data Management > Available Integrations > Login Services
   2. Add an SSO integration. For more information, see Single Sign On.
   3. Go to Settings > General Settings > User Management.
      1. • Use the same identity source as the paired Splunk Cloud Platform stack, and assign the application to the users who have the o11y_access role in Splunk Cloud Platform. This keeps access synchronous between the primary and secondary login methods.
         • The email address in the identity provider (IdP) assertion must match the email address the user verified during their first Unified Identity login. The email address is how Splunk Observability Cloud identifies the user as the same account.
         • Give the integration a display name that identifies Unified Identity as the preferred login method. You can also clear Show on login page while fallback is not in use.
   4. Select a role from the default SSO role drop-down list. For more information, see Set up a default SSO role.
3. Contact Support if all steps followed and there is no change and 403 persists.

Related Links

https://help.splunk.com/en/splunk-observability-cloud/administer/splunk-platform-users/centralized-user-and-role-management/login-and-permission-fallback