SNOW incident tickets are not being created by Actions Rules from existing ITSI Episodes

Applicability

Description

Action Rules are configured in the Notable Events Aggregation Policy (NEAP) so it can create a SNOW incident ticket.

The action rules execute successfully when it is run manually, but it doesn't when it is run by the Splunk user.

The notable event actions run all-time searches.

Investigation

from the splunkd _internal logs search is blocked, see as an example:

timestamp_1 process:3123428 thread:MainThread ERROR [itsi.notable_event_actions_queue_consumer] [notable_event_actions:1162] [execute_action] actionId="pvdw-432018", actionName="notable_event_action", Status="Failed", FunctionName="actionExecution", actionInternalName="snow_incident_ext", exception="All time searches are forbidden. Please use a specific time range to search."

timestamp_1 process:3123428 thread:MainThread ERROR [itsi.notable_event_actions_queue_consumer] [__init__:1433] [exception] All time searches are forbidden. Please use a specific time range to search.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/event_management/notable_event_actions.py", line 1155, in execute_action
    latestTime=latest_time
  File "/opt/splunk/lib/python3.7/site-packages/splunk/search/__init__.py", line 315, in dispatch
    raise splunk.SearchException(msg['text'])
splunk.SearchException: All time searches are forbidden. Please use a specific time range to search.

from $SPLUNK_HOME/etc/apps/search/local/workload_rules.conf it's known that an Workload Management admission rule is active and denies access to every user and app, unless explicit here, see example:

GUI:

ITSI_SH/<LANG>/manager/system/workload_management

Backend:

$SPLUNK_HOME/etc/apps/search/local/workload_rules.conf

[search_filter_rule:filter_alltime]
action = filter
predicate = search_time_range=alltime AND role!=admin AND role!=sc_admin
user_message = All time searches are forbidden. Please use a specific time range to search.

Resolution

Allow app=SA-ITOA and/or user=splunk-system-user into the Workload Management admission rule.

Example

GUI:

ITSI_SH/<LANG>/manager/system/workload_management

Backend:

$SPLUNK_HOME/etc/apps/search/local/workload_rules.conf

[search_filter_rule:filter_alltime]
action = filter
predicate = search_time_range=alltime AND app!=SA-ITOA AND user!=splunk-system-user
user_message = All time searches are forbidden. Please use a specific time range to search.

Related Links

CreateWLMRules