Action Rules are not executed for episodes as expected.

Applicability

Description

When one Episode matches a NEAP condition, the Action Rules action should execute ok.

However, the Action Rules don't execute for some episodes and no activity is recorded under the Activity menu. 

* RUN_ACTION_ASYNC is a feature introduced in Splunk IT Service Intelligence 4.17.2

Investigation

This behaviour is due to concurrent execution of the Action Rules, for the same episode. 

See ERROR which indicates a Java exception of a concurrent execution:

( ERROR is found in the splunk logs for the rules engine ie: $SPLUNK_HOME/var/log/splunk/itsi_rules_engine.log ) 

2024-06-10 11:48:51,409 ERROR [rulesengine] [ForkJoinPool.commonPool-worker-139] PolicyExecutor:814 - aiId=C0HM, PolicyId=40559bb6-e5d8-11ee-ba8d-658cc1b0136d, PolicyTitle=GMP : PROD, FunctionName=RunAction, Status=Failed, EventId=null, EventGroupId=null, GroupId=ac3be8a1-9608-4202-b45e-d0869c3e12f7, GroupStatus=5, ActionKey=notable_event_execute_action, ActionItemsCount=1
2024-06-10 11:48:51,409 ERROR [rulesengine] [ForkJoinPool.commonPool-worker-139] EventOperations:950 - Status=Failed, FunctionName=executeAction, ErrorMessage="(was java.util.ConcurrentModificationException) (through reference chain: java.util.HashMap["_group_data"]->java.util.HashMap["event"]->java.util.HashMap["rid"])", StackTrace=com.fasterxml.jackson.databind.JsonMappingException: (was java.util.ConcurrentModificationException) (through reference chain: java.util.HashMap["_group_data"]->java.util.HashMap["event"]->java.util.HashMap["rid"])
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392)
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
	at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:811)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4568)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3821)
	at com.splunk.itsi.event.management.sdk.SplunkdEventOperations.updateNotableEventWithResponse(SplunkdEventOperations.java:387)
	at com.splunk.itsi.event.management.sdk.SplunkdEventOperations.executeAction(SplunkdEventOperations.java:947)
	at com.splunk.itsi.event.management.sdk.SplunkdEventOperations.executeActionOnGroupEvents(SplunkdEventOperations.java:741)
	at com.splunk.itsi.rule.engine.core.utils.CommonUtils$EventActionUtils.batchNotableEventExecuteAction(CommonUtils.java:736)
	at com.splunk.itsi.rule.engine.core.PolicyExecutor.runActionLogic(PolicyExecutor.java:797)
	at com.splunk.itsi.rule.engine.core.PolicyExecutor.lambda$runAction$1(PolicyExecutor.java:768)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1728)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.ConcurrentModificationException
	at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1511)
	at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1544)
	at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1542)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:786)

Impacts experienced:

• failure to update the status/severity/comments for an episode
• sometimes, intermittently, email notifications

Resolution

Why is this happening:

• due to the async execution of Action Rules - RUN_ACTION_ASYNC feature introduced in ITSI v4.17.2to resolve this situation the async execution should be disabled

How to fix:

• RUN_ACTION_ASYNC can be disabled 
  • If itsi_rules_engine.properties doesn't exist in the local directory of SA_ITOA, create "$SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties"
    • If local/itsi_rules_engine.properties already exists follow to next step
  • Add the following line and save         

rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY, SORT_NOTABLE_EVENTS, RUN_ACTION_ASYNC

• • Restart the itsi_event_grouping real time search:
    •  GUI:  Activity → Jobs → Set filters to “All” → search label="itsi_event_grouping" and STOP the job ( itsi_event_grouping restarts after a couple of minutes)
  • Do not change anything in $SPLUNK_HOME/etc/apps/SA-ITOA/default

Outcome:

• Action Rules execute and the Activity menu shows the action from the matching NEAP