Episode present in itsi_grouped_alerts but does not exist in itsi_notable_group_system_lookup

Applicability

Description

The observed issue involves episodes in the itsi_grouped_alerts collection but missing in itsi_notable_group_system_lookup.

Investigation

The missing episodes were linked to a specific correlation search. 

Additionally, the below error logs revealed that the issue is with the Rules Engine:

[rulesengine] [pool-17-thread-1] PolicyExecutor:1574 - aiId=hO7B, PolicyId=a4b2c296-c289-11ed-acdf-005056bb4684, PolicyTitle=XXX Episodes by Alarm, FunctionName=handlePolicyGroupUpdatesKVStoreSyncScheduleMessage, msgId=MudoB9, Status=Failed, ErrorMessage="HTTP 400 -- Request exceeds API limits - see limits.conf for details. (Too many documents for a single batch save, max_documents_per_batch_save=1000)"

From the _audit log It was noted that the max_documents_per_batch_save parameter was reduced to 1000 in limits.conf, impacting rules engine functionality.

Resolution

Update Configuration:

• Increase the max_documents_per_batch_save parameter in limits.conf to an appropriate value to handle batch save operations.
• Restart Splunk services to apply changes.

Example:

[kvstore]
max_documents_per_batch_save = 2000

Verify Rules Engine Functionality:

• Confirm the rules engine operates without errors by monitoring logs.

Check Episode Availability:

• Search for missing episodes in itsi_grouped_alerts and verify they populate in the itsi_notable_group_system_lookup collection.

Restart ITSI Event Grouping Search:

• Navigate to Activity > Jobs, filter to All, locate itsi_event_grouping, and stop the job.
• Wait a few minutes before restarting to allow the rules engine to stabilize.

Monitor Performance:

• Use the following query to ensure the itsi_event_grouping search is running on all indexers:
  
  

index=_introspection sourcetype=splunk_resource_usage "data.search_props.label"=itsi_event_grouping | timechart dc(host)