App ITSI - Some Notable Events Are not Grouped into Episodes because of Indexer Realtime Performance

Applicability

Description

In the ITSI Incident review page, if you notice some Notable events not being grouped in to Episodes, o​​​​​​r if in the ITSI healthcheck page, you see some events reported missed in the "skipped events" panel.

The grouping is being done by an indextime realtime scheduled search "itsi_event_grouping" in the app SA-ITOA, and a java process "itsi rules engine". The realtime search is asking the indexers to return all the "notable events" that were just indexed to be processed by the java process.

The reason why some events may be missed is that the indexers load or performance cause the notable events to be indexed slowly, in particular the time when the events go from the indexing queue to be written down to the disk in the buckets. There is a default cache offset of 60 seconds.

If is possible that slow indexers, or indexers busy replicating buckets, or with a slow storage with a i/o contention may sometimes take more than the default offset to commit the events to disk, and not send a copy to the search-head.

PS : Eventually, when the rules_engine restarts, it does a backfill of missed events in the last 24h.

Investigation

Use the ITSI dashboard "ITSI Healthcheck"them under the "skipped events" panel
- See if you have some events missing (not necessarily 100% of events missing); you can also disregard the last minute's events, as they may still be being processed.  
- Or use a search like this one to find events that are not a member of any groups.
 

(index=itsi_tracked_alerts event_id=* event_id!=null  )  OR ( index=itsi_grouped_alerts event_id=* event_id!=null)
| stats first(_time) AS _time dc(itsi_group_id) AS member_of_groups by aggpolicy event_id
| eval member_of_groups=if(member_of_groups=0,"zero",member_of_groups)
| timechart span=10m count by member_of_groups

Resolution

The default waiting time for indexed time real time is 60 seconds, you want to start by doubling it to 120 seconds.

To modify it

- in limits.conf under indexed_realtime_disk_sync_delay (for all searches)
- but the best is to overwrite it for the search itsi_event_grouping only

• from the UI search manager under the "advanced editor"
  • find the setting dispatch.indexedRealtimeOffset
  • add a value to it "120"
• then disable/enable the search to apply

[itsi_event_grouping]
dispatch.indexedRealtimeOffset=120
#no value will default to indexed_realtime_disk_sync_delay
#pick a value higher than 60 sec, start to double it.

Then you have to wait a bit and reuse the ITSI healthcheck dashboard to see if this helped :

- If there are no missed events, you could stop there, or you could progressively reduce the dispatch.indexedRealtimeOffset by making sure you do not go to low
- If they are still missed events, try a higher offset, but beware it will end up slowing down the reaction time to your notables.