ITSI Notable events grouped mostly during backfill instead of realtime since Indexers maintenance

Applicability

Splunk IT Service Intelligence (ITSI) versions 4.* till 418 and 4.19, Premises with remote indexers, or on SplunkCloud. After an indexer maintenance / auto-scaling event

Description

Following an indexer maintenance, or adding new indexers, swapping indexers, or auto-scaling indexers (a SplunkCloud specific feature)

All or a consistent subset of the notables events in ITSI stop being grouped immediately in realtime, and all or as subset of them are consistently grouped with a delay by the historical backfill safeguard mechanism. The delay is usually up to 12 minutes as the backfill runs periodically. 

 

Investigation

To confirm the behavior : 

Check ITSI troubleshooting dashboards

Go to ITSI dashboard menu > Event Analytics Monitoring > "skipped events" tab > "Backfilled Events Percentage" panel

• Check if the backfill ratio stays consistently high since the maintenance
• On the other panels, you can also check the number of skipped events over time.
• you can drilldown to the populating search and adjust the span (the default span is 1hour chunks)

 Check the location of the notables events, on the indexers

Use this search to see the repartition of the notables grouping, count per indexer and type of grouping.

(index=itsi_tracked_alerts event_id!=null )
| stats first(_time) AS _time count by splunk_server index event_id
| join event_id type=outer [ search index="_internal" source="*itsi_rules_engine*" EventId reMode=RealTime IsBackfilledEvent=*
| dedup EventId | eval event_id=EventId | table event_id IsBackfilledEvent
| eval is_backfill_grouping=if(IsBackfilledEvent=="true",1,0)
| eval is_realtime_grouping=if(IsBackfilledEvent=="false",1,0) ]
|bucket _time span=1h
| chart sum(is_backfill_grouping) sum(is_realtime_grouping) by _time splunk_server

• Run with a timerange overlapping before&after the issue started
• it will show you the notables based on their origin indexers (splunk_server names as indexers)
• and what mechanism did the rules engine used to do the grouping :  type of grouping (is_backfill_grouping versus is_realtime_grouping)
• Interpretation : 
  • if you see that new indexers appeared and are only grouping in backfill
  • if you see that existing indexers were doing realtime and are not doing backfill only
  • if you can see that the number of indexers changed, and new ones were added

Check the runtime of the "itsi_event_grouping" search job.

• go to activity > jobs, look for "itsi_event_grouping", and check the job creation time
• confirm that it has been running for a long time, and was not restarted since the maintenance

Resolution

Workaround : 

The solution is to restart the scheduled realtime search "itsi_event_grouping".

1. Go to Settings > searches & reports.
2. Search for the search "itsi_event_grouping" in the context of the app "SA-ITOA", and usually owned by the user "admin".
3.  Disable it, then re-enable it.

A new search job will be spawn under a minute, and will dispatch the realtime search to all the indexers currently connected.

Double check the result with the investigation steps above in the next 10-20 minutes.

Long term solution :

ITSI 4.20.* and later has a new feature "Rules Engine Queue Mode" that does not rely on the Realtime Search anymore.

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release