Incorrect Episode Grouping in EventIQ

Applicability

Description

When using Splunk IT Service Intelligence (ITSI) EventIQ to group episodes based on fields, for example key/node and company name, all alerts from different companies, alert keys, and nodes may be grouped into a single episode. This behavior occurs because EventIQ’s machine learning-based grouping algorithm uses an OR logic on ranked fields rather than enforcing an AND logic for exact matches. As a result, episodes do not differentiate as expected, impacting incident triage and resolution. No explicit error messages are generated, but the symptom is incorrect episode grouping.

Investigation

To determine if this issue is present in your environment, follow these steps:

1. From the ITSI main menu, navigate to Configuration > Event Management > Notable Event Aggregation Policies (NEAPs).

2. Select an existing NEAP that uses EventIQ or create a new one.

3. Review the Filtering Criteria and Instructions section to identify the fields used for grouping.

4. Verify if the EventIQ toggle is enabled for the policy.

5. Set the analysis window to a period that includes sufficient historical alert data.

6. Run the EventIQ analysis by clicking Run analysis to generate suggested grouping fields.

7. Review the Advanced settings to confirm which fields are included in the EventIQ ranking and similarity calculations.

8. Search for episodes in the episode review dashboard and check if alerts from different companies, alert keys, or the fields you want and check if they are grouped together despite the intended differentiation fields.

Resolution

The grouping behavior is by design in ITSI 4.21.0 and later, where EventIQ groups events matching on any ranked field (OR logic) rather than enforcing exact-match separation (AND logic). To address this:

1. Option A: Disable EventIQ for the affected NEAP(s)
   Use static, rule-based grouping with explicit filtering criteria per company or alert key. Create separate custom NEAPs for each company or alert key with specific filter conditions.

2. Option B: Use multiple custom NEAPs with pre-filtering
   Implement multiple NEAPs with filtering criteria to pre-separate events by Company name or other fields before EventIQ processes them within each policy.

3. Option C: Engage Professional Services
   Contact your Splunk Account Team to discuss involving Splunk Professional Services or On-Demand Services for a tailored architectural design and implementation assistance.

Steps to Disable EventIQ on a NEAP:

1. Navigate to Configuration > Event Management > Notable Event Aggregation Policies.

2. Select the NEAP to modify.

3. Toggle EventIQ off.

4. Define explicit filtering criteria using AND clauses to separate alerts by company, alert key, or node.

5. Save and activate the policy.

Steps to Create Multiple NEAPs for Pre-Separation:

1. Create separate NEAPs for each group of fields you want to filter them out.

2. In each NEAP, specify filtering criteria to include only alerts from the targeted fields.

3. Enable EventIQ on each NEAP to allow grouping within the filtered subset.

4. Activate all NEAPs.

Additional Notes:

• EventIQ’s current ML algorithm uses OR logic on ranked fields, which may cause grouping of unrelated alerts.

• No ITSI version currently guarantees exact-match (AND logic) grouping on multiple fields.