Corrupted Notable Event Aggregation Policy

Applicability

Description

Notable Event Aggregation Policies help us to group desired similar notable events into episodes which match the defined conditions.

These policies are stored in the kvstore in the notable_event_aggregation_policies_lister and can be managed using REST API via the event_management_interface

curl -k -u admin --request <operation> 'https://localhost:8089/servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy/xxxxxxxx'

If any of the entries have gone corrupt due to any reason such as bad format while using curl, it can cause errors and issues with episode generation, clog the refresh queue or even impact the entire rules engine operation.

Investigation

• check internal logs:
  

  index=_internal sourcetype=itsi_internal log_level=error

• following errors are observed:
  

  2025-07-18 06:53:48,794 ERROR [itsi_re(reId=wLtp,reMode=RealTime)] [pool-2-thread-2] PolicyManager:154 - FunctionName=UpdatePolicyQueue, Status=Failed, ErrorMessage="Failed to get policies from collection: itsi_notable_event_aggregation_policy"2025-07-18 06:53:48,794 ERROR [itsi_re(reId=wLtp,reMode=RealTime)] [pool-2-thread-2] KVStorePolicyFetcher:125 - FunctionName=getPolicies, Status=Failed, ErrorMessage="Cannot deserialize value of type `int` from Boolean value (token `JsonToken.VALUE_TRUE`)
  at [Source: (StringReader); line: 1, column: 41655] (through reference chain: java.util.ArrayList[13]->com.splunk.itsi.rule.engine.message.policy.Policy["disabled"])"

• Above errors indicate issue with the policy retrieval due to a serialization error, hence needs kvstore backup analysis to spot the faulty NEAP.
• In the following example the variable Disabled is incorrectly set to true instead of a integer or a boolean value or 0 or 1.
  

Resolution

• List the problematic policies:

curl -ku admin  https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_notable_event_aggregation_policy |jq '.[]|select(._key|contains("xxxxxxxxx","xxxxxxxx","xxxxxxx"))|._key,.title'

• Use the REST API command to remove the problematic policy as follows: 

curl -k -u admin --request DELETE 'https://localhost:8089/servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy/><policy id>'

• test by running the first command to confirm deletion.
• perform restart of the search head

Related Links

Notable Event Aggregation Policies

REST API