Unable to process IT Service Intelligence (ITSI) refresh queue because the queue is too large

Applicability

Description

If ITSI change stops to be processed and pile up. Changes like : modify a Share-Base-Search or KPI, delete en entity, change in a service or service template, change in thresholds, and many other propagation tasks. 

It may be because the itsi_refresh_queue collection filled up with too many tasks and hit memory limit when sorting it.
The ITSI consumer will fails to sort the tasks to prioritize them, and the queue will not be processed.
A possible solution is to accelerate the collection to pre-sort the content, and reduce the memory cost.

Investigation

Check the following items

Check the ITSI consumer log for errors:

index=_internal source=*itsi_consumer* ERROR log_level=ERROR

Look for errors like "Sort operation used more than the maximum":

File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 703, in simpleRequest
    raise splunk.InternalServerError(None, serverResponse.messages)
splunk.InternalServerError: [HTTP 500] Splunkd internal error; [{'type': 'ERROR', 'code': None, 'text': "An error occurred. (Internal read failed with error code '96' and message 'Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.')"}]

Check the itsi_refresh_queue collection object count

• go to the ITSI > Dashboard > ITSI health-check> look in the kvstore collections panel for the number of records
• or use a search like

  • index=_introspection source=*introspection*kvstore.log* data.ns="SA-ITOA.itsi_refresh_queue" | timechart first(data.count)  min(data.count)  max(data.count) by data.ns

• Or use a search like that one to review the content of the queue

  • |inputlookup itsi_refresh_queue| stats count by change_type changed_object_type | sort by -count | addcoltotals

If the queue number of objects is large (more than 100000 by example), or recently grew too big, this may be the root cause.

Resolution

For more more recent ITSI (4.20 etc..), check the SA-ITOA/default/collections.conf to see if this acceleration is already present.

On older ITSI versions (like 4.17, 4.18, 4.19 etc...) a possible solution is to add acceleration to the collection. This will presort the tasks, and reduce the memory cost of sorting them by the consumer.

1. on the ITSI search-heads , edit SA-ITOA/local/collections.conf
2. add to the collection those 2 accelerations

   • [itsi_refresh_queue] 
     accelerated_fields.high_priority_create_time_accel = {"priority": -1, "create_time": 1} 
     accelerated_fields.low_priority_create_time_accel = {"priority": 1, "create_time": 1}
     # added to solve memory limit issue with too large queues

3. restart Splunk to apply, give a few minutes to generate the acceleration.
4. Check if the errors are still occurring, then check if the tasks in the queue are processing 

Remark : in a large environment the task processing can still take time, please engage Splunk Support if needed to remove tasks from the queue.