ITSI issue: ITSI-5623 Bulk Import

Applicability

Description

mvcombine and mvexpand are critical Splunk commands used in the entity_filter_rules macro (added in 4.4.0). In order to reduce the strain of editing the searches for KPIs, the storing the entity moved filters in a separate collection. The macros are used to construct the actual entity filter used by KPI searches.

The macros are as follows. They can be found in /opt/splunk/etc/apps/SA-ITOA/default/macros.conf

Macro to capture entity filter rule multivalue expansion and join used in both macros for adhoc kpis or shared base searches. 

Sets the values to the expected "host='123' OR host='abc' OR host='xyz'" format

If no results OR alias_value is empty (non matched entities), uses `no_entities_macro` as the value returned

[entity_filter_rule_mvevals]
definition = rename entity_info.alias_value AS alias_values | mvexpand entity_filtering_field | mvexpand alias_values\
| eval matched_alias_values=if(isnull(alias_values),"`no_entities_matched`", entity_filtering_field."="."\"".alias_values."\"")\
| mvcombine matched_alias_values | eval results=mvjoin(matched_alias_values," OR ") | table results\
| mvcombine results | eval value=mvjoin(results," OR ")\
| appendpipe [stats count | where count=0] | eval value=if(isnull(count),value,"`no_entities_matched`")

Given a KPI ID, perform an inputlookup against the itsi entity_rules filtering collection to construct the complete filter rule 

[lookup_kpi_entity_filter_rule(1)]
args = kpi_id
definition = | inputlookup itsi_entity_filter_rules_lookup where kpi_id="$kpi_id$" | `entity_filter_rule_mvevals`

Given a base search ID, perform an inputlookup against the itsi entity_rules filtering collection to construct the complete filter rule 

[lookup_shared_base_search_entity_filter_rule(1)]
args = search_id
definition = | inputlookup itsi_entity_filter_rules_lookup where base_search_id="$search_id$" | `entity_filter_rule_mvevals`

However, the mvcombine and mvexpand commands have memory constraints defined in limits.conf. As a result, depending on the "entity filter rule", the number of actual entities used to filter results can be significantly reduced than what it should be.

In below examples, 100 entities are returned vs actually 4000:

warn : command.mvcombine: output will be truncated at 100 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvcombine] / max_mem_usage_mb has been reached.warn : command.mvexpand: output will be truncated at 4400 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.

While it's unlikely that a single service will have more than a few hundred entities, KPI base searches share a single entity_filter_rule. (entities from all Services/KPIs that use the shared base search). As a result, it's possible a single entity_filter_rule object will have thousands of entities.

How to confirm whether the ITSI server is affected?

The warning message can be seen in the search.log of affected search jobs. For example:

/opt/splunk/var/run/splunk/dispatch/scheduler__nobody__itsi__RMD598fe407928707731_at_1644380820_12/search.log:

02-08-2022 23:27:01.634 WARN MultiValueProcessor [6150 phase_1] - sid:scheduler__nobody__itsi__RMD598fe407928707731_at_1644380820_12 command.mvexpand: output will be truncated at 1000 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.

The search.log is located in the search head inside /opt/splunk/var/run/splunk/dispatch/<search_id>

Search through all search.logs by running this command:

find /opt/splunk/var/run/splunk/dispatch -type f |grep search.log |xargs grep "excessive memory"

Resolution

Increase the memory limits for mvcombine and mvexpand commands. The default for both commands is set at 500, doubling the memory limit by making an update to the limits.conf.
 

limits.conf:[mvcombine]
max_mem_usage_mb = 1000[mvexpand]
max_mem_usage_mb = 1000

Where should the change be applied?
This change should be done on the ITSI search head(s). If they are clustered, please push the change from the deployer. Please restart splunkd service on the search head(s) after that.

No need to do the change on the indexers.

Related Links

ITSI 4.9.6 ReleaseNotes Known Issues Bulk Import