Workload rules not functioning due to invalid runtime predicate in workload management configuration

Applicability

Description

Several workload rules were configured to be applied in chronological order. However, none of the configured workload rules appeared to be executing.

The following warning was found in the logs, indicating a workload management configuration validation error:

09-23-2026 01:12:29.357 +0000 WARN WorkloadConfig [1825549 MainThread] - Failed to process invalid rule=all_time with error=invalid value of predicate 'runtime'. Valid input is a number with a time unit. Valid time units include 's', 'second', 'seconds', 'm', 'minute', 'minutes', 'h', 'hour', 'hours'.

The all_time rule used the predicate:

search_time_range=alltime AND runtime>0

The runtime value lacks a time unit, causing the rule to be rejected during configuration processing. This invalid rule prevented the workload configuration from being processed correctly, which also affected the execution of other workload rules.

Investigation

1.Review the internal logs for workload management warnings or errors related to rule processing:

index=_internal sourcetype=splunkd log_level IN (WARN, ERROR) component=WorkloadConfig

2. Confirm whether the logs contain a warning similar to:

Failed to process invalid rule=all_time with error=invalid value of predicate 'runtime'.

3. Review the workload rules configured to be evaluated in chronological order.

In this case, the first rule uses the following predicate and action:

runtime > 15m AND search_mode=realtime

Action: Abort search

This predicate is valid. However, the rule was not executed as expected.

The second rule uses the following predicate:

search_time_range=alltime AND runtime > 0

Note: The search_time_range=alltime value is valid. However, the runtime value must include a valid time unit. Because runtime > 0 does not specify a unit, this rule is rejected during configuration processing.

4. Verify workload rule predicates for correct syntax, ensuring runtime values include valid time units (e.g., 1s, 15m). See Specify predicate values as reference.

5. Verify that all workload rules are enabled and correctly configured on the impacted search heads.

6. Confirm whether the workload rules trigger their expected actions, such as aborting real-time searches after the configured runtime threshold:

index=internal host=<impacted_search_head) <name_of_the_rule>

Resolution

1. Identify workload rules with invalid runtime predicates missing time units.
2. Update the predicates to include valid runtime values with time units. For example, change:

search_time_range=alltime AND runtime>0

to

search_time_range=alltime AND runtime>1s

3. Save and apply the updated workload rule configuration.
4. Verify that the workload management configuration validation error is cleared.
6.Confirm that the workload rules trigger as expected, including rules configured to abort real-time searches after their specified runtime.
7. Monitor logs to ensure no further workload management errors occur and that workload rules execute as expected.

Related Links

Configure workload rules