Health Report IOWait Showing Red

Applicability

Description

Investigation

The default threshold setting for IOWait is pre-set to a low value and may not be relevant to the specific environment/performance.  For new installations or upgrades, the IOWait threshold will need to be increased to accommodate the server environment.

Splunk's IOWait Calculations:

Every 10 seconds, this feature takes 3 measurements from the introspection log:

avg_cpu__max_perc_last_3m:

index=_introspection source=*resource_usage.log* component=IOStats data.cpu_pct=* | timechart max("data.cpu_pct") by host span=1m

single_cpu__max_perc_last_3m:

index=_introspection source=*resource_usage.log* component=IOStats data.cpu_pct=* | streamstats count as rank | eventstats sum("data.cpu_pct") as sum_max_cpu by rank | timechart max(sum_max_cpu) by host span=1m

sum_top3_cpu_percs__max_last_3m:

index=_introspection source=*resource_usage.log* component=IOStats data.cpu_pct=* | timechart max("data.cpu_pct") span=3m by host

Resolution

How to determine the correct IOWait threshold from the environment?
 

These searches benchmark iowait performance over time. Set the relevant thresholds for the environment. 

CPU IOwait average:

index=_internal source=*health.log* 
feature="IOWait" component=PeriodicHealthReporter node_type=indicator indicator=avg_cpu__max_perc_last_3m | table _time host measured_value due_to_threshold_value

How to Configure IOWait Thresholds:
 

• ​​​​​​Navigate to Settings > Health report manager > filter for: iowait.  Click Edit to see the configurable values.

Documentation: Healthconf

How to Disable IOWait Health Feature:

If there are other ways to monitor the IOWait or if the business doesn't mandate it, is possible to disable it.

To disable the feature: 
Create a $SPLUNK_HOME/etc/system/local/health.conf file and add the following lines:
 

[feature:iowait]
disabled = 1

To disable other features follow the instructions in the below link:

Disable A_feature