iowait avg_cpu__max_perc or single_cpu__max_perc or sum_top3_cpu_percs

Applicability

Description

Alerts about iowait can be seen in the Graphical User Interface (GUI).

These alerts come from a scheduled splunk search which checks resource_usage.log for this data.

This data is ingested from hosts which produce statistics from /proc/stat file.

Investigation

How do I know if Splunk's introspection log is correct?

•  this can be calculated manually from the /proc/stat file. This file has an entry for wall clock time spent in the iowait state. It also lists the time for all the other states. Dividing the iowait time by the summation of all time will give the iowait percentage.

Example:

• comparison of iostat output:

Linux 3.10.0-862.11.6.el7.x86_64 (testbox) 23/01/22 x86_64 (4 CPU)avg-cpu: %user %nice %system %iowait %steal %idle
          1.30  0.00   0.33    0.07   0.00   98.29

•  /proc/stat output:

# cat /proc/stat
cpu 251151436 336174 61608598 18919711142 14044048 0 2748422 0 0 0
cpu0 90195868 140743 15463282 4702418189 2323189 0 1799976 0 0 0
cpu1 53899850 83314 15210032 4739627760 4469582 0 283279 0 0 0
cpu2 54238314 77434 15595352 4737529225 3419583 0 376088 0 0 0
cpu3 52817404 34683 15339930 4740135966 3831692 0 289078 0 0 0
(...)

• To get the %iowait, divide the value in (row 1 column 5) with the sum of the values in (row 1)

%iowait = 14044048 / (251151436 + 336174 + 61608598 + 18919711142 + 14044048 + 0 + 2748422 + 0 + 0 + 0) * 100 %
%iowait = 0.07 %

In Splunk the health.conf will alert if:

• Every 10 seconds, this feature takes 3 measurements from the instrospection log:

avg_cpu__max_perc_last_3m:
index=_introspection source=/resource_usage.log component=IOWait "data.avg_cpu_pct"=""| timechart max("data.avg_cpu_pct") by host span=1msingle_cpu__max_perc_last_3m:
index=_introspection source=resource_usage.log component=IOWait "data.max_cpus_pct"="*"| streamstats count as rank | eventstats sum("data.max_cpus_pct") as sum_max_cpus by rank | timechart max(sum_max_cpus) by host span=1msum_top3_cpu_percs__max_last_3m:
index=_introspection source=resource_usage.log component=IOWait "data.max_cpus_pct"="*" | timechart max("data.max_cpus_pct") span=3m by host

• The introspection log itself gets the statistics from /proc/stat file (which is also used by built-in Linux tools such as iostat and vmstat). If the splunkd process cannot read this file, this feature will not work.

• If at any time during the last 3 minutes, there has been a condition met by the thresholds, then the alert will fire.

• The 10seconds or 3 minutes settings are not currently configurable but may be in the future.

• The default thresholds are defined in $Splunk_HOME/etc/system/default/health.conf

[feature:iowait]
display_name = IOWaitindicator:avg_cpu__max_perc_last_3m:description = This indicator tracks the average IOWait percentage across all CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 1% and Red if it exceeds 3% during this window.
indicator:avg_cpu__max_perc_last_3m:red = 3
indicator:avg_cpu__max_perc_last_3m:yellow = 1indicator:single_cpu__max_perc_last_3m:description = This indicator tracks the IOWait percentage for the single most bottle-necked CPU on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 5% and Red if it exceeds 10% during this window.
indicator:single_cpu__max_perc_last_3m:red = 10
indicator:single_cpu__max_perc_last_3m:yellow = 5indicator:sum_top3_cpu_percs__max_last_3m:description = This indicator tracks the sum of IOWait percentage for the three most bottle-necked CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the sum exceeds 7% and Red if it exceeds 15% during this window.
indicator:sum_top3_cpu_percs__max_last_3m:red = 15
indicator:sum_top3_cpu_percs__max_last_3m:yellow = 7

Resolution

There are four solutions:
- understand and tackle the issue with the in-house Linux Administrators team
- increase the alert values for this feature via health.conf [feature:iowait], so it won't alert with lower indicators
- ignore the alert in messages in the GUI
- disable the alert itself in health.conf [feature:iowait]

Related Links

linuxhowtos

redhat

proc