Splunk Add-on for Unix and Linux : linux cpu_metric.sh event is not properly structured.

Applicability

Description

For the cpu_metric.sh inout, the metric events are unstructured, and are causing error messages to pop-up upon splunk indexer restart.

Investigation

While doing troubleshooting using below SPL:

index=_internal "*ERROR*" "*metric*"

Found below error message:

Search peer xxxx.xxxx.xxx.xx.xxx has the following message: The metric event is not properly structured, source=interfaces, sourcetype=interfaces_metric, host=xxxx.xxxx.xxx.xx.xxx, index=metrics_linux_perf. Metric event data without a metric name and properly formated numerical values are invalid and cannot be indexed. Ensure the input metric data is not malformed, have one or more keys of the form "metric_name:<metric>" (e.g..."metric_name:cpu.idle") with corresponding floating point values.

> The warning message that is currently seen suggests that there is a mismatch in the metric values for a few of the events that are being sent from the universal forwarder up to the indexer

>  whenever the indexer performs a rolling restart it checks that if the data which is being sent from the forwarder is in the desired format or not

> Upon checking, there was a mismatch from the source, and the indexer, in the version of the Splunk Add-on for Unix and Linux.

> previous instances where the mismatch in the version of the add-on has caused multiple issues with extractions and this can be correlated with the regex  for the add-on

Resolution

Maintaining consistent add-on versions across the entire Splunk environment fixed the issue.