The Splunk Infrastructure Monitoring Add-on (SIM) is fetching metrics inconsistently or some metrics are missing.

Applicability

Description

The Splunk Infrastructure Monitoring Add-on was configured, and a query was written to fetch all the metrics from SignalFX.

Investigation

SPL:

index="_internal" sourcetype=splunkd "*sim*"

Error:

2024-09-13 03:02:47,975, Level=ERROR, Pid=2927042, RequestId=8dccdf8d221e40878ba6b9c3c248c714, Logger=SIMModularInput, File=sim_modular_input.py, Line=456, status=error, error_code=limit_reached, error_msg=SignalFlow Program reached metadata message limit of {"_logical_timestamp_ms": 1726196400000, "_message": {"blockContexts": [{"column": 161, "line": 1}], "blockSerialNumbers": [3, 9, 10, 11], "contents": {"limitSize": 30000, "matchedSize": 30000}, "messageCode": "FIND_LIMITED_RESULT_SET", "messageLevel": "WARNING", "numInputTimeSeries": 0, "timestampMs": 1726196400000}}., action=stream_sim_flow_program_results, instance_name=Splunk Kubernetes - GE Prod, org_id=Ej734ivA0AA, thread_id=b937886a31a342958a25ceff1c1314a3, signalflow_program="data('container_cpu_utilization', filter=filter('k8s.pod.name', '*'), rollup='rate').promote('plugin-instance', allow_missing=True).publish('DSIM:Kubernetes'); data('container.memory.usage', filter=filter('k8s.pod.name', '*')).promote('plugin-instance', allow_missing=True).publish('DSIM:Kubernetes'); data('k8s.container.memory_limit', filter=filter('k8s.pod.name', '*')).promote('plugin-instance', allow_missing=True).publish('DSIM:Kubernetes'); data('pod_network_receive_errors_total', filter=filter('k8s.pod.name', '*'), rollup='rate').publish('DSIM:Kubernetes'); data('pod_network_transmit_errors_total', filter=filter('k8s.pod.name', '*'), rollup='rate').publish('DSIM:Kubernetes');", computation_id=GXUoe1RAyvA, message_type=InfoMessage, message={"_logical_timestamp_ms": 1726196400000, "_message": {"blockContexts": [{"column": 161, "line": 1}], "blockSerialNumbers": [3, 9, 10, 11], "contents": {"limitSize": 30000, "matchedSize": 30000}, "messageCode": "FIND_LIMITED_RESULT_SET", "messageLevel": "WARNING", "numInputTimeSeries": 0, "timestampMs": 1726196400000}}, metadata_msg_count=498965, data_msg_count=227, mts_count=3463909, other_msg_count=125081, retry_count=0

Based on the error "sim flow search reached metadata message limit of 30,000," it seems that data points are being lost due to the exceeded limit. The default limit is set to 30,000. The working theory is that each event in the search results is a reconstruction of metadata and data points. Therefore, if all the metadata is not returned, some events in the MTS may not appear, potentially resulting in missing entire events, not just metadata.

Resolution

The limit of 30,000 MTS per data block is a built-in limitation of the Splunk Infrastructure Monitoring (SignalFx) platform, and it cannot be modified as this is by design.

As as work around ,divide the datablocks into multiple SignalFlow programs as per the nature of the data (metrics) being collected.

For example:

"data('CPUUtilization', filter=filter('stat', 'mean') and filter('namespace', 'AWS/EC2') and filter('InstanceId', '*'), rollup='average').publish(); data('NetworkIn', filter=filter('stat', 'sum') and filter('namespace', 'AWS/EC2') and filter('InstanceId', '*'), rollup='sum').publish(); data('NetworkOut', filter=filter('stat', 'sum') and filter('namespace', 'AWS/EC2') and filter('InstanceId', '*'), rollup='sum').publish();" 

An individual data block is equivalent to a single SignalFlow expression. A data block has the following structure:

data('CPUUtilization', filter=filter('stat', 'mean') and filter('namespace', 'AWS/EC2') and filter('InstanceId', '*'), rollup='average').publish();

Related Links

Splunk Infrastructure Monitoring Add-on

System limits for Splunk Infrastructure Monitoring