Splunk Infrastructure Monitoring Add-on: Data Ingestion Stopped in "sim_metrics" Index

Applicability

Description

Splunk Infrastructure Monitoring Add-on (SIM Add-on) lets a Splunk administrator collect metrics and event data from Splunk Infrastructure Monitoring.

After Configuring the Splunk Infrastructure Monitoring Add-on's Modular Input, data ingestion is not happening in the "sim_metrics" index due to a SignalFlow Program error being observed in the input's internal logs.

Investigation

As part of troubleshooting, internal logs can be checked using the following SPL query : 

index=_internal sourcetype=splunk_ta_sim Logger=SIMModularInput level=ERROR

The error message below can be observed in the results of the SPL query :

2025-02-11 21:49:07,283, Level=ERROR, Pid=749456, RequestId=149d4d59bd6c45768095fc057ca73ef4, Logger=SIMModularInput, File=sim_modular_input.py, Line=477, status=error, action=run_signalflow_program, instance_name=Test-Host, org_id=Abcde-AwAE, thread_id=a80e5865d7f146ad84df7ed5813990ae, retry_count=2, error_msg=400: Error executing SignalFlow program (ref: GjhygFlA0AA): [File "<program>", line 1, in
data('cpu.utilization').publish(), data('memory.utilization').publish(), data('memory.free').publish(), data('memory.total').publish(), data('memory.used').publish(), data('disk.utilization').publish(), data('disk_ops.read').publish(), data('disk_ops.write').publish()" with_derived_metadata=true
SyntaxError: token recognition error at: '" with_derived_metadata=true']
Traceback (most recent call last):
File "/opt/splunk/etc/apps/splunk_ta_sim/bin/sim_modular_input.py", line 349, in run_signalflow_program
for msg in computation.stream():
File "/opt/splunk/etc/apps/splunk_ta_sim/bin/../libs/external/signalfx/signalflow/computation.py", line 122, in stream
message = next(iterator)
File "/opt/splunk/etc/apps/splunk_ta_sim/bin/../libs/external/signalfx/signalflow/channel.py", line 43, in __next__
return self._next()
File "/opt/splunk/etc/apps/splunk_ta_sim/bin/../libs/external/signalfx/signalflow/ws.py", line 315, in _next
raise errors.SignalFlowException(
signalfx.signalflow.errors.SignalFlowException: 400: Error executing SignalFlow program (ref: GjhygFlA0AA): [File "<program>", line 1, in
data('cpu.utilization').publish(), data('memory.utilization').publish(), data('memory.free').publish(), data('memory.total').publish(), data('memory.used').publish(), data('disk.utilization').publish(), data('disk_ops.read').publish(), data('disk_ops.write').publish()" with_derived_metadata=true
SyntaxError: token recognition error at: '" with_derived_metadata=true']

The error occurs when the SIM Add-on attempts to execute a SignalFlow program to fetch and publish metric data but encounters a syntax error due to an incorrect format in the SignalFlow program.

This error is due to the incorrect formatting in the SignalFlow program, where an additional token, " with_derived_metadata=true", is causing a syntax error.

Below is the sample format of the Modular Input with SignalFlow Program :

"data('cpu.utilization').publish(), data('memory.utilization').publish(), data('memory.free').publish(), data('memory.total').publish(), data('memory.used').publish(), data('disk.utilization').publish(), data('disk_ops.read').publish(), data('disk_ops.write').publish()"

The issue caused by the incorrect format is mentioned as below: 

"data('cpu.utilization').publish(), data('memory.utilization').publish(), data('memory.free').publish(), data('memory.total').publish(), data('memory.used').publish(), data('disk.utilization').publish(), data('disk_ops.read').publish(), data('disk_ops.write').publish()" with_derived_metadata=true

Resolution

To resolve the issue, please provide the SignalFlow Program in the correct format as per the SignalFlow Documentation: SignalFlow Documentation.

Related Links

Configure Modular Input in SIM Add-on: Splunk Infrastructure Monitoring Add-on.

SignalFlow Documentation: SignalFlow Methods.