Splunk Infrastructure Monitoring Add-on (SIM) Add-On Command Data Ingestion Latency

Applicability

Splunk Infrastructure Monitoring Add-on Splunk Enterprise Splunk Cloud

Description

Experiencing a latency issue with the "|sim" flow command, which retrieves metrics from Splunk Observability. For specific metrics, there is a discrepancy where data points from the last few minutes are not captured.

Despite the metrics being accessible almost in real-time through the Splunk Observability UI and its REST API, a delay of a few minutes is observed when employing the "|sim" command.

Investigation

Run the below SPL Query to check if there are any error messages in the _internal logs,

SPL:

index=_internal *sim*

The internal logs exclusively consist of INFO entries; no ERROR or WARNING logs are present.

After a more in-depth review, it was found that the issue is coming from Splunk SDK, simlib/singalfx:
https://github.com/signalfx/signalfx-python/blob/master/signalfx/signalflow/computation.py

The Sim-Addon is using a computation stream to get the data by using the above-mentioned Splunk SDK which adds the delay when trying to retrieve the data. Also saw the data delay/stuck in the Observability itself.



A 15-minute gap is evident, and upon data resumption, a sudden linear graph appears with no data gap between 18:57:00 and 19:03:00. The delay in the Add-on, attributed to the Splunk SDK, is the challenge currently faced. There's an assumption that the SDK is crafted to prevent data delays or obstructions in Observability, ensuring a continuous flow of data.

For the delay of more than 15mins, it is due to the SIM Add-on - Intermittent Modular input hang issues

The modular inputs configured in the Splunk Infrastructure Monitoring Add-on v1.2.3 seem to become unresponsive after a certain period and in the internal logs, there do not appear to be any error or warning messages. When these inputs are re-enabled, the logs start flowing again.

The issue was due to SignalFx Library and for the same, a new version with an enhancement has introduced ie. v1.2.4 to resolve the issue.

In the newer version, the SIM-Addon team will add a heartbeat check on their own end which will monitor the flow of data. If data is not being received after a particular time interval existing input will terminate the existing connection and let the modular input run again forming a new connection and resuming the data flow.

Heartbeat check mechanism which will monitor the data streams and terminate the connection if the streams are stuck and start it again.

Resolution

The Latency issue is from the Streaming Library SignalFx used by the Add-on. Since it is not a request-response API, the data streaming will be returned once it satisfies the condition as per the SignalFx Dataset.

A delay of up to 15 minutes is expected.

If the delay is more than 15 Minutes, need to upgrade the SIM Add-on to v1.2.4 which involves the implementation of a heartbeat check mechanism to resolve the Intermittent Modular input hang issues.

Related Links

SIM Add-on v1.2.4 Fixed Issues

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release