o11y IM Wrong numbers/missing hosts on single value/list/heatmap chart - Missing recent or delayed data

Applicability

Description

Delayed Data

Wrong numbers/missing hosts on a single value/list/heatmap chart could be caused by missing recent or delayed data. Delayed data could be caused by high value of the data lag. Lag for data can be verified by using rollup lag: https://docs.splunk.com/Observability/data-visualization/charts/data-resolution-and-rollups.html#rollups
 

Recent data is missing or delayed.

Example: 
Missing hosts/services/endpoints/data on Single value chart.

More than 1 result returned from a chart

Another common issue is when more than result is returned from the chart. The single value chart will choose any of these values at random and display it.

Investigation

Delayed Data

A high value of lag -> wrong time set on a host
Single value, List and Heatmap charts require recent data points to plot data for that MTS.

Resolution

Delayed Data

Lag for data can be verified by using rollup lag:  https://docs.splunk.com/Observability/data-visualization/charts/data-resolution-and-rollups.html#rollups

Lag is the delta between logical time (the time stamp that accompanies the datapoints, i.e. when the measurements are actually taken) and wall time (the time at which the datapoints actually arrive at SignalFx).

If the value of the lag is higher than expected then consider the potential causes of data delay between collecting them from the infrastructure instance and delivering them to the SignalFx servers.

Two of the most popular causes of delayed/missing data are:

1. incorrect time setting on the host
2. network issues 

Ad. 1.

Incorrect time set on a host which causes sending datapoints with wrong timestamps.

On host time will be delayed 4 hours it will ingest those datapoints not as current datapoints but with time -4 hours. This will lead to a 4-hour delay in the presentation of current data.

Lag can also be a negative number. That means the timestamp of gathering datapoints is older than timestamps of ingesting data to our system. That would suggest that time setting on a host is set later than the current time.

Ad. 2.
High lag may also result from issues in the client's infrastructure, e.g. network problems may cause delays in delivering data points on time

More than 1 value returned

The best approach here is to do an aggregate or mean on the entire data so that what is returned is a single value.

Related Links

Data-visualization charts data-resolution-and-rollups