Discrepancy in GCP Distribution Metrics Between Google Cloud Platform and Splunk Observability Cloud

Applicability

Description

When monitoring GCP distribution metrics in Splunk Observability Cloud, users may observe discrepancies between chart values or detector signals and the metrics reported in the GCP console.

This behaviour occurs because Splunk Observability Cloud introduced native histogram support for GCP distribution metrics.

Prior to histogram support, GCP distribution metrics were split into three regular scalar metrics:

• baseMetricName + ".sumOfSquareDeviation" - sum of squared deviations.

• baseMetricName + ".count" - number of values reported at each interval.

• baseMetricName - mean value of the metric.

With native histogram support enabled, GCP distribution metrics are converted into two metrics:

• baseMetricName + ".sumOfSquareDeviation" - sum of squared deviations (unchanged).

• baseMetricName - the histogram metric created based on the GCP distribution metric.

Because histogram metrics do not define rollups (rollup='count' or rollup='average') and instead require specifying aggregation functions (count() or mean()) with dimensions (by=['dimension']), existing charts and detectors referencing legacy split metrics must be migrated to use the histogram() function.

Investigation

To confirm whether a chart or detector is affected by GCP distribution metric migration:

1. In Splunk Observability Cloud, open the affected Dashboard Chart or Detector.

2. Select Edit (or expand the SignalFlow editor).

3. Inspect whether the query uses data() with GCP distribution metrics, such as .count suffix or legacy rollup parameters.

Example Legacy SignalFlow Queries (data() function):

# Legacy count query
A = data('request_latency/e2e_latencies.count', rollup='sum').publish(label='A')# Legacy mean query
B = data('cpu_idle', rollup='average').mean(by=['host']).publish(label='B')

If the query uses data() for GCP distribution metrics, the chart or detector requires migration to histogram().

Resolution

Follow the step-by-step procedures below to migrate charts and detectors from legacy data() metrics to native histogram() metrics.

Prerequisites

Ensure Splunk Support has enabled the GCP distribution histogram feature for the organisation.

Migrating Dashboard Charts

Option A: Via Splunk Observability UI

1. Navigate to Dashboards and open the affected dashboard.

2. Select Edit on the chart.

3. Select the histogram function for the metric signal in the chart builder.

4. Select Save.

Option B: Via SignalFlow

Replace data() blocks with histogram() blocks and apply appropriate aggregation functions.

• Metric Count (.count):
  
  

  # Legacy
  A = data('request_latency/e2e_latencies.count', rollup='sum').publish(label='A')# Migrated
  A = histogram('request_latency/e2e_latencies').count(by=['response_code']).publish(label='A')

   

• Metric Mean (baseMetricName):
  
  

  # Legacy
  A = data('some/metric', rollup='average').publish(label='A')# Migrated
  A = histogram('some/metric').mean(by=['some_dimension']).publish(label='A')

Migrating Detectors

Option A: Via Splunk Observability UI

1. Open the affected detector and create a clone.

2. In the Signals tab of the cloned detector, select the histogram function for the metric signal.

3. Save and activate the new detector.

4. Remove or archive the old detector once the new detector is verified.

Option B: Via SignalFlow

Update the detector SignalFlow program by replacing data() statements with histogram() statements:

# Legacy Detector SignalFlow
A = data('cpu_idle', rollup='average').mean(by=['host'])# Migrated Detector SignalFlow
A = histogram('cpu_idle').mean(by=['host'])

Important Considerations and Caveats

1. Aggregations vs. Rollups

Histograms do not support rollups. Always specify an aggregation function (mean(), count()) and group by dimensions (by=['dimension1', 'dimension2']) where applicable. If no dimensions are specified in count() or mean(), the output aggregates across all matching time series.

2. Missing Minimum and Maximum Values

GCP distribution metrics do not currently expose minimum or maximum values. Therefore, min() or max() aggregations on histogram metrics will return 0 or no data.

To replicate legacy max() behaviour across time series, compute the mean per metric dimension first before taking the max across group dimensions:

# Legacy conversion
A = data('container/cpu/usage').max(by=['gcp_project_name']).publish(label='A')# Migrated histogram conversion (using container_name as time series distinguishing dimension)
A = histogram('container/cpu/usage').mean(by=['container_name', 'gcp_project_name']).max(by=['gcp_project_name']).publish(label='A')

3. Fallback Mechanism for Invalid Distribution Metrics

If a GCP metric contains invalid/missing bucket bounds or bucket counts exceeding limits, it enters a fallback path:

• baseMetricName + ".sumOfSquareDeviation" continues reporting normally.

• baseMetricName + ".count" remains unchanged (legacy charts/detectors continue working).

• baseMetricName becomes baseMetricName + ".mean" (legacy data() queries referencing baseMetricName will break).

Related Documentation

• SignalFlow data function documentation

• SignalFlow histogram function documentation