Inaccurate histogram_percentile() Results with Prometheus Histogram Metrics in Splunk Observability Cloud

Available Languages

Updated:October 1, 2026
Document ID:SPLK052113

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Infrastructure Monitoring

Description

There were observed inconsistent results when using histogram_percentile() on metrics derived from Prometheus histograms. The issue appeared more prominently when querying over longer timeframes (e.g. 5h), where the p50 values unexpectedly dropped to zero despite bucket data showing otherwise. The pipeline involved Prometheus-instrumented applications forwarding data through a Prometheus receiver and OTLP exporter to an OpenTelemetry Collector configured with the SignalFx exporter.

The SignalFlow query applied rollup='delta' to histogram bucket metrics and aggregated across partitions before calling .histogram_percentile(pct=50).


Investigation

The problem is as follows:

  • The histogram bucket metrics (*_bucket) were ingested as GAUGE type with AVERAGE native rollup, as verified through internal MTS queries.
  • Applying rollup='delta' to a GAUGE metric leads to incorrect calculations. Instead of computing the delta between cumulative values, SignalFlow calculates the delta across averaged samples, resulting in distorted or zero values—particularly during restarts or traffic variation.
  • The OpenTelemetry Collector’s SignalFx exporter did not have send_otlp_histograms: true configured. As a result, incoming OTLP histogram metrics were decomposed into _sum, _count and potentially dropped or altered during translation.

Resolution

To resolve the issue and ensure accurate percentile calculations using histogram_percentile():

    •    Set send_otlp_histograms: true in the SignalFx exporter configuration of the OpenTelemetry Collector. This preserves the histogram structure in OTLP format instead of decomposing it.

    •    Avoid using rollup='delta' on GAUGE metrics. Percentile calculations require cumulative counters or properly ingested histogram types.


https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/DESIGN.md#histogram

Revision History

Revision Publish Date Comments
1.0
01-Oct-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco