Multiple Host Availability Alerts Fired and MTS Count Multiplied After Upgrading Splunk OpenTelemetry Collector Gateway

Applicability

Description

After upgrading the Splunk OpenTelemetry Collector Gateway to v0.157.0 or newer, users may observe an unexpected increase in active Metric Time Series (MTS) counts along with multiple host availability or CPU utilisation alerts firing for a single host (for example, four separate alerts for a single four-core server).

This behaviour occurs because default metric translations in the signalfx exporter were updated in v0.157.0. In earlier versions (such as v0.126.0), the gateway exporter automatically aggregated translated per-core CPU metrics across the cpu attribute into a single host-level value before sending data to Splunk Observability Cloud. Starting in v0.157.0, default translations no longer aggregate across the cpu attribute. If upstream agents or local scrapers transmit individual per-core metrics (e.g. cpu0, cpu1, cpu2, cpu3), the gateway passes these metrics through individually, multiplying the overall MTS volume and triggering per-core alerts on host detectors.

Investigation

To determine whether an environment is impacted by unaggregated per-core CPU metric proliferation, perform the following steps:

1. Verify active Metric Time Series (MTS) count in Splunk Observability Cloud:
   Execute the following SignalFlow query in Metric Finder or on a dashboard chart to monitor active MTS volume:
   
   

A = data('system.cpu.utilization').count().publish(label='A')

Observe whether the count of active metric streams for system.cpu.utilization or cpu.utilization multiplied proportionally to the number of CPU cores per host following the gateway upgrade.

2. Check for the presence of the cpu dimension in metric datapoints:
   Run a SignalFlow metadata search or inspect the metric breakdown in Splunk Observability Cloud:

data('system.cpu.utilization', filter=filter('host.name', '<host_name>')).publish()

Inspect the output dimensions. If individual datapoints contain distinct cpu attributes such as cpu0, cpu1, cpu2, and cpu3 rather than a single aggregated metric per host, unaggregated per-core metric transmission is confirmed.

3. Inspect the gateway collector configuration and version:
   Review the active gateway configuration file gateway_config.yaml and verify the running collector version:
   
   

/opt/splunk-otel-collector/bin/splunk-otel-collector --version

Confirm if the running version is v0.157.0 or higher.

Resolution

To resolve CPU MTS proliferation and prevent per-core duplicate alerting, apply one of the following configuration options.

Option 1: Configure Aggregation at the Receiver Level

If metrics are generated using the hostmetrics receiver, ensure the CPU scraper is configured to report aggregated metrics rather than per-core metrics.

Update gateway_config.yaml under the receivers stanza:

receivers:
  hostmetrics:
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true

Note: In newer collector versions, system.cpu.utilization aggregates across logical CPUs by default when specific per-core metrics are not explicitly requested.

Option 2: Remove the cpu Attribute Using a Transform Processor

To drop the cpu attribute from incoming metrics at the gateway before metrics are exported to Splunk Observability Cloud, add a transform processor to the pipeline.

1. Add the transform processor definition to gateway_config.yaml:

processors:
  transform/drop_cpu_attribute:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "cpu")

2. Include the processor in the metrics service pipeline:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, transform/drop_cpu_attribute, batch]
      exporters: [signalfx]

3. Restart the Splunk OpenTelemetry Collector service to apply changes:
   
   

sudo systemctl restart splunk-otel-collector

Option 3: Filter Per-Core CPU Datapoints Using a Filter Processor

Alternatively, apply a filter processor to exclude datapoints containing specific per-core attributes.

1. Define the filter processor in gateway_config.yaml:

processors:
  filter/remove_per_core_cpu:
    metrics:
      datapoint:
        - 'attributes["cpu"] != nil'

2. Include filter/remove_per_core_cpu in the metrics pipeline within gateway_config.yaml and restart the collector service.