Wrong attribute value appears on traces in Splunk APM

Applicability

Description

A user configured an identity attribute (in this example, gear_id) via OTEL_RESOURCE_ATTRIBUTES in the application's web.config, intending the value to be applied as a resource attribute on every span produced by the application. The user also had a collector-side attributes processor with action: insert configured for the same key, intended as a fallback for any application that does not set OTEL_RESOURCE_ATTRIBUTES.

When viewing traces in Splunk APM, two different gear_id values appeared on the same trace. One appeared in the Process section of the right-hand details panel, the other in the Tags section.

Example application configuration (web.config):

<add key="OTEL_RESOURCE_ATTRIBUTES" value="gear_id=1234,deployment.environment=PROD,service.version=1.0" />

Example collector configuration:

processors:
  attributes/fallback:
    actions:
      - action: insert
        key: gear_id
        value: 9999service:
  pipelines:
    traces:
      processors: [memory_limiter, batch, attributes/fallback]

Investigation

Open the affected trace in Splunk Observability Cloud APM Trace Analyzer. The right-hand details panel contains two relevant sections: "Tags" near the top and "Process" further down. Use the browser find function (Cmd+F or Ctrl+F) and search the panel for the attribute name (for example gear_id).

If the search returns 2 of 2 with different values, the trace has the attribute populated in two separate places:

• Match 1 appears in the Tags section. The value shown is the one inserted by the collector's attributes processor (in the example, 9999).
• Match 2 appears in the Process section. The value shown is the one from OTEL_RESOURCE_ATTRIBUTES in web.config (in the example, 1234).

In Splunk APM:

• The Process section displays resource attributes (set once at SDK initialisation, including those from OTEL_RESOURCE_ATTRIBUTES).
• The Tags section displays span attributes (set per span by instrumentation libraries or by collector processors that operate on span attributes).

Resource attributes and span attributes are separate namespaces in OpenTelemetry. The Splunk Distribution of OpenTelemetry Collector's attributes processor operates on the attributes of spans, logs, or metrics depending on the pipeline; it does not see or modify resource attributes. The resource processor is the equivalent component for resource attributes.

References:

• Attributes processor
• Resource processor

Because the attributes processor does not see the resource-level value set by OTEL_RESOURCE_ATTRIBUTES, its insert action does not detect that the key is already populated and proceeds to insert a separate value at the span-attribute level. Both values then coexist on the same trace in their respective namespaces.

Resolution

Two options are available depending on whether collector-side fallback behaviour is required.

Option 1 (recommended): Remove collector-side insertion and rely on OTEL_RESOURCE_ATTRIBUTES

This option aligns with the standard OpenTelemetry pattern where each service owns its own identity attributes. It removes the namespace mismatch entirely and simplifies the collector configuration.

1. Remove the attributes/* processor references for service identity attributes from the traces pipeline. The definitions in the top-level processors: block may remain if they are still referenced by other pipelines such as metrics.
   
   
2. Set identity attributes per application via OTEL_RESOURCE_ATTRIBUTES in each web.config. These will appear in the Process section of the trace view as resource attributes.

Example web.config:

<add key="OTEL_RESOURCE_ATTRIBUTES" value="gear_id=1234,business_unit=Finance,sub_business_unit=APAC,deployment.environment=PROD,service.version=1.0" />

processors:
  attributes/fallback:
    actions:
      - action: insert
        key: gear_id
        value: 9999

processors:
  resource/fallback:
    attributes:
      - action: insert
        key: gear_id
        value: 9999