My APM trace volume increased 17–20% and I see duplicate attributes on my spans

Available Languages

Updated:October 1, 2026
Document ID:SPLK054908

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Application Performance Monitoring

Description

When viewing APM traces in Splunk Observability Cloud, spans show duplicate database attributes. For example, a single span may contain both:

 

  • db.statement (legacy) and db.query.text (stable)

  • db.system and db.system.name

  • db.name and db.namespace

  • db.operation and db.operation.name

  • db.sql.table and db.collection.name

 

The user has not set OTEL_SEMCONV_STABILITY_OPT_IN as an environment variable or JVM system property, and is not using a K8s Operator or Instrumentation CRD. The agent version is pinned and no custom extensions are activating the database/dup mode.

 

This raises concerns about whether the duplicated attributes are inflating APM trace volume and billing.

 


Investigation

  1. Confirm the agent is only emitting legacy attributes by checking agent debug logs or exporter output. Spans leaving the agent should contain only legacy DB attributes (e.g. db.statement, db.system, db.name).

  2. Confirm no semconv opt-in flags are set:

 
kubectl exec -it <pod_name> -n <namespace> -- \ cat /proc/1/cmdline | tr '\0' '\n' | grep -iE "semconv|stability|opt.in"
 

  1. Confirm no environment variable is set:

 
kubectl exec -it <pod_name> -n <namespace> -- env | grep -i SEMCONV
 

  1. If using a custom OTel Java agent extension (-Dotel.javaagent.extensions=), verify the extension does not contain an AutoConfigurationCustomizerProvider that calls addPropertiesCustomizer or addPropertiesSupplier to set semconv-related properties. Request the SPI manifest (META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider) and the provider source code.

  2. If all of the above are clean, the duplication is server-side (see Resolution).

 


Resolution

The duplication is applied server-side by the APM backend, not by the agent.

 

The apm-svc-trace-analysis service remaps legacy DB attributes to their stable equivalents after the trace is received. Most are configured as copies, meaning both the legacy and stable attribute names are retained on the span.

 

The full DB remapping list:

Legacy attribute

Stable attribute (copy)

db.system

db.system.name

db.name

db.namespace

db.statement

db.query.text

db.operation

db.operation.name

db.sql.table

db.collection.name

 

Billing is not affected. Volume is metered at the point of ingest, before any remapping occurs. The copied attributes added by the backend do not count toward billed APM trace volume (sf.org.apm.numSpanBytesReceived).

 

Feature flag (opt-out of copy behaviour):

 

The feature flag apm.tag.remapping.db.statement.enabled controls this behaviour:

 

  • Opt-in (default): COPY. Both legacy and stable attribute names are kept.

  • Opt-out: REMAP. Only the stable name is kept (reduces visual noise, no billing impact either way).

 

Contact Splunk Support to request a change to this feature flag for a specific org.

 


 

Revision History

Revision Publish Date Comments
1.0
01-Oct-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco