HTTP 500 Responses Not Counted as Errors in APM Tag Spotlight Despite Appearing in Traces

Applicability

Description

A service is returning HTTP 500 responses which are visible in individual APM traces (span tags correctly show http.response.status_code = 500 and otel.status_code = ERROR on child spans). However, in Tag Spotlight the error count remains at zero:

The APM Error Code Configuration has been set to capture all 1xx/2xx/3xx/4xx/5xx HTTP status codes, yet errors are still not counted.

The service is using custom OpenTelemetry instrumentation (identified by otel.scope.version: 0.0.1 and the absence of standard SDK resource attributes such as telemetry.sdk.language, telemetry.sdk.version, or splunk.distro.version).

Investigation

1. Confirm the 500 is on child spans, not the entry span

Navigate to APM > Traces and filter by the affected service. Open a trace containing an HTTP 500 response.

Identify the first span in the trace for that service (the service entry span). Check the following attributes:

• span.kind — Is it SERVER or CONSUMER?

• http.response.status_code — Is it present?

• otel.status_code — Is it set to ERROR?

In this scenario, the entry span shows:

span.kind: INTERNAL
http.response.status_code: (not present)
otel.status_code: (not present)

Child spans deeper in the trace (e.g., 3rd, 9th, 10th spans) correctly show:

http.response.status_code: 500
otel.status_code: ERROR
span.kind: SERVER (or CLIENT)

2. Understand why Tag Spotlight shows the 500 but Err = 0

Tag Spotlight is powered by Troubleshooting MetricSets (TMS). TMS uses a "union" view of tags — it extracts tag values from ALL spans in the service tier, not just the entry span. This is why http.response.status_code = 500 appears with Req = 13.

However, error counting is powered by Monitoring MetricSets (MMS). MMS only generates endpoint-level metrics from spans with span.kind = SERVER or span.kind = CONSUMER. Since the entry span has span.kind = INTERNAL:

• MMS cannot generate metrics for this service endpoint

• The Error Code Configuration has no eligible span to evaluate

• Errors will always show as zero

3. Confirm custom instrumentation

Check for the absence of standard OpenTelemetry SDK resource attributes on any span:

telemetry.sdk.language: (not present)
telemetry.sdk.version: (not present)
splunk.distro.version: (not present)
otel.scope.version: 0.0.1

This indicates the service is using manual/custom OTel instrumentation rather than a standard Splunk or OpenTelemetry auto-instrumentation agent.

Resolution

Update the service entry span's span.kind

The instrumentation must set span.kind = SERVER (or CONSUMER for message-based services) on the service's top-level entry span. This is required for Splunk APM to generate Monitoring MetricSets.

Propagate HTTP status code to the entry span

Ensure http.response.status_code is set as a tag on the entry span. Without this, the APM Error Code Configuration cannot evaluate the response.

Set error status on the entry span

When the service returns an HTTP 5xx response, set otel.status_code = ERROR on the entry span. Alternatively, set the error = true tag.

Consider adopting Splunk's auto-instrumentation agent

Splunk's OpenTelemetry auto-instrumentation agents handle span.kind, HTTP attribute capture, and error status propagation automatically. This eliminates the need for manual instrumentation of these attributes.

Refer to Instrument back-end applications to send spans to Splunk APM