NGINX Service Not Appearing in Splunk APM Service Map with OpenTelemetry Operator Annotation

Applicability

Description

A user was attempting to onboard Python-based microservices and NGINX to Splunk Observability Cloud. While the Python services successfully appeared in the APM Service Map, the NGINX service remained absent.

The user followed standard helm commands to install the splunk-otel-collector and applied a Kubernetes patch to enable Python instrumentation. For NGINX, the user attempted to use the Kubernetes annotation instrumentation.opentelemetry.io/inject-nginx: "true", expecting it to enable NGINX traces in APM. Despite NGINX metrics potentially being visible in Infrastructure Monitoring, the NGINX service did not appear in the APM Service Map.

During investigation, internal logs from the OpenTelemetry Operator sometimes showed errors such as "error":"no OpenTelemetry Instrumentation instances available", indicating that the operator was not properly injecting or applying the expected instrumentation to the NGINX deployment. Attempts to align the NGINX version with those explicitly supported by the OpenTelemetry Operator (e.g., downgrading to NGINX 1.23.1) also did not resolve the issue of missing APM traces.

Investigation

To determine if a user is encountering this specific issue, the following steps can be taken:

1. Verify NGINX Presence in Infrastructure Monitoring vs. APM:
   
   

   • Confirm that NGINX metrics are being collected and displayed in Splunk Observability Cloud's Infrastructure Monitoring. This indicates the collector is generally functioning for NGINX metrics.
   • Check the APM Service Map to confirm the NGINX service is indeed missing, while other instrumented services (e.g., Python microservices) are visible.
     
     

2. Review Kubernetes Annotations for NGINX:
   
   

   • Inspect the NGINX deployment or pod configuration for the presence of the instrumentation.opentelemetry.io/inject-nginx: "true" annotation.
   • Example command to check pod annotations
     
     

3. Examine OpenTelemetry Operator Logs:
   
   

   • If the OpenTelemetry Operator is deployed, review its logs for errors related to NGINX instrumentation. Look for messages similar to:
      

     "error":"no OpenTelemetry Instrumentation instances available"

   • Example command to view operator logs:
     
     

      

     kubectl logs -l app.kubernetes.io/name=opentelemetry-operator -n opentelemetry-operator-system

     (Adjust label and namespace as per deployment)
     
     

4. Confirm NGINX Role:
   
   

   • Determine if NGINX is intended to act as a proxy for other instrumented applications, or if traces are expected directly from NGINX itself. The APM Service Map visualizes application distributed traces, and NGINX will only appear if it's configured to proxy traces from instrumented applications behind it, or if it's specifically instrumented to generate its own traces.
     
     

5. Consult Splunk OpenTelemetry Collector NGINX Documentation:
   
   

   • Refer to the official Splunk OpenTelemetry Collector documentation for NGINX auto-instrumentation to understand the supported configuration methods. This helps differentiate between generic OpenTelemetry community approaches and Splunk's specific implementation.

Resolution

The issue of NGINX traces not appearing in the Splunk APM Service Map, despite attempts at auto-instrumentation, is typically resolved by understanding and correctly implementing Splunk's specific NGINX auto-instrumentation capabilities rather than relying on generic OpenTelemetry Operator annotations.

Here's how to address this:

1. Understand Splunk's NGINX Auto-Instrumentation:
   
   

   • The generic instrumentation.opentelemetry.io/inject-nginx: "true" annotation provided by the OpenTelemetry Operator is primarily designed for nginx-ingress-controller and is maintained by the broader OpenTelemetry community. Splunk does not officially support or validate this specific method for general NGINX trace auto-instrumentation within its Observability Cloud ecosystem.
   • The Splunk OpenTelemetry Collector has its own built-in NGINX auto-instrumentation for both traces and metrics.
     
     

2. Refer to Official Splunk Documentation:
   
   

   • The definitive guide for NGINX auto-instrumentation with the Splunk OpenTelemetry Collector is available here
   • This documentation outlines the specific configuration required within the Splunk OpenTelemetry Collector Helm chart values (e.g., enabling agent.discovery.enabled=true and nginx: enabled: true under the appropriate sections) to ensure NGINX traces are collected and sent to APM.
     
     

3. Implement Splunk-Specific Configuration:
   
   

   • Instead of using the generic OpenTelemetry Operator annotation, configure the Splunk OpenTelemetry Collector according to the official Splunk documentation. This typically involves modifying the Helm chart values used during the splunk-otel-collector installation or upgrade.
     
     

4. Consider Manual Instrumentation (If Applicable):
   
   

   • If auto-instrumentation proves challenging or if NGINX is acting as a proxy for other instrumented services, manual instrumentation of NGINX can be an alternative. This involves configuring NGINX to forward trace context headers from upstream services.