Java Instrumentation fails to export data with io.opentelemetry.exporter.internal.http.HttpExporter ERROR

Applicability

Description

The new major 2.x version of splunk-otel-java contains significant breaking changes that will most likely affect all users. Please be sure to read all the breaking changes listed in the release notes carefully. 

In Java 2.x instrumentation, the default OTLP protocol changed from grpc to http/protobuf in order to align with the specification. This means that, in 2.x versions of splunk-otel-java, the default exporter has been changed from grpc on port 4317 to HTTP on port 4318 (this is an OpenTelemetry standardization).

If a custom configuration overrides the default endpoint setting, you may see issues with data being exported from the instrumented Java application.

Please note: Splunk OTel Java version 1.x is deprecated. Existing userss should consider migrating to version 2.5.0 or higher. 

Investigation

To investigate if data is not being ingested due to this protocol change from grpc to http/protobuf, enable and collect Java instrumentation debug logging.

Activate debug logging for the Java agent with one of the following options: 

1. Pass the following argument when running your application: 

   Dotel.javaagent.debug=true

2. Set the following environment variable before running your application: 

   OTEL_JAVAAGENT_DEBUG=true

Then, review the debug logs and look for errors involving the HttpExporter. For example, you may see errors like the following: 

[otel.javaagent 2024-06-28 12:59:00:034 +0000] [OkHttp http://opentelemetry-collector-service:4317/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export metrics. The request could not be executed. Full error message: Connection reset
java.net.SocketException: Connection reset
  at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source)
  at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source)
  at java.base/java.net.Socket$SocketInputStream.read(Unknown Source)]

[ERR] [otel.javaagent] [OkHttp http://localhost:4318/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4318

ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317

ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4318

Resolution

To resolve issues due to this default protocol change in Java 2.x instrumentation, you can either ensure your configuration is using the default http/protobuf exporter or switch back to gRPC protocol as described below.

Note: We recommend and encourage the use of defaults whenever possible. Unless you have a compelling reason for switching back to the non-default protocol, please use the default http/protobuf exporter configuration.

1. Use the default http/protobuf exporter 
   • Verify the following custom protocol configurations are not set:

     OTEL_EXPORTER_OTLP_PROTOCOL
     OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
     OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
     OTEL_EXPORTER_OTLP_LOGS_PROTOCOL

   • And, the custom endpoint configurations are not set: 

     OTEL_EXPORTER_OTLP_ENDPOINT
     OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
     OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
     OTEL_EXPORTER_OTLP_LOGS_ENDPOINT

      

     


2. Switch back to gRPC protocol
   
   • Set the custom protocol configuration to use grpc: 

     OTEL_EXPORTER_OTLP_PROTOCOL=grpc

   • Verify the custom endpoint configuration uses the correct port, e.g. 

     OTEL_EXPORTER_OTLP_ENDPOINT=http://<host>:4317

   • If AlwaysOn Profiling is configured for the Java agent, set the Profiler protocol to use grpc: 

     SPLUNK_PROFILER_OTLP_PROTOCOL=grpc

Then, in the OTel Collector configuration file, ensure the associated OTLP receiver protocols match those used by the Java agent. For example: 

otlp:
  protocols:
    grpc:
      endpoint: "${SPLUNK_LISTEN_INTERFACE}:4317"
    http:
      endpoint: "${SPLUNK_LISTEN_INTERFACE}:4318"

Related Links

Troubleshooting common-java telemetry-export issues

Java Migrate metrics.

Opentelemetry java instrumentation releases v2.5.0

Splunk-otel-java releases v2.0.0-alpha

Configure the Java agent