[Bug]: `otehttp.NewTransport` causes requests to hang when server responds without consuming the request body in v0.65.0

[akhilerm]

Component

Instrumentation: otelhttp

Describe the issue you're facing

containerd project uses a request implementation in which request body is via an io.Pipe. When the server responds without consuming the body of the request, the subsequent request hangs.

Bisected the changes and found that failures are seen after #8352

Expected behavior

Adding tracing should not cause the request to hang.

Steps to Reproduce

The below code has been referred from https://github.com/containerd/containerd/blob/bde439b02feea42936feb03bc40759b5427a42b8/core/remotes/docker/pusher.go#L70 to reproduce the issue

package otelhttp

import (
	"context"
	"io"
	"net/http"
	"net/http/httptest"
	"sync/atomic"
	"testing"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func TestOtelHTTPHangReproduction(t *testing.T) {
	var requestCount atomic.Int32

	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		count := requestCount.Add(1)
		if count == 1 {
			// the first request returns an error
			w.WriteHeader(http.StatusInternalServerError)
			return
		}
		io.Copy(io.Discard, r.Body)
		w.WriteHeader(http.StatusOK)
	}))
	defer server.Close()

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	type request struct {
		url  string
		body func() (io.ReadCloser, error)
		size int64
	}

	var pipeWriter *io.PipeWriter

	req := &request{
		url:  server.URL,
		size: 4,
		body: func() (io.ReadCloser, error) {
			pr, pw := io.Pipe()
			pipeWriter = pw
			return pr, nil
		},
	}

	doRequest := func() error {
		body, err := req.body()
		if err != nil {
			return err
		}

		httpReq, err := http.NewRequestWithContext(ctx, http.MethodPut, req.url, body)
		if err != nil {
			return err
		}
		httpReq.GetBody = req.body

		client := &http.Client{}

		// this modifies the transport
		client.Transport = otelhttp.NewTransport(http.DefaultTransport)

		go func() {
			pipeWriter.Write([]byte("test"))
			pipeWriter.Close()
		}()

		resp, err := client.Do(httpReq)
		if err != nil {
			return err
		}
		defer resp.Body.Close()

		if resp.StatusCode >= 400 {
			return &httpStatusError{code: resp.StatusCode}
		}
		return nil
	}

	// First request - should fail
	err := doRequest()
	if err != nil {
		t.Logf("First request failed as expected: %v", err)
	}

	// retry the request again
	err = doRequest()
	if err != nil {
		t.Fatalf("retry failed: %v", err)
	}
	t.Log("retry succeeded")
}

type httpStatusError struct {
	code int
}

func (e *httpStatusError) Error() string {
	return http.StatusText(e.code)
}

When using 0.64.0 of otehttp the above test passes and on 0.65.0 it hangs

The original test that fails in containerd on update : https://github.com/containerd/containerd/blob/bde439b02feea42936feb03bc40759b5427a42b8/core/remotes/docker/pusher_test.go#L78

Ref: containerd/containerd#12853
https://github.com/containerd/containerd/actions/runs/21660288757/job/62443840764?pr=12853#step:11:469

Operating System

Ubuntu

Device Architecture

x86_64

Go Version

1.24.13 / 1.25.7

Component Version

go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0

[dmathieu]

cc @adomaskizogian

[dmathieu]

This issue will be fixed by the bugfix in #8532