feat(tracer): Implement OTLP Traces Export by mtoffl01 · Pull Request #4600 · DataDog/dd-trace-go

mtoffl01 · 2026-03-25T16:45:33Z

What does this PR do?

Implements the OTLP trace export pipeline (See: RFC) for dd-trace-go: when OTEL_TRACES_EXPORTER=otlp is set, the tracer converts Datadog spans into OTLP protobuf format and sends them directly to an OpenTelemetry Collector instead of the Datadog agent.

This adds three new components:

span_to_otlp.go — converts internal Datadog spans to OTLP TracesData protobuf, including resource attributes, span kind/status, events, links, 128-bit trace IDs, and sampling filtering (only sampled spans are exported).
otlp_writer.go — a traceWriter implementation that buffers converted OTLP spans, marshals them to protobuf, and flushes with retry logic.
otlp_transport.go — a lightweight HTTP transport for sending protobuf payloads to an OTLP endpoint, separate from the existing Datadog-protocol transport.

The PR also renames the existing transport interface and config.transport field to ddTransport to make it clear that it handles Datadog-specific traffic (msgpack traces, stats), and ensures ddTransport always points at the Datadog agent even when OTLP mode is active.

Motivation

We are adding native OTLP export support to dd-trace-go so users can send traces directly to any OTLP-compatible collector. A prior PR added the configuration layer that resolves OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, and OTEL_EXPORTER_OTLP_TRACES_HEADERS. This PR builds the runtime pipeline on top of that configuration.

IMPORTANT NOTE!

The existing agentTraceWriter and ddTransport was too tightly coupled to Datadog conventions — msgpack encoding, agent sampling rate feedback, stats collection — to extend it for the otlp path, so a separate writer and transport were introduced instead. The otlpTraceWriter owns its own otlpTransport directly (rather than reaching through config.ddTransport), keeping OTLP concerns isolated from the Datadog agent path. A follow-up PR may remove ddTransport from the config struct entirely, instead passing the transport directly to each writer and consumer at construction time. This would make all writer implementations structurally consistent — each owning its transport as a field — and eliminate the implicit coupling through config.

An additional PR to disable stats collection in otlp mode is in progress here.

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
New code is free of linting errors. You can check this by running make lint locally.
New code doesn't break existing tests. You can check this by running make test locally.
Add an appropriate team label so this PR gets put in the right place for the release notes.
All generated files are up to date. You can check this by running make generate locally.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running make fix-modules locally.

Unsure? Have a question? Request a review!

…port

… to determine otlp mode

Made-with: Cursor

…traceURL

…nfig registry to reflect this

…ut ddspan -> otlpspan conversion

… span_to_otlp_test.go

… service name differs from global service name

…me (#4576) ### What does this PR do? Moves the `strconv.ParseInt` call for `_dd.p.dm` (decision maker) out of the v1 payload flush path and into the write path. A `dm uint32` field is added to the `trace` struct. It is kept in sync with `propagatingTags[keyDecisionMaker]` at all mutation sites (`setPropagatingTagLocked`, `unsetPropagatingTagLocked`, `replacePropagatingTags`). A shared `unsetPropagatingTagLocked` helper is introduced so the delete-and-clear logic is not duplicated between `unsetPropagatingTag` and `setSamplingPriorityLockedWithForce`. `payloadV1.push` now reads the pre-computed value via `trace.decisionMaker()` instead of fetching the string from the propagating tags map and parsing it on every flush. ### Motivation `payloadV1.push` is called once per finished trace chunk. On every call it was acquiring a read lock, doing a map lookup, and then running `strconv.ParseInt` — all to read a value that was set at most once or twice during the lifetime of the trace. Moving the parse to write-time eliminates the per-flush map lookup and parse, replacing them with a single scalar read behind an `RLock`. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [x] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. Co-authored-by: dario.castane <dario.castane@datadoghq.com>

…ED` once, use a global `bool` (#4548) Stop checking environment variable `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED` on span start. Checks it once, use a global `bool`. It'd be better to load it on `newConfig` and make the value available, but this one happens in `newSpanContext`, which modification will cause a lot of changes in multiple files, including tests. Complete #1905 work using the current benchmarking platform. - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] There is a benchmark for any new code, or changes to existing code. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [x] Add an appropriate team label so this PR gets put in the right place for the release notes. Unsure? Have a question? Request a review! Co-authored-by: kakkoyun <kakkoyun@users.noreply.github.com> Co-authored-by: kemal.akkoyun <kemal.akkoyun@datadoghq.com>

Signed-off-by: Moe Zein <moe.zein@datadoghq.com>

Co-authored-by: Benjamin De Bernardi <debernardi.benjamin@gmail.com>

Co-authored-by: Dario Castañé <d@rio.hn>

ddtrace/tracer/span_to_otlp.go

… ParentId from proto when 0

hannahkm · 2026-04-06T18:26:51Z

ddtrace/tracer/span_to_otlp.go

+
+// +checklocksignore — Post-finish: reads finished span fields during payload encoding.
+func convertSpan(s *Span, defaultServiceName string) *otlptrace.Span {
+	if p, ok := s.context.SamplingPriority(); ok && p < ext.PriorityAutoKeep {


I don't think there's ever a case where we get ok = false (right?), but what is the intended behavior if we get ok=false? Should the span be kept by default (as it is currently)?

In practice, no we never expect to get ok = false. But theoretically if we were to, this would mean the span has no sampling (or trace-level) information on it, and we would drop the span. Given that we would consider this span would corrupt, dropping it -- as we do here -- makes sense.

Also: This particular implementation is dropping spans that are not marked keep, as is the otel spec for client side sampling.

ddtrace/tracer/otlp_transport.go

…he same TCP connection instead of creating a new one for every request. Also, add tests.

hannahkm

one last question!

hannahkm · 2026-04-07T20:16:10Z

ddtrace/tracer/otlp_writer.go

+	needsFlush := w.buffSize > payloadSizeLimit
+	w.mu.Unlock()
+	if needsFlush {
+		w.flush()
+	}


What do you think is the possibility of w getting changed between the unlock and the flush() call? I wonder if it's worth holding the lock until after the flush is done, and having a separate flushWithLock() function? What do you think?

Hmm..
needsFlush captures a snapshot of w in a moment that can serve as a trigger. flush() acquires the lock itself, so it is safe. If, say some other goroutine concurrently accessed flush while we were on line 78-79, we would then:

enter flush with a lock

see len(w.spans) == 0, and return immediately

So there wouldn't be any duplicate sending or race condition here.

kakkoyun · 2026-03-27T10:26:26Z

ddtrace/tracer/option.go

-// resolveTraceTransport returns the trace URL and headers for the transport
-// based on whether OTLP export mode is active.
+// resolveTraceTransport returns the trace URL and headers for the Datadog
+// agent transport. In OTLP export mode the ddTransport is not used for trace


kakkoyun · 2026-04-09T15:15:38Z

ddtrace/tracer/otlp_transport.go

+}
+
+// send posts a protobuf-encoded payload to the configured OTLP endpoint.
+func (t *otlpTransport) send(data []byte) error {


This method is super close to io.Writer I wonder if it would make sense to match them 🤔 I'm thinking out loud here.

I believe this is what the msgpack implementation is based on. But, in the otlp implementation the "send" logic is quite different - we are storing everything in memory up until the point that we are ready to flush, and only at that point do we encode the data. The encoding happens just once per payload - at flush time - not iteratively.

kakkoyun · 2026-04-09T15:19:56Z

ddtrace/tracer/otlp_writer.go

+	readySpans := w.reset()
+	w.mu.Unlock()
+
+	w.climit <- struct{}{}


We can just use https://pkg.go.dev/golang.org/x/sync/errgroup instead of implementing this on our own.

How do you envision this?

otlpTraceWriter.flush was modeled after agentTraceWriter.flush; changing one but not the other would break consistency, and I'd rather not modify agentTraceWriter in this PR.

Also, as I undestand it, errgroup would clean up the concurrency-limiting boilerplate, but it wants to cancel on the first error, whereas we just log and continue.

mtoffl01 and others added 30 commits March 23, 2026 09:30

Allow Config to resolve traceURL; pass this as a field into httptrans…

8f7f60f

…port

Fix otlp protocol resolution and add tests for all new logic

3e3fec8

Implement OTEL_EXPORTER_OTLP_TRACES_HEADERS

c559f8f

Fix superfluous referecne to unsupport OTLP header config

cb6908e

Introduce otlpExportMode on Config; use this instead of traceProtocol…

a85fedf

… to determine otlp mode

fix relationship between otlpexportmode and protocol

33a9a37

replace tracer.config.otlpExportMode with internalconfig otlpExportMode

3e51fdd

Made-with: Cursor

Clean up resolveOTLPHeaders

5e252be

cleanup

51f72a2

more cleanup and remove OTEL_EXPORTER_OTLP_ENDPOINT

b71eec7

Store otlpTraceURL instead of traceURL on Config; let tracer resolve …

f48d44f

…traceURL

Make DD_TRACE_AGENT_PROTOCOL_VERSION override OTEL_TRACES_EXPORTER

f4360b4

Merge branch 'main' into mtoff/otlp-export-config

49d0937

Use GetMap to resolve OTEL_EXPORTER_OTLP_TRACES_HEADERS and change co…

a88b7b2

…nfig registry to reflect this

give OTEL_EXPORTER_OTLP_TRACES_ENDPOINT default value of null

d475920

Merge branch 'main' into mtoff/otlp-export-config

8188bbe

flesh out traceWriterOtlp

af909c3

add new otlpSender for transporting otlp traces specifically; build o…

9246084

…ut ddspan -> otlpspan conversion

rename file payload_otlp_convert.go -> span_to_otlp.go; and introduce…

a934fdd

… span_to_otlp_test.go

introduce otlp_writer_test.go: basic otlp writer tests

2cfa217

Fix: only export sampled spans, and correct telemetry.sdk.version value

7bba29d

Test buildResource

89cb7e6

implement edge case: add 'service.name' attribute to span when span's…

da1efd4

… service name differs from global service name

add comment about s.metaStruct

49053c1

chore: disable automated dependency updater config [incident-51602]

ffbea4e

Signed-off-by: Moe Zein <moe.zein@datadoghq.com>

merge conflicts

9882338

merge conflicts

5072b2a

Apply suggestion from @genesor

78e160f

Co-authored-by: Benjamin De Bernardi <debernardi.benjamin@gmail.com>

darccio requested review from darccio and removed request for darccio March 27, 2026 17:46

mtoffl01 and others added 4 commits March 27, 2026 14:08

Fix converSpanAttribute

cd63872

Apply suggestions from code review

12d75b3

Co-authored-by: Dario Castañé <d@rio.hn>

Make sure newBenchOTLPWriter has scope

af69c6b

Address agent UDS url issue: have otlp writer create its own client

35cf849

zacharycmontoya reviewed Mar 30, 2026

View reviewed changes

ddtrace/tracer/span_to_otlp.go Outdated Show resolved Hide resolved

PR comments: promote service.name and operation.name attributes; omit…

6480f53

… ParentId from proto when 0

mtoffl01 requested a review from zacharycmontoya April 6, 2026 14:22

hannahkm reviewed Apr 6, 2026

View reviewed changes

Add new benchmarks to BENCHMARK_TARGETS

e8aab3a

mtoffl01 requested review from a team as code owners April 7, 2026 18:20

mtoffl01 and others added 2 commits April 7, 2026 14:21

Merge branch 'main' into mtoff/otlp-export-traces

4223bd3

Fix: Capture response body in otlpTransport.send, in order to reuse t…

037eec3

…he same TCP connection instead of creating a new one for every request. Also, add tests.

mtoffl01 requested a review from hannahkm April 7, 2026 19:39

hannahkm reviewed Apr 7, 2026

View reviewed changes

mtoffl01 and others added 3 commits April 7, 2026 16:36

fix lint problem in otlp_transport_test

ca17d6a

Fix race condition in TestOTLPTransportConnectionReuse

63bc9ea

Merge branch 'main' into mtoff/otlp-export-traces

f9ae432

ddyurchenko approved these changes Apr 8, 2026

View reviewed changes

kakkoyun reviewed Apr 9, 2026

View reviewed changes

mtoffl01 added 5 commits April 9, 2026 14:06

drain otlpTransport response body after reading status code

91fac2a

Cache base TracesData size

7c12988

Use Span_SPAN_KIND_UNSPECIFIED for unknown span types

f577ca2

wrap error in otlpTransport.send

ceea3dd

enforce Content-Type header at the otlpTransport.send level

b3a521d

mtoffl01 requested review from hannahkm and kakkoyun April 13, 2026 17:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tracer): Implement OTLP Traces Export#4600

feat(tracer): Implement OTLP Traces Export#4600
mtoffl01 wants to merge 63 commits intomainfrom
mtoff/otlp-export-traces

mtoffl01 commented Mar 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

hannahkm Apr 6, 2026

Uh oh!

mtoffl01 Apr 7, 2026

Uh oh!

Uh oh!

hannahkm left a comment

Uh oh!

hannahkm Apr 7, 2026

Uh oh!

mtoffl01 Apr 7, 2026

Uh oh!

kakkoyun Mar 27, 2026

Uh oh!

kakkoyun Apr 9, 2026

Uh oh!

mtoffl01 Apr 9, 2026

Uh oh!

kakkoyun Apr 9, 2026

Uh oh!

mtoffl01 Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

mtoffl01 commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

IMPORTANT NOTE!

Reviewer's Checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hannahkm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mtoffl01 commented Mar 25, 2026 •

edited

Loading