[BUG] OTLP exporter drop spans due since trace_buffer size is equal to 0

[maximebedard]

Agent Environment

# agent version
Agent 7.50.3 - Commit: abce0cb - Serialization version: v5.0.100 - Go version: go1.20.12

The log I'm observing:

2024-01-31 15:39:04 UTC | CORE | INFO | (exporter@v0.87.0/internal/common/logging_exporter.go:27 in pushTraces) | kind:exporter,data_type:traces,name:logging,resource spans:7,spans:7 | TracesExporter
2024-01-31 15:39:04 UTC | TRACE | WARN | (pkg/trace/api/otlp.go:306 in ReceiveResourceSpans) | Payload in channel full. Dropped 1 payload.

Describe what happened:

When publishing spans to the opentelemetry otlp grpc endpoint, I'm consistently getting dropped spans due. The log clearly indicates that the span will be dropped when writes to the channel will block. Since the channel is unbuffered by default (trace_buffer is set to 0 by default), this would almost always immediately block a drop spans.

I'm able to fix this issue by increasing the buffer size by setting DD_APM_TRACE_BUFFER=16, but according to the doc it is not recommended since it will increase memory usage.

Describe what you expected:

I would not expect the span to be immediately dropped. I would expect the request to block either for a given amount of time to at least give a chance to the channel to consume the data. Alternatively, the buffer size could be greater than 0 by default.

Steps to reproduce the issue:

Here is the configuration with which I'm running the agent:

docker run --rm --name dd-agent \
-p 4317:4317 \
-e DD_API_KEY="<API KEY>" \
-e DD_SITE="datadoghq.com" \
-e DD_ENV="dev" \
-e DD_LOGS_ENABLED="true" \
-e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_MAX_RECV_MSG_SIZE_MIB=16 \
-e DD_OTLP_CONFIG_TRACES_SPAN_NAME_AS_RESOURCE_NAME=true \
-e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT="0.0.0.0:4317" \
-e DD_OTLP_CONFIG_LOGS_ENABLED="true" \
-e DD_APM_NON_LOCAL_TRAFFIC="true" \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-v /var/lib/docker/containers:/var/lib/docker/containers:ro \
gcr.io/datadoghq/agent:7

I have a small demo application written in rust available at https://github.com/maximebedard/otlp-debug/. Make sure you have a recent version of the rust toolchain and simply run the demo app by running cargo run.

Additional environment details (Operating System, Cloud provider, etc):

I was running this locally on a 2022 MBP and using the latest version of the datadog agent using docker for mac.

uname -a
Darwin Maximes-MBP.lan 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64

[gmcnaught]

At first blush this appears to be an interaction due to the changes introduced in https://github.com/DataDog/datadog-agent/releases/tag/7.48.0 changing the apm queueing and backpressure logic.

In fact, there looks to be a PR that was closed attempting to address this issue in December:

#21372

[falsyvalues]

At first blush this appears to be an interaction due to the changes introduced in https://github.com/DataDog/datadog-agent/releases/tag/7.48.0 changing the apm queueing and backpressure logic.

In fact, there looks to be a PR that was closed attempting to address this issue in December:

#21372

Looks like this affects more OTEL communication than original DataDog app agent.

[knusbaum]

See: #23085

[dd-octo-sts]

This issue has been automatically marked as stale because it has not had activity in the past 15 days.

It will be closed in 30 days if no further activity occurs. If this issue is still relevant, adding a comment will keep it open. Also, you can always reopen the issue if you missed the window.

Thank you for your contributions!