-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Description
Datadog agent is currently using the following payload to check /api/v1/check_run endpoint
| checkRunPayload := []byte("{\"check\": \"test\", \"status\": 0}") |
While the Datadog backend seems to accept such payload, it is not a valid one according to Datadog Rest api docs (which states it should be an array, not a single object).
At the same time, Datadog OTEL receiver expects an array and fails to unmarshall those test checks.
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/ea52fa64e3cc07cb57d358762ccf0d146f3461a2/receiver/datadogreceiver/receiver.go#L410-L412
Some custom logs:
2025-10-18T19:53:33.514Z error [email protected]/receiver.go:418 Failed to unmarshal check_run payload {"resource": {"service.instance.id": "55c1e475-8c83-447f-affd-c9a7e3405725", "service.name": "otelcontribcol", "service.version": "0.137.0-custom"}, "otelcol.component.id": "datadog", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "json: cannot unmarshal object into Go value of type []translator.ServiceCheck", "payload": "{\"check\": \"test\", \"status\": 0}"}
While it doesn't cause any issues with real service checks (it looks like the agent sends it even if the endpoint check fails ), it makes the agent falsely report issues and generate a lot of errors on the collector side.
It is up to the Datadog team to decide which side to adjust: the agent payload or OTEL receiver. I am happy to bring pr in both cases if needed. But since the agent payload doesn't follow official API docs, I decided to open an issue in this repo first.
Agent Environment
K8s - 1.33
Agent - 7.71.1
OTEL collector - 0.137.0
Describe what happened:
The agent uses an invalid payload to test the check run endpoint, which the collector can't process.
Describe what you expected:
The agent uses a valid payload, or the collector can handle the current implementation of the check.
Steps to reproduce the issue:
- Deploy Datadog agent to k8s cluster
- Deploy OTEL collector with the enabled Datadog receiver.
- Override DD_URL to point to collector svc.
- Execute
agent diagnose/ check agent status/ check collector logs for unmarshall issue.