Skip to content

[BUG] Datadog agent uses invalid payload to check /api/v1/check_run endpoint #42124

@tr3mor

Description

@tr3mor

Datadog agent is currently using the following payload to check /api/v1/check_run endpoint

checkRunPayload := []byte("{\"check\": \"test\", \"status\": 0}")

While the Datadog backend seems to accept such payload, it is not a valid one according to Datadog Rest api docs (which states it should be an array, not a single object).

At the same time, Datadog OTEL receiver expects an array and fails to unmarshall those test checks.
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/ea52fa64e3cc07cb57d358762ccf0d146f3461a2/receiver/datadogreceiver/receiver.go#L410-L412

Some custom logs:
2025-10-18T19:53:33.514Z error [email protected]/receiver.go:418 Failed to unmarshal check_run payload {"resource": {"service.instance.id": "55c1e475-8c83-447f-affd-c9a7e3405725", "service.name": "otelcontribcol", "service.version": "0.137.0-custom"}, "otelcol.component.id": "datadog", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "json: cannot unmarshal object into Go value of type []translator.ServiceCheck", "payload": "{\"check\": \"test\", \"status\": 0}"}

While it doesn't cause any issues with real service checks (it looks like the agent sends it even if the endpoint check fails ), it makes the agent falsely report issues and generate a lot of errors on the collector side.

It is up to the Datadog team to decide which side to adjust: the agent payload or OTEL receiver. I am happy to bring pr in both cases if needed. But since the agent payload doesn't follow official API docs, I decided to open an issue in this repo first.

Agent Environment
K8s - 1.33
Agent - 7.71.1
OTEL collector - 0.137.0

Describe what happened:
The agent uses an invalid payload to test the check run endpoint, which the collector can't process.

Describe what you expected:
The agent uses a valid payload, or the collector can handle the current implementation of the check.

Steps to reproduce the issue:

  1. Deploy Datadog agent to k8s cluster
  2. Deploy OTEL collector with the enabled Datadog receiver.
  3. Override DD_URL to point to collector svc.
  4. Execute agent diagnose / check agent status/ check collector logs for unmarshall issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    pendingLabel for issues waiting a Datadog member's response.team/triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions