Enable Opentelemetry protocol for metrics

[vigith]

Discussed in #3035

^{Originally posted by cdangelo-kline October 22, 2025}
Dear all,
Currently, Numaflow supports integration with Prometheus using an Operator for scraping metrics.
To make the system more vendor-neutral, Numaflow could consider supporting the OpenTelemetry protocol. This would allow it to expose metrics via both Prometheus scraping and OTLP.
As a reference, this could be implemented similarly to Argo Workflows' OTLP integration and Argo Workflow Prometheus Scraping.
Does this approach make sense?

[juniemariam]

Hi! I’m new here and would love to work on this issue if it’s available.

[vigith]

I have assigned this to you

[juniemariam]

Hi everyone,
I’ve been exploring how Numaflow exposes metrics through Prometheus and wanted to check if the following direction makes sense before starting a PR.

Summary of what I’m thinking:

1. Keep Prometheus scraping as-is (controller on :9090, daemons on their existing ports).
2. Add OTLP exporters (enabled when OTEL_EXPORTER_OTLP_ENDPOINT is set).
3. Start with the controller-manager, then extend to daemon servers, UI backend, and Rust serving components in phases.
4. Add an otlp: block in the controller/server ConfigMaps for endpoint, protocol (grpc/http), TLS, and temporality settings.
5. Update Helm/Kustomize overlays to propagate these settings as environment variables.
6. Keep Prometheus the default; OTLP is opt-in.
7. Document how to enable the dual-exporter setup (similar to Argo Workflows).

Questions:
Does this plan match the direction you were thinking? And are there any existing design or telemetry guidelines I should follow before starting?
Note: I plan to start with controller-only OTLP support first and expand later if that makes sense.

Thanks!

[whynowy]

@juniemariam - thanks for looking into it.

Agreed that we should keep Prometheus scraping as-is.

I would suggest starting from the daemon servers for the exploration. Unfortunately, controller-runtime (which controller-manager depends on) does not support otel at this moment, we need to how that goes.