[Flow Control] Observability (Prom Metrics, not Tracing)

[LukeAVanDrie]

What would you like to be added: Add prometheus metrics for the experimental Flow Control layer. This will provide essential visibility into the performance and behavior of this layer.

Some suggested metrics include:

• WIP: will add shortly in the comment thread.

Why is this needed: Operability and observability. These metrics may provide several additional benefits:

• Backpressure Signals: The metrics can be used as a context-rich source for backpressure signals.
• Enhanced Autoscaling: The metrics will enable more intelligent autoscaling decisions.
• Custom L7LB Balancing: The metrics can be used to inform custom routing decisions in a Layer 7 load balancer across EPP replicas.

[LukeAVanDrie]

/assign

[LukeAVanDrie]

/cc @JeffLuoo

[ahg-g]

/milestone Flow Control

[RyanRosario]

Per discussion with Luke, I can take this one!

[LukeAVanDrie]

Let's start with expanding our queue depth metrics to also include enqueued prompt bytes (not just request count). This is actually valuable for routine monitoring.

Then, let's look into some flow control instrumentation for latency (mainly for debugging purposes):

• dispatch cycle latency (most important)
• enqueue path latency (less important)
• and then within the dispatch cycle, capture latencies per "plugin": Interflow, Intraflow

Later, we can also look at some saturation metrics, but this is tricker.

[RyanRosario]

Some references

Example metric:
https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/497f376b201acd76a62330da6f6e42534cae453f/pkg/epp/metrics/metrics.go#L313C2-L323C3

Example latency metric:

gateway-api-inference-extension/pkg/epp/metrics/metrics.go

Line 290 in 497f376

SchedulerE2ELatency = prometheus.NewHistogramVec(

Additional metadata shipped with metrics:
#2010

Additional considerations:

• Start with e2e dispatch cycle latency / e2e enqueue path latency (see sample code for e2e latency from request perspective)
• Queue length in bytes (already have length in requests)

[LukeAVanDrie]

/assign @RyanRosario