InferencePool port specification (e.g., serving, metrics and health)

[elevran]

Currently the InferencePool specifies (the set of) Ports exposed by the selected inference Pods (e.g., #1336). These are assumed to be the model serving ports.
vLLM uses the same port for model serving, metrics and health - there's no native way to override or specify different ports for each function. However, IGW allows specifying a metrics port separately via a command line option.

1. is overriding metrics port only consistent with a specific use case or feature?
2. if so, should inferencepool allow separating the port(s) by role (serving, metrics, health)? Once could still default to the current Ports specification if no per-role specification is provided.

/cc @robscott @smarterclayton

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elevran]

/remove-lifecycle stale

See #1398 for additional comments regarding port override in CLI.

[janghyukjin]

@elevran Returning to this discussion with a concrete use case I encountered.

In the current implementation, EPP treats each port in the InferencePool as a distinct endpoint and makes scheduling decisions based on per-endpoint metrics. However, there is currently no way to express different port roles (e.g. serving, metrics, grpc) in InferencePool.spec.

This came up while working on a use case where metrics are exposed on a separate port — for example, in Istio mTLS STRICT environments where EPP's direct pod-IP scraping bypasses Service-level mTLS and the receiving sidecar rejects it with 503. (See #2762 for more context.)

Supporting port groups or roles at the InferencePool level (serving, metrics, grpc) would provide a more consistent and extensible solution across the system.

Happy to help drive this discussion or contribute to a design / implementation.

[elevran]

/cc @ahg-g @kfswain @danehans @nirrozenbaum
Think we can reopen and revisit the issue?
How do you recommend we proceed?

[janghyukjin]

Gently bumping this @elevran — would love to see this reopened. Given the concrete use case around Istio mTLS STRICT environments (and likely other scenarios where metrics/serving ports differ), I think this is worth revisiting.
Happy to contribute to a design doc or prototype implementation if the direction gets clarified. 🙏

[elevran]

@janghyukjin I don't have permission to reopen issues here. I would separate the discussion in two

• any changes to the inference pool should be discussed in this repo
• if there's a good way to support without pool spec changes, please open an issue on llm-d-inference-scheduler to discuss alternatives.