Support changing InferencePool port at runtime?

[elevran]

Currently we allow setting the metrics fetching client Port only once:

• initially set to the command line value (in runner.go)
• if not specified (i.e., it is zero), it is overridden by the pool.Spec.Ports[0].Number (see here)

This means that changes in the InferencePool port specification are not supported. However, changing the Selector is supported.
It is not clear (to me) if this is intentional and desirable behavior or a bug.
If we do allow changing the port based on the Pool.Spec, we should define how it should interact with the command line value.

Side note: #1396... Would be good to understand the use case of setting the metrics port from the command line and not via the Spec.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elevran]

/remove-lifecycle stale

@kfswain @ahg-g @nirrozenbaum - with the introduction of multiple ports in the inference pool, the use of a single metrics port override seems wrong.
Unclear what customer use case would allow setting the metrics port for all endpoints to a single value while supporting inference serving over multiple ports.

Do you know the original case (and whether it's still valid) for introducing a metrics port?
If it's longer valid, I'd suggets we deprecate it (now) and remove the option to set the port via CLI (in the next release).