Refactor: Prepare EPP SaturationDetection as an Extension Point

[LukeAVanDrie]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
This PR refactors the directory structure for the saturation control logic within the Endpoint Picker (EPP). It moves the current implementation to framework/plugins/utilizationdetector.

This change is a preparatory step to establish Saturation Detector as a formal extension point within the EPP, allowing
for multiple implementations. The existing logic is now housed as the utilizationdetector plugin, making way
for future additions like concurrencydetector.

This PR contains only file moves and renames; no functional changes are introduced. Note: it is not aligned with the
EPP plugin system yet. This is a preparatory step only.

Which issue(s) this PR fixes:
Tracks #1405 ('cc @nirrozenbaum) and #1793

Does this PR introduce a user-facing change?:

NONE

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	6f17d94
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/694488b05fae89000859c4d4
😎 Deploy Preview	https://deploy-preview-1976--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

Hi @LukeAVanDrie. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ahg-g]

/ok-to-test

[nirrozenbaum]

/approve

FWIW, I think we should draw on board (or paper :) ) the updated design with the updated extension points and make sure we're aligned on how we think EPP should end up.
This is not a comment about this PR but more of a general comment, mainly because there are multiple different threads that push changes in parallel into our pluggability story (cc: @kfswain @ahg-g).

will stamp with lgtm once the CI tests pass (should fix boilerplate headers).

[kfswain]

/approve

FWIW, I think we should draw on board (or paper :) ) the updated design with the updated extension points and make sure we're aligned on how we think EPP should end up. This is not a comment about this PR but more of a general comment, mainly because there are multiple different threads that push changes in parallel into our pluggability story (cc: @kfswain @ahg-g).

will stamp with lgtm once the CI tests pass (should fix boilerplate headers).

I agree with this. There is another PR that is proposing to fundamentally change how plugins can operate, and I'm worried about the precedent that sets.

[nirrozenbaum]

I agree with this. There is another PR that is proposing to fundamentally change how plugins can operate, and I'm worried about the precedent that sets.

@kfswain not sure which other PR is the one you mentioned. from my point of view this PR can be stamped (it's just putting the existing saturation detector under a different directory).

leaving to Kellen the final stamp just to be on the safe side.

[LukeAVanDrie]

@kfswain not sure which other PR is the one you mentioned. from my point of view this PR can be stamped (it's just putting the existing saturation detector under a different directory).

He's referring to my other PR: #1977. I am working on the configuration and extensibility story for Flow Control. Flow Control has two potential extension points: interflow (fairness) and intraflow (ordering) policies. These, however, are scoped to priority bands or flows respectively and not well-supported by the singleton plugin model in their current state. I have a draft to support transient lifecycle for stateful, scoped plugin instances.

Shmuel recommended a stateless singleton approach relying on state-passing which I am now exploring instead as it is more closely aligned with our existing plugin model and less easy to abuse.

---

FWIW, I think we should draw on board (or paper :) ) the updated design with the updated extension points and make sure we're aligned on how we think EPP should end up.

This will be a good exercise. I am currently working toward adding (or rather promoting) the following extension points for our Flow Control story:

• InterFlowDispatchPolicy (exposing SelectQueue): the fairness policy for picking which flow gets the next dispatch opportunity
• IntraFlowDispatchPolicy (exposing Compare): the temporal scheduling policy that is responsible for defining the comparator used to sort the requests within a flow's queue
• SaturationController (exposing ShouldDispatch) $\to$ formerly SaturationDetector

These then compose nicely into a flow control / scheduling regime.

• Serve requests with Virtual Token Count (VTC) fairness (InterFlowDispatchPolicy) between flows ordered by Earliest Deadline First (IntraFlowDispatchPolicy) subject to a hard concurrency limit per pod (SaturationController) for pool saturation.
• Serve requests with Round Robin fairness between flows ordered by FCFS subject to queue depth and kv-cache utilization thresholds for pool saturation.

And so on... These can all be mixed and matched.

---

I figured it was generally agreed upon to start this for saturation detection given #1405. I plan on doing exactly what's in the bug. Promoting our simple heuristic-based detector to the default implementation before adding a new in-tree concurrency-limit based controller as a followup.

[nirrozenbaum]

as a side note I must admit that I’ve read the explanation about what is InterFlowPolicy and what is IntraFlowPolicy at least 20 times in different threads, and I still find these names very confusing.

do you think we can rename it to a descriptive name- just by what it does?
e.g., FairnessDispatchPolicy, FlowOrderingPolicy, etc

[LukeAVanDrie]

as a side note I must admit that I’ve read the explanation about what is InterFlowPolicy and what is IntraFlowPolicy at least 20 times in different threads, and I still find these names very confusing.

do you think we can rename it to a descriptive name- just by what it does? e.g., FairnessDispatchPolicy, FlowOrderingPolicy, etc

Yes, these can be changed. They are not sticky at all yet. I think FairnessPolicy or FairnessStrategy is clear and to the point and will be the first type I migrate to the plugin model. When I open that PR (moving InterFlowDispatchPolicy definition to a plugins.go file and adding embedding plugins.Plugin for TypedName), we can discuss names there.

These will eventually be reflected in our configuration guide docs and such, so we should pick the best user-facing names.

---

Edit: mocking the config example makes choices more clear. Thoughts?

fairnessPolicy:
  pluginRef: virtual-token-count
orderingPolicy:
  pluginRef: earliest-deadline-first

[LukeAVanDrie]

Rebased and resolved merge conflict

[LukeAVanDrie]

/assign @nirrozenbaum

PR should be in a ready state to merge now. Thanks for the feedback!

[shmuelk]

I don't like the use of the word controller in the name here. The EPP is a K8S controller for several CRDs (and growing). I think that is the only context for use of the word controller in a name.

Is the plan for this new extension point to control saturation or just detect saturation and let other components decide what to do?

Also as the word static is used in the name here, is there an intention to a dynamic threshold plugin?

[nirrozenbaum]

++ agree with @shmuelk's comments.
why do we rename from detector to controller? that is confusing when reading this code in the context of k8s.

[LukeAVanDrie]

@shmuelk Thanks for the feedback. I agree that "Controller" is overloaded in the K8s ecosystem and we should avoid confusion with CRD reconcilers. However, I want to move away from "Detector" because the scope of this component is expanding. In the legacy implementation, it was a passive observer returning a boolean. I have some new implementations that act as a closed-loop control systems. They manage internal state and actively calculate dispatch rates or concurrency limits based on feedback variables.

Roadmap:

• Static Thresholds: (This PR) Throttles based on statically configured metric thresholds.
• Static Concurrency: Throttles based on statically configured concurrency limits per-pod.
• Adaptive/Dynamic Concurrency (exploratory/stretch goal): A feedback loop that auto-tune limits.

So, we have a component that actively manages request flow (via its process variable, "saturation") and is somewhat steeped in control theory. This is how I landed on "Controller."

To avoid overloading this term, I suggest renaming this extension point to "SaturationRegulator". This preserves the semantics distinguishing it from passive observation. The path would look like: pkg/epp/saturationregulator/framework/plugins/staticthresholdregulator/regulator.go.

[LukeAVanDrie]

@nirrozenbaum and @shmuelk

Thanks again for the feedback! I’ve updated the PR to stick with the 'Detector' terminology to align with existing conventions and your suggestions. To better reflect the specific logic of this implementation and the pluggable architecture, I've made the following changes:

1. Renamed to utilizationdetector: This maintains the 'Detector' suffix you requested while being more descriptive and less academic than the previous 'staticthreshold' proposal. It specifically detects saturation based on resource utilization (Queue/KV Cache).
2. Path Changes: I've moved the logic into: pkg/epp/saturationdetector/framework/plugins/utilizationdetector/detector.go.

My planned concurrency plugins then become: concurrencydetector and adaptiveconcurrencydetector (or similar).

I have force-pushed these changes over the previous commit. I wanted to avoid a messy history of chained git mv commands and ensure the file history for the original detector logic remains clean and easy to follow in the new path.

Hopefully this resolves your concerns here. PTAL!

[kfswain]

That sounds good to me! Thanks Luke!

[kfswain]

/approve

Will leave final stamp to others! Thanks! Stoked about this

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, LukeAVanDrie, nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [kfswain,nirrozenbaum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nirrozenbaum]

/lgtm

Thanks Luke 🙏🏼