SLO Aware Routing Sidecar + Plugin EPP Integration and Helm Deployment by BenjaminBraunDev · Pull Request #1839 · kubernetes-sigs/gateway-api-inference-extension

BenjaminBraunDev · 2025-11-07T23:56:26Z

This PR is stage 3/3 for adding in the latency prediction and SLO-Aware Routing functionality to EPP.

New Features:

New -enable-latency-predictor flag in EPP arg to inform it that sidecars are present and to register slo routing plugins.
Requests now support latency SLOs in their header (x-slo-ttft-ms and x-slo-tpot-ms) and a boolean for whether to use the SLO routing scheduling profile with slo scoring (x-prediction-based-scheduling). If false, use the default profile and just track and train for future requests.

Plugins

Registers and deploys the plugins added in #1849 via scheduling profiles:

The requestcontrol plugins that track requests and their latencies and sends data back to training sidecar for processing.
A Scorer plugin based on latencies and headroom.

PodMetrics

Adds (back) the totalRunningRequestsMetric prometheus metric from vLLM, which was removed for being unused in the past, but is now a feature of our latency prediction model.

Guide

Added a guide for how to deploy IGW with SLO-Aware Routing in site-src/guides/slo-aware-routing.md

Fixes #1323

Does this PR introduce a user-facing change?:

Yes:
EPP has a new runtime/deployment argument: `-enable-latency-predictor`

When enabled EPP, will register and support the new SLO routing plugins and sidecars. With this flag, EPP will assume the predictor and training sidecars also exist in its same container.

When latency prediction is enabled for EPP, requests support 3 new headers:
- `x-slo-ttft-ms` is a float for the desired "Time To First Token" SLO
- `x-slo-tpot-ms` is a float for the desired average "Time Per Output Token" SLO
- `x-prediction-based-scheduling` is a boolean for whether to route this request with the default scheduling profile or with the slo-aware scheduling profile

netlify · 2025-11-07T23:56:33Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`5b853dd`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/692648f52a9f400007963b13
😎 Deploy Preview	https://deploy-preview-1839--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-11-07T23:56:36Z

Hi @BenjaminBraunDev. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ahg-g · 2025-11-14T22:19:26Z

/ok-to-test

BenjaminBraunDev · 2025-11-15T01:22:10Z

@ahg-g Due to it's large nature, Kellen requested we split this PR up. Here is the first one: #1849

ahg-g

it would be great if you can send a separate PR for adding the running requests metric

ahg-g · 2025-11-22T01:09:38Z

@kaushikmitr @BenjaminBraunDev this is predicted latency, not slo, right? if so, please use predicted-latency-scorer as the name of the plugin.

…diciton id off

…, add predictor to new 2 phase configuration parser

…n, running routines there, move predictor helm section into new tpl file, rename slo-aware-routing guide and names in docs

…cars not fail immediatly during EPP spinup

BenjaminBraunDev · 2025-11-26T00:04:42Z

Here's the issue regarding the HandleResponseBodyModelStreaming weirdness: #1903

kfswain · 2025-11-26T00:05:54Z

/approve

k8s-ci-robot · 2025-11-26T00:06:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenjaminBraunDev, kfswain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

BenjaminBraunDev · 2025-11-26T00:26:29Z

@kfswain Sorry, forgot to add a small bugfix to the scorer plugin, could you re-approve?

kfswain · 2025-11-26T00:34:37Z

Np, approval remains. It's the LGTM that's removed. I'll leave that to @kaushikmitr to stamp

kfswain · 2025-11-26T00:36:42Z

Sry, I wont make you chase down a person again, was just trying to let other people be involved is all.

/lgtm

BenjaminBraunDev · 2025-11-26T00:43:26Z

Thanks!

kubernetes-sigs/gateway-api-inference-extension#1839) * Add latency predictor plugins, deployment, and runner.go integration * Update dockerfile, fix issues with SLO context not being set when prediciton id off * Remove outdated inferencepool-resources deployment * Fix streamed request being called one final time after request complete, add predictor check to the beginning of each requestcontrol hook * add guide, update helm charts and readme, minor scorer changes * Make small guide update * Add helm values and polish README and SLO routing guide * Clean up errors from rebase, add running request metric to datasource, add predictor to new 2 phase configuration parser * Fix epp image and add placeholder docker repos for latency sidecars * Update guide, README, and values.yaml * Moved predictor setup logic into plugin * Move predictor startup login completely out of manager and into plugin, running routines there, move predictor helm section into new tpl file, rename slo-aware-routing guide and names in docs * Remove max-score-picker from list of plugin types in helm chart * Fix formatting * Revert go.mod to main * Fix typo in config, remove depreicated runtime flag * Rename latency prediction plugins, change docs accordingly, make sidecars not fail immediatly during EPP spinup * Update docs with new total running requests metric * Small plugin bugfix

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 7, 2025

k8s-ci-robot requested review from ahg-g and elevran November 7, 2025 23:56

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 7, 2025

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 7, 2025

This was referenced Nov 8, 2025

[Feat] SLO-Aware Routing with Latency Prediction #1323

Closed

SLO Aware Routing Plugins Only #1849

Merged

BenjaminBraunDev force-pushed the slo-aware-routing-stage-3 branch from 2c56616 to f63bc01 Compare November 20, 2025 21:42

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 20, 2025

BenjaminBraunDev changed the title ~~SLO Aware Routing Plugins and Metrics~~ SLO Aware Routing Sidecar + Plugin EPP Integration and Helm Deployment Nov 20, 2025

kaushikmitr reviewed Nov 20, 2025

View reviewed changes

Comment thread config/charts/inferencepool/values.yaml Outdated

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 20, 2025

BenjaminBraunDev force-pushed the slo-aware-routing-stage-3 branch from d177545 to ddee4c7 Compare November 21, 2025 01:12

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 21, 2025

kfswain reviewed Nov 21, 2025

View reviewed changes

ahg-g reviewed Nov 21, 2025

View reviewed changes

ahg-g mentioned this pull request Nov 25, 2025

feat: Add totalRunningRequests metric for latency predictor #1899

Merged

BenjaminBraunDev added 3 commits November 25, 2025 23:00

Add latency predictor plugins, deployment, and runner.go integration

ee4a8c3

Update dockerfile, fix issues with SLO context not being set when pre…

cb88003

…diciton id off

Remove outdated inferencepool-resources deployment

42754f5

BenjaminBraunDev added 12 commits November 25, 2025 23:00

Make small guide update

32aec98

Add helm values and polish README and SLO routing guide

d7fd091

Clean up errors from rebase, add running request metric to datasource…

07f7430

…, add predictor to new 2 phase configuration parser

Fix epp image and add placeholder docker repos for latency sidecars

1e57cf6

Update guide, README, and values.yaml

979ecb9

Moved predictor setup logic into plugin

6c73dd2

Move predictor startup login completely out of manager and into plugi…

6dc8b02

…n, running routines there, move predictor helm section into new tpl file, rename slo-aware-routing guide and names in docs

Remove max-score-picker from list of plugin types in helm chart

e4b8eec

Fix formatting

415d93f

Revert go.mod to main

e0bf3b0

Fix typo in config, remove depreicated runtime flag

5844d8f

Rename latency prediction plugins, change docs accordingly, make side…

1eb5d8a

…cars not fail immediatly during EPP spinup

BenjaminBraunDev force-pushed the slo-aware-routing-stage-3 branch from b94d598 to 1eb5d8a Compare November 25, 2025 23:15

Update docs with new total running requests metric

e0afb89

BenjaminBraunDev mentioned this pull request Nov 26, 2025

HandleResponseBodyModelStreaming is called one additional time after streamingEndMsg is received by the server. #1903

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 26, 2025

Small plugin bugfix

5b853dd

k8s-ci-robot assigned kfswain Nov 26, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 26, 2025

k8s-ci-robot merged commit a51e074 into kubernetes-sigs:main Nov 26, 2025
12 checks passed

BenjaminBraunDev deleted the slo-aware-routing-stage-3 branch November 26, 2025 00:49

zdtsw mentioned this pull request Nov 26, 2025

[WIP] SLO Aware Routing support llm-d/llm-d-inference-scheduler#439

Closed

Conversation

BenjaminBraunDev commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Features:

Plugins

PodMetrics

Guide

Uh oh!

netlify Bot commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Nov 7, 2025

Uh oh!

ahg-g commented Nov 14, 2025

Uh oh!

BenjaminBraunDev commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahg-g commented Nov 22, 2025

Uh oh!

BenjaminBraunDev commented Nov 26, 2025

Uh oh!

kfswain commented Nov 26, 2025

Uh oh!

k8s-ci-robot commented Nov 26, 2025

Uh oh!

BenjaminBraunDev commented Nov 26, 2025

Uh oh!

kfswain commented Nov 26, 2025

Uh oh!

kfswain commented Nov 26, 2025

Uh oh!

BenjaminBraunDev commented Nov 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BenjaminBraunDev commented Nov 7, 2025 •

edited

Loading

netlify Bot commented Nov 7, 2025 •

edited

Loading

BenjaminBraunDev commented Nov 15, 2025 •

edited

Loading