Support for vLLM Data parallel by shmuelk · Pull Request #1663 · kubernetes-sigs/gateway-api-inference-extension

shmuelk · 2025-09-28T11:09:51Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR adds support for the vLLM Data Parallel feature. The vLLM Data Parallel feature causes the vLLM "launcher" to launch many vLLM instances in the same Pod, each listening on a different port.

The InferencePool CRD has already been changed to support this by allowing up to eight TargetPorts to be specified. It is assumed that all pods in the InferencePool have been configured the same way WRT Data Parallelism.

In an attempt to minimize the amount of changes to the code, the datastore has been modified to create "virtual pods" from the real pods that are found by the pod reconciler. These virtual pods are all given names that are the real pod's name concatenated with the string "-rank-N", where N is a number from zero to seven. The term rank was used as that's what each of the separate vLLM "servers" in a Data Parallel configuration are called.

The former code has a notion of a globally known ports for inference and metrics scraping. This has been eliminated and instead inference port and metrics port fields have been added to the PodInfo struct. In addition a field was added, PodName, that contains the name of the real pod used to create the "virtual pods".

Lastly the API of the PreRequest extension point has been changed, removing the inference port parameter. Any PreRequest extensions must get the inference port of the pod(s) in question from the PodInfo's GetPort() API.

Which issue(s) this PR fixes:
Fixes #1519

Does this PR introduce a user-facing change?:

netlify · 2025-09-28T11:09:58Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`cbcb07e`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68f0c20a3f4f7900086e4740
😎 Deploy Preview	https://deploy-preview-1663--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

danehans · 2025-10-02T21:22:23Z

In an attempt to minimize the amount of changes to the code, the datastore has been modified to create "virtual pods" from the real pods that are found by the pod reconciler.

Pods are comprised of one or more containers. Will each vLLM engine instance run as a separate container in the vLLM pod?

danehans · 2025-10-03T17:07:00Z

-	if p.ModelServerMetricsPort == 0 {
-		p.ModelServerMetricsPort = targetPortNumber
-	}


I see that GetMetricsPort() does not implement this default behavior, which makes sense now that targetPortNumber is a list. We need to note this as a breaking change in the PR description.

There is no breaking change. See the code in pkg/epp/datastore/datastore.go lines 242-244. If there is only one targetPort in the InferencePool and the ModelServerMetricsPort from the command line is not zero it will be used to fill the metricsPort in the PodInfo struct. The function GetMetricsPort() simply returns what was placed in the struct earlier.

danehans · 2025-10-04T00:19:00Z

@shmuelk please create a tracker issue for adding a conformance test that includes multiple InferencePool targetPorts.

shmuelk · 2025-10-05T11:54:07Z

In an attempt to minimize the amount of changes to the code, the datastore has been modified to create "virtual pods"
from the real pods that are found by the pod reconciler.

Pods are comprised of one or more containers. Will each vLLM engine instance run as a separate container in the vLLM pod?

They are run as separate processes in the same container as far as I know. What launches Data parallel in a real vLLM is a parameter to vLLM. I don't think that can add containers to the pod.

kfswain · 2025-10-15T17:41:28Z

 	}
 }

+func (ds *datastore) PodRemove(podName string) {


Can we add this to the PodDelete function? It should be the same behavior it was for one engine per pod

kfswain · 2025-10-15T21:39:34Z

-	}
-	return fmt.Sprintf("%s://%s:%d%s", p.ModelServerMetricsScheme, pod.Address, p.ModelServerMetricsPort, p.ModelServerMetricsPath)
+func (p *PodMetricsClientImpl) getMetricEndpoint(pod *backend.Pod) string {
+	return fmt.Sprintf("%s://%s:%d%s", p.ModelServerMetricsScheme, pod.GetIPAddress(), pod.GetMetricsPort(), p.ModelServerMetricsPath)


Just noticed this, but since this is called in FetchMetrics, this is a highly trafficked function, can we just cache this endpoint string instead of calling Sprintf? I don't really see a case in which this is mutated within the pod/endpoint lifecycle.

Understood that it was this way before this PR, but would be a nice cleanup

PodInfo now has the metrics pre-built. This code is now a simple concatenation of strings.

kfswain · 2025-10-15T21:42:18Z

-		return fmt.Errorf("expected 1 target port, got %d", len(pool.Spec.TargetPorts))
-	}
-	updated, err := pm.pmc.FetchMetrics(ctx, pm.GetPod(), pm.GetMetrics(), int32(pool.Spec.TargetPorts[0].Number))
+	updated, err := pm.pmc.FetchMetrics(ctx, pm.GetPod(), pm.GetMetrics())


this is much cleaner, I like.

kfswain · 2025-10-15T21:49:44Z

 // PodInfo represents the relevant Kubernetes Pod state of an inference server.
 type PodInfo struct {
 	NamespacedName types.NamespacedName
+	PodName        string


We have an 'Endpoint` struct also defined in this package, would it make more sense to define that as the object that the rest of the system uses?

Endpoint is actually an interface. It is more related to the data gathering done in the Datalayer.

I think a refactoring all over the place from Pod to Endpoint, is not something to be done in this PR. I also think this needs to be done in phases due to the size of the effort. Some potential steps:

Rename the current Endpoint interface to something like EndpointMetrics

Replace all over the place backendmetrics.PodMetrics with datalayer.EndpointMetrics. In reality backendmetrics.PodMetrics is simply a reference to datalayer.EndPoint, which is somewhat confusing.

Appropriately rename the pod related structs, functions, and variables to Endpoint related names.

SGTM. Can we open an issue to track this work?

In reality backendmetrics.PodMetrics is simply a reference to datalayer.EndPoint, which is somewhat confusing.

Agreed, we should get rid of excess layers where we can.

kfswain · 2025-10-15T21:51:58Z

-	pm.UpdatePod(pod)
-	return ok
+
+	pods := []*datalayer.PodInfo{}


Yeah, i think the use of the nomenclature pods when referring to a specific rank/engine is making this PR more difficult to read, and subsequently, I think that will impact the codebase. Suggest using Endpoint

See my comment above

kfswain · 2025-10-15T21:57:19Z

+				Labels:      labels,
+			})
+	}
+	if len(pods) == 1 && ds.modelServerMetricsPort != 0 {


nit: we can check the length of ds.pool.Spec.TargetPorts before iteration and assembling the endpoint list

kfswain · 2025-10-15T22:07:29Z

 	NamespacedName types.NamespacedName
+	PodName        string
 	Address        string
+	Port           int32


Wondering if we want to make this a string (would prevent Itoa calls), do we ever want to use this like a number?

Signed-off-by: Shmuel Kallner <[email protected]>

kfswain · 2025-10-16T20:42:19Z

/lgtm
/approve
/hold

Looks good to me, just holding to give others a chance to review if they want. If we don't hear anything after an hr or two we can unhold and keep moving

k8s-ci-robot · 2025-10-16T20:42:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, shmuelk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kfswain · 2025-10-17T16:51:42Z

/unhold

* Removed global inference port from Prerequest extension API Signed-off-by: Shmuel Kallner <[email protected]> * Inference port and metrics port now per pod Signed-off-by: Shmuel Kallner <[email protected]> * Differentiate between real pod delete and virtual pod delete Signed-off-by: Shmuel Kallner <[email protected]> * Pass default metrics port to datastore Signed-off-by: Shmuel Kallner <[email protected]> * Updates to reflect newer APIs Signed-off-by: Shmuel Kallner <[email protected]> * Updates to tests Signed-off-by: Shmuel Kallner <[email protected]> * Fail tests that have errors, don't just log the errors Signed-off-by: Shmuel Kallner <[email protected]> * Remove tests that are no longer applicable Signed-off-by: Shmuel Kallner <[email protected]> * Set an InferencePool into the datastore Signed-off-by: Shmuel Kallner <[email protected]> * Added tests with multiple TargetPorts Signed-off-by: Shmuel Kallner <[email protected]> * Fix lint issues Signed-off-by: Shmuel Kallner <[email protected]> * Updated a new test due to updated interface Signed-off-by: Shmuel Kallner <[email protected]> * Store inference port and metrics host as strings Signed-off-by: Shmuel Kallner <[email protected]> * Concatenate metrics URL parts together without fmt.Sprintf Signed-off-by: Shmuel Kallner <[email protected]> * Use already stored metrics host Signed-off-by: Shmuel Kallner <[email protected]> * No need to convert inference port to a string Signed-off-by: Shmuel Kallner <[email protected]> * Updates due to PodInfo changes Signed-off-by: Shmuel Kallner <[email protected]> * Test updates due to PodInfo changes Signed-off-by: Shmuel Kallner <[email protected]> * Merged PodRemove into PodDelete Signed-off-by: Shmuel Kallner <[email protected]> * Changes due to merging of PodRemove into PodDelete Signed-off-by: Shmuel Kallner <[email protected]> * Test changes due to merging of PodRemove into PodDelete Signed-off-by: Shmuel Kallner <[email protected]> --------- Signed-off-by: Shmuel Kallner <[email protected]>

…-extension#1663) * Removed global inference port from Prerequest extension API Signed-off-by: Shmuel Kallner <[email protected]> * Inference port and metrics port now per pod Signed-off-by: Shmuel Kallner <[email protected]> * Differentiate between real pod delete and virtual pod delete Signed-off-by: Shmuel Kallner <[email protected]> * Pass default metrics port to datastore Signed-off-by: Shmuel Kallner <[email protected]> * Updates to reflect newer APIs Signed-off-by: Shmuel Kallner <[email protected]> * Updates to tests Signed-off-by: Shmuel Kallner <[email protected]> * Fail tests that have errors, don't just log the errors Signed-off-by: Shmuel Kallner <[email protected]> * Remove tests that are no longer applicable Signed-off-by: Shmuel Kallner <[email protected]> * Set an InferencePool into the datastore Signed-off-by: Shmuel Kallner <[email protected]> * Added tests with multiple TargetPorts Signed-off-by: Shmuel Kallner <[email protected]> * Fix lint issues Signed-off-by: Shmuel Kallner <[email protected]> * Updated a new test due to updated interface Signed-off-by: Shmuel Kallner <[email protected]> * Store inference port and metrics host as strings Signed-off-by: Shmuel Kallner <[email protected]> * Concatenate metrics URL parts together without fmt.Sprintf Signed-off-by: Shmuel Kallner <[email protected]> * Use already stored metrics host Signed-off-by: Shmuel Kallner <[email protected]> * No need to convert inference port to a string Signed-off-by: Shmuel Kallner <[email protected]> * Updates due to PodInfo changes Signed-off-by: Shmuel Kallner <[email protected]> * Test updates due to PodInfo changes Signed-off-by: Shmuel Kallner <[email protected]> * Merged PodRemove into PodDelete Signed-off-by: Shmuel Kallner <[email protected]> * Changes due to merging of PodRemove into PodDelete Signed-off-by: Shmuel Kallner <[email protected]> * Test changes due to merging of PodRemove into PodDelete Signed-off-by: Shmuel Kallner <[email protected]> --------- Signed-off-by: Shmuel Kallner <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Sep 28, 2025

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 28, 2025

k8s-ci-robot requested review from danehans and nirrozenbaum September 28, 2025 11:09

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 28, 2025

danehans reviewed Oct 3, 2025

View reviewed changes

shmuelk mentioned this pull request Oct 5, 2025

Conformance: Add a test for Gateway support of multiple targetPorts in an InferencePool #1680

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 14, 2025

shmuelk force-pushed the data-parallel branch from fec83a2 to 885d81c Compare October 15, 2025 16:50

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 15, 2025

shmuelk changed the title ~~WIP: Support for vLLM Data parallel~~ Support for vLLM Data parallel Oct 15, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 15, 2025

kfswain reviewed Oct 15, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 15, 2025

shmuelk added 9 commits October 16, 2025 08:48

Removed global inference port from Prerequest extension API

608959d

Signed-off-by: Shmuel Kallner <[email protected]>

Inference port and metrics port now per pod

e4f1f3a

Signed-off-by: Shmuel Kallner <[email protected]>

Differentiate between real pod delete and virtual pod delete

cd8a700

Signed-off-by: Shmuel Kallner <[email protected]>

Pass default metrics port to datastore

ae825f1

Signed-off-by: Shmuel Kallner <[email protected]>

Updates to reflect newer APIs

985e414

Signed-off-by: Shmuel Kallner <[email protected]>

Updates to tests

b5bd9b3

Signed-off-by: Shmuel Kallner <[email protected]>

Fail tests that have errors, don't just log the errors

a422cb9

Signed-off-by: Shmuel Kallner <[email protected]>

Remove tests that are no longer applicable

8ed09b5

Signed-off-by: Shmuel Kallner <[email protected]>

Set an InferencePool into the datastore

b84b772

Signed-off-by: Shmuel Kallner <[email protected]>

shmuelk added 9 commits October 16, 2025 10:37

Store inference port and metrics host as strings

44b5b97

Signed-off-by: Shmuel Kallner <[email protected]>

Concatenate metrics URL parts together without fmt.Sprintf

70df521

Signed-off-by: Shmuel Kallner <[email protected]>

Use already stored metrics host

c66104b

Signed-off-by: Shmuel Kallner <[email protected]>

No need to convert inference port to a string

9dbec2e

Signed-off-by: Shmuel Kallner <[email protected]>

Updates due to PodInfo changes

4e4ca48

Signed-off-by: Shmuel Kallner <[email protected]>

Test updates due to PodInfo changes

221dd4e

Signed-off-by: Shmuel Kallner <[email protected]>

Merged PodRemove into PodDelete

ba954de

Signed-off-by: Shmuel Kallner <[email protected]>

Changes due to merging of PodRemove into PodDelete

36697c4

Signed-off-by: Shmuel Kallner <[email protected]>

Test changes due to merging of PodRemove into PodDelete

cbcb07e

Signed-off-by: Shmuel Kallner <[email protected]>

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 16, 2025

k8s-ci-robot assigned kfswain Oct 16, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 16, 2025

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 17, 2025

k8s-ci-robot merged commit 51dff7b into kubernetes-sigs:main Oct 17, 2025
11 checks passed

syw14 mentioned this pull request Oct 17, 2025

Remove validation for epp to have multi-port #1738

Closed

danehans mentioned this pull request Oct 23, 2025

DP: EPP Not Load Balancing chat/completions Requests #1767

Closed

kfswain mentioned this pull request Oct 23, 2025

Update E2E tests to include multiport case #1768

Closed

RyanRosario mentioned this pull request Nov 14, 2025

[WIP][Reference] E2E test setup for multiport case #1864

Closed

shmuelk mentioned this pull request Nov 16, 2025

Rename PodInfo struct to EndpointMetadata to better reflect its purpose #1866

Merged

shmuelk deleted the data-parallel branch November 16, 2025 14:18

kfswain mentioned this pull request Nov 18, 2025

Enable EPP to support endpoint discovery using pod selector #1833

Merged

delavet mentioned this pull request Dec 8, 2025

Feature Request: Allow EPP to dynamically discover ports based on matching Pods #1965

Closed

Conversation

shmuelk commented Sep 28, 2025

Uh oh!

netlify Bot commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

danehans commented Oct 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans commented Oct 4, 2025

Uh oh!

shmuelk commented Oct 5, 2025

Uh oh!

kfswain Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain commented Oct 16, 2025

Uh oh!

k8s-ci-robot commented Oct 16, 2025

Uh oh!

kfswain commented Oct 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented Sep 28, 2025 •

edited

Loading

kfswain Oct 15, 2025 •

edited

Loading