Add Gang Scheduling Support for lws and volcano implementation example #498

JesseStutler · 2025-04-11T03:31:39Z

What type of PR is this?

/kind feature

What this PR does / why we need it

Related to #496, provide the current implementation code for Gang Scheduling and Volcano Implementation. For coscheduling scheduler plugin, I can separate it in the other PR.

Which issue(s) this PR fixes

Related #407

Special notes for your reviewer

Does this PR introduce a user-facing change?

Added gang scheduling management support through the `Configuration` API. Gang scheduling can now be enabled by setting `gangSchedulingManagement.schedulerProvider` in the controller configuration file. For Helm installations, use `--set gangSchedulingManagement.schedulerProvider=[xxx]` to enable this feature. Currently, only volcano is supported.

Validation

Create a replicas=2, size=4 leaderWorkerSet:

The pg's name is [lws name]-[group Index]-[revision] 2. Scale down to 1:

3. Scale up to 3:

4. Rolling Update: (1) Configure maxSurge=2, and then execute rolling update:

(2) Rolling update finished, the final state:

k8s-ci-robot · 2025-04-11T03:31:49Z

Hi @JesseStutler. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2025-04-11T03:31:56Z

✅ Deploy Preview for kubernetes-sigs-lws canceled.

Name	Link
🔨 Latest commit	`a2cbb68`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-lws/deploys/6890c1a6ffda5200088112b0

Edwinhr716 · 2025-04-14T15:51:57Z

pkg/utils/replicaresource/volcano_provider.go

+
+func NewVolcanoProvider(client client.Client) *VolcanoProvider {
+	return &VolcanoProvider{
+		client: client,


Missing the defaultBaseResourceProvider right? Which would be the same for every provider

Yes, you are right. I add it now.

yankay · 2025-04-15T02:35:39Z

/ok-to-test

JesseStutler · 2025-07-22T02:33:41Z

/cc @Edwinhr716 @kerthcet
Please review it thanks, I'm still following the design in #496, using the ReplicaResourceProvider interface. The CreateHeadlessService and CreateResourceClaim methods are reserved, please check whether you want to delete these two methods and keep only the SchedulerProvider interface in this PR

JesseStutler · 2025-07-22T02:35:23Z

I added a volcano implementation for you to refer to. I can also add coscheduling plugin implementation if we also need it in v0.7.0 😄

kerthcet · 2025-08-02T08:01:21Z

@JesseStutler can we finish this week? I don't want to last until next week. Thanks.

JesseStutler · 2025-08-03T15:42:47Z

@JesseStutler can we finish this week? I don't want to last until next week. Thanks.

Sorry for the late reply, I have something to do on the weekend. Can I finish it on Monday and ask you to review again if it's okay? @kerthcet

JesseStutler · 2025-08-03T16:02:25Z

@JesseStutler can we finish this week? I don't want to last until next week. Thanks.

@kerthcet You can review first about the e2e testing, I'll verify it on Monday. Is that OK?

kerthcet

Once you fix the env error in the e2e test, I'll create a new ci workflow in test-infra, so we can verify the gang scheduling tests.

test/e2e/e2e_test.go

test/testutils/util.go

test/e2e/e2e_test.go

pkg/schedulerprovider/volcano_provider.go

hack/e2e-test.sh

test/e2e/e2e_test.go

Signed-off-by: JesseStutler <[email protected]>

JesseStutler · 2025-08-04T11:16:34Z

/retest

JesseStutler · 2025-08-04T11:33:37Z

Once you fix the env error in the e2e test, I'll create a new ci workflow in test-infra, so we can verify the gang scheduling tests.

@kerthcet Now I have fixed all the CIs, please help me create a workflow, thanks. Does this require codes to merge?

kerthcet · 2025-08-04T13:13:22Z

test/testutils/util.go

+func ExpectValidPodGroups(ctx context.Context, k8sClient client.Client, provider schedulerprovider.ProviderType, lws *leaderworkerset.LeaderWorkerSet, expectedCount int) {
+	gvk := getPodGroupGVK(provider)
+	if gvk.Empty() {
+		ginkgo.Skip("Unsupported scheduler provider for PodGroup validation")


I think we should return error here? Empty gvk is not allowed I think.

OK I have changed to ginkgo.Fail, please check

kerthcet · 2025-08-04T13:13:32Z

test/testutils/util.go

+// UpdatePodGroupAtIndex manually updates the PodGroup for a specific group index
+func UpdatePodGroupAtIndex(ctx context.Context, k8sClient client.Client, providerType schedulerprovider.ProviderType, schedulerProvider schedulerprovider.SchedulerProvider, lws *leaderworkerset.LeaderWorkerSet, groupIndex string) {
+	gvk := getPodGroupGVK(providerType)
+	if gvk.Empty() {


OK I have changed to ginkgo.Fail, please check

kerthcet · 2025-08-04T13:14:00Z

test/testutils/util.go

+// ExpectValidPodGroupAtIndex checks that a PodGroup exists for the specified group index and has the correct owner reference
+func ExpectValidPodGroupAtIndex(ctx context.Context, k8sClient client.Client, provider schedulerprovider.ProviderType, lws *leaderworkerset.LeaderWorkerSet, groupIndex string) {
+	gvk := getPodGroupGVK(provider)
+	if gvk.Empty() {


OK I have changed to ginkgo.Fail, please check

kerthcet · 2025-08-04T13:34:35Z

pkg/utils/utils.go

 	return defaultNamespace
 }
+
+// CalculatePGMinResources calculates the minimum resources needed for an entire PodGroup [1 Leader + (size-1) Worker pods]


we need to have a unit test for this later, it could be a follow up PR.

OK, I'll do it in later PR, thanks for your reminder

test/e2e/e2e_test.go

Signed-off-by: JesseStutler <[email protected]>

kerthcet · 2025-08-04T14:50:33Z

/lgtm
/approve

Thanks @JesseStutler

k8s-ci-robot · 2025-08-04T14:50:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JesseStutler, kerthcet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kerthcet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ardaguclu · 2025-08-06T10:14:23Z

@JesseStutler I was expecting changes under /config directory which aligns with the changes under /charts. Is it missing or intentional?. For example, where is clusterrole changes for volcano under config directory?

ardaguclu · 2025-08-06T10:16:22Z

Currently, I don't think gang scheduler is supported via kustomize, unless user manually applies required configurations.

JesseStutler · 2025-08-06T13:10:44Z

Currently, I don't think gang scheduler is supported via kustomize, unless user manually applies required configurations.

Thanks for your reminder @ardaguclu I was also thinking about this question. I'm not very familiar with kustomize. Are the configurations under /config enabled by default? Since gang scheduling is turned off by default, I didn't add any volcano-related configurations under /config. I don't know if kustomize can work like Helm, allowing you to specify which to enable and which to disable?

ardaguclu · 2025-08-06T13:15:06Z

For example, if the only requirement is to define a ClusterRole, you need to add it in here https://github.com/kubernetes-sigs/lws/tree/main/config/rbac. After that you need to add this file

lws/config/rbac/kustomization.yaml

Line 1 in b73c72c

resources:

in commented format by mentioning "if you want to enable gang scheduling, you need to comment out this section, etc."

JesseStutler · 2025-08-06T15:01:01Z

For example, if the only requirement is to define a ClusterRole, you need to add it in here https://github.com/kubernetes-sigs/lws/tree/main/config/rbac. After that you need to add this file

lws/config/rbac/kustomization.yaml

Line 1 in b73c72c

resources:

in commented format by mentioning "if you want to enable gang scheduling, you need to comment out this section, etc."

Good idea, thanks! @ardaguclu I will add it in later PRs

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 11, 2025

k8s-ci-robot requested review from ahg-g and yankay April 11, 2025 03:31

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 11, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 11, 2025

JesseStutler mentioned this pull request Apr 11, 2025

Add KEP#407: Gang Scheduling #496

Merged

Edwinhr716 reviewed Apr 14, 2025

View reviewed changes

JesseStutler force-pushed the gang_dev_volcano branch from de5fb19 to 7a8c16a Compare April 15, 2025 02:15

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 15, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 15, 2025

JesseStutler force-pushed the gang_dev_volcano branch from 7a8c16a to 9c3d676 Compare July 22, 2025 02:19

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 22, 2025

JesseStutler changed the title ~~[WIP] Add Gang Scheduling Support for lws and volcano implementation example~~ Add Gang Scheduling Support for lws and volcano implementation example Jul 22, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2025

JesseStutler force-pushed the gang_dev_volcano branch from 9c3d676 to baf9e25 Compare July 22, 2025 02:28

k8s-ci-robot requested review from Edwinhr716 and kerthcet July 22, 2025 02:33

JesseStutler force-pushed the gang_dev_volcano branch from baf9e25 to 251bf4b Compare July 22, 2025 03:54

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

JesseStutler force-pushed the gang_dev_volcano branch from 251bf4b to 6103dc2 Compare July 23, 2025 02:00

kerthcet reviewed Aug 3, 2025

View reviewed changes

hack/e2e-test.sh Outdated Show resolved Hide resolved

hack/e2e-test.sh Outdated Show resolved Hide resolved

test/e2e/e2e_test.go Outdated Show resolved Hide resolved

JesseStutler force-pushed the gang_dev_volcano branch from 8ad887b to 4e6b368 Compare August 4, 2025 11:03

Implement gang scheduling and add volcano provider example

d7d70f8

Signed-off-by: JesseStutler <[email protected]>

JesseStutler force-pushed the gang_dev_volcano branch from 4e6b368 to 940e139 Compare August 4, 2025 11:16

JesseStutler requested a review from kerthcet August 4, 2025 11:31

kerthcet mentioned this pull request Aug 4, 2025

Add gang-scheduling e2e test for lws kubernetes/test-infra#35258

Merged

kerthcet reviewed Aug 4, 2025

View reviewed changes

Add gang scheduling integration tests and e2e tests

a2cbb68

Signed-off-by: JesseStutler <[email protected]>

JesseStutler force-pushed the gang_dev_volcano branch from 940e139 to a2cbb68 Compare August 4, 2025 14:20

k8s-ci-robot assigned kerthcet Aug 4, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 4, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 4, 2025

k8s-ci-robot merged commit 2d0e76b into kubernetes-sigs:main Aug 4, 2025
13 checks passed

JesseStutler mentioned this pull request Aug 7, 2025

Fix Volcano instaillation guide/ Add gang scheduling configuration in /config dir / Add unit test #611

Open

JesseStutler mentioned this pull request Aug 15, 2025

[FEATURE]: Support for multi-node replicas with LeaderWorkerSet. ai-dynamo/dynamo#869

Closed

JesseStutler mentioned this pull request Sep 24, 2025

[WIP]Support LeaderWorkerSet volcano-sh/volcano#4043

Closed

Add Gang Scheduling Support for lws and volcano implementation example #498

Add Gang Scheduling Support for lws and volcano implementation example #498

Uh oh!

Conversation

JesseStutler commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

Validation

Uh oh!

k8s-ci-robot commented Apr 11, 2025

Uh oh!

netlify bot commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-lws canceled.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yankay commented Apr 15, 2025

Uh oh!

JesseStutler commented Jul 22, 2025

Uh oh!

JesseStutler commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kerthcet commented Aug 2, 2025

Uh oh!

JesseStutler commented Aug 3, 2025

Uh oh!

JesseStutler commented Aug 3, 2025

Uh oh!

kerthcet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JesseStutler commented Aug 4, 2025

Uh oh!

JesseStutler commented Aug 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kerthcet commented Aug 4, 2025

Uh oh!

k8s-ci-robot commented Aug 4, 2025

Uh oh!

Uh oh!

ardaguclu commented Aug 6, 2025

Uh oh!

ardaguclu commented Aug 6, 2025

Uh oh!

JesseStutler commented Aug 6, 2025

Uh oh!

ardaguclu commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

JesseStutler commented Apr 11, 2025 •

edited

Loading

netlify bot commented Apr 11, 2025 •

edited

Loading

JesseStutler commented Jul 22, 2025 •

edited

Loading

ardaguclu commented Aug 6, 2025 •

edited

Loading