Skip to content

feat: implement node locking for NodeSet worker pods#130

Open
giuliocalzo wants to merge 8 commits intoSlinkyProject:mainfrom
giuliocalzo:sync
Open

feat: implement node locking for NodeSet worker pods#130
giuliocalzo wants to merge 8 commits intoSlinkyProject:mainfrom
giuliocalzo:sync

Conversation

@giuliocalzo
Copy link
Contributor

@giuliocalzo giuliocalzo commented Feb 24, 2026

Summary

Add lockNodes and lockNodeLifetime fields to NodeSetSpec to pin worker pods to their assigned Kubernetes nodes. When enabled, the controller records each pod-to-node mapping in NodeSetStatus and injects a requiredDuringSchedulingIgnoredDuringExecution NodeAffinity on pod recreation so each worker always returns to the same physical node.

The lockNodeLifetime field controls how long the lock persists: 0 means permanent, and a positive value (in seconds) causes the lock to expire after the pod stops running, allowing it to reschedule freely. Running pods continuously refresh their assignment timestamp so the countdown only begins once the pod is no longer active on the node.

Breaking Changes

none

Testing Notes

local testing with kind

status:
  nodeAssignments:
    "0":
      node: node-gpu-1
      at: 1740384000
    "1":
      node: node-gpu-2
      at: 1740384000

Additional Context

Add `lockNodes` and `lockNodeLifetime` fields to NodeSetSpec to pin
worker pods to their assigned Kubernetes nodes. When enabled, the
controller records each pod-to-node mapping in NodeSetStatus and injects
a requiredDuringSchedulingIgnoredDuringExecution NodeAffinity on pod
recreation so each worker always returns to the same physical node.

The lockNodeLifetime field controls how long the lock persists: 0 means
permanent, and a positive value (in seconds) causes the lock to expire
after the pod stops running, allowing it to reschedule freely. Running
pods continuously refresh their assignment timestamp so the countdown
only begins once the pod is no longer active on the node.
Document the lockNodes and lockNodeLifetime features in the workload
isolation guide, nodeset controller concept page, and Helm chart README.
Use ordinal index as map key instead of full pod name, Unix epoch int64
instead of RFC 3339 timestamp, and shorter JSON field names (node, at)
to reduce per-entry size from ~90 bytes to ~42 bytes (~53% reduction).
@vivian-hafener vivian-hafener self-assigned this Mar 2, 2026
@vivian-hafener vivian-hafener self-requested a review March 2, 2026 22:44
@giuliocalzo
Copy link
Contributor Author

good morning @vivian-hafener I rebase and adjust based on the last pre-commit checks, feel free to review it

@vivian-hafener vivian-hafener removed their request for review March 3, 2026 15:13
@vivian-hafener vivian-hafener removed their assignment Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants