use headless service for upload server by akalenyu · Pull Request #4052 · kubevirt/containerized-data-importer

akalenyu · 2026-03-09T16:41:26Z

What this PR does / why we need it:
apparently, there's no k8s API that says
"kube-proxy has synced iptables rules for this service on all nodes"
so, pod readiness/healthz (even endpointslices) could all report success,
but the client would still face a connection refused
(this is causing unnecessary restarts in the k8s 1.35 bump PR)

we faced something similar in virtctl too
#3545

arguably, a headless service was the right fit for us anyway
and cuts the above overhead out of the equation.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3851

Special notes for your reviewer:
Not sure if there's a backwards compatibility consideration in here,
since upload pods are not usually carried over an upgrade

Release note:

Enhancement: use headless service for upload server

apparently there's no k8s API that says "kube-proxy has synced iptables rules for this service on all nodes" so pod readiness/healthz (even endpointslices) could all report success but the client would still face a connection refused. we faced something similar in virtctl kubevirt#3545 arguably, a headless service was the right fit for us anyway and cuts the above overhead out of the equation. Signed-off-by: Alex Kalenyuk <[email protected]>

akalenyu · 2026-03-09T16:45:01Z

/cc @Acedus @mhenriks
would especially appreciate thoughts on

Special notes for your reviewer:
Not sure if there's a backwards compatibility consideration in here,
since upload pods are not usually carried over an upgrade

coveralls · 2026-03-09T16:53:17Z

coverage: 49.29% (-0.01%) from 49.3%
when pulling b9f9073 on akalenyu:headless-upload-svc
into 2acfe66 on kubevirt:main.

Acedus · 2026-03-09T18:43:42Z

-						IntVal: 8443,
-					},
+					Protocol:   corev1.ProtocolTCP,
+					Port:       8443,


Why do we change the exposed port to 8443 instead of 443?

this is a no op for headless, there will be no remapping

Right, no port translation... I wonder if some users rely on this somehow (although I doubt it due to the proxy).

yeah definitely hyrums law territory, thought about this too. if anyone did, they can still use their own services

Acedus · 2026-03-09T18:43:54Z

/cc @Acedus @mhenriks would especially appreciate thoughts on

Special notes for your reviewer:
Not sure if there's a backwards compatibility consideration in here,
since upload pods are not usually carried over an upgrade

I'd argue that it shouldn't be problematic since the upload-server Pod is transient and bound to the lifetime of an upload.

Acedus

I think this is good, thanks!

akalenyu · 2026-03-10T16:09:14Z

/cc @0xFelix
Hopefully this is the time where we can finally make the virtctl image upload tests parallel lol

0xFelix · 2026-03-12T14:56:38Z

@akalenyu Let's hope so! 😁

mhenriks · 2026-03-12T21:05:37Z

Fine with me. Getting rid of the cluster IP is fine for this case but endpointslices still have to be created and that may not happen right away either

akalenyu · 2026-03-15T10:37:58Z

Fine with me. Getting rid of the cluster IP is fine for this case but endpointslices still have to be created and that may not happen right away either

Yeah good point. I haven't seen that play out but I agree it's completely possible.
I think this PR stands regardless of that though

mhenriks · 2026-03-16T17:22:01Z

/approve

kubevirt-bot · 2026-03-16T17:22:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mhenriks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mhenriks]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

latest CDI release includes kubevirt/containerized-data-importer#4052 which finally allows us to trust the service/pod readiness (more info in the linked PR) Signed-off-by: Alex Kalenyuk <[email protected]>

akalenyu · 2026-05-11T14:15:45Z

/cherrypick release-v1.64 release-v1.63 release-v1.62 release-v1.61

kubevirt-bot · 2026-05-11T14:16:43Z

@akalenyu: new pull request created: #4122

Details

In response to this:

/cherrypick release-v1.64 release-v1.63 release-v1.62 release-v1.61

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Since the upload service became headless (kubevirt#4052), DNS resolves directly to the upload server pod IP. In OVN-Kubernetes clusters where the target namespace uses network segmentation, the pod's default network interface is locked down. The k8s.ovn.org/open-default-ports annotation opens port 8443 on the default interface, allowing the upload proxy to connect. The annotation is harmless on non-OVN clusters where it is simply ignored. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]>

latest CDI release includes kubevirt/containerized-data-importer#4052 which finally allows us to trust the service/pod readiness (more info in the linked PR) Signed-off-by: Alex Kalenyuk <[email protected]>

Since the upload service became headless (kubevirt#4052), DNS resolves directly to the upload server pod IP. In OVN-Kubernetes clusters where the target namespace uses network segmentation, the pod's default network interface is locked down. The k8s.ovn.org/open-default-ports annotation opens port 8443 on the default interface, allowing the upload proxy to connect. The annotation is harmless on non-OVN clusters where it is simply ignored. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]>

RamLavi · 2026-05-12T18:09:18Z

/cherry-pick release-v1.64

@akalenyu FYI following our conversation

kubevirt-bot · 2026-05-12T18:10:00Z

@RamLavi: new pull request could not be created: failed to create pull request against kubevirt/containerized-data-importer#release-v1.64 from head kubevirt-bot:cherry-pick-4052-to-release-v1.64: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for kubevirt-bot:cherry-pick-4052-to-release-v1.64."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"422"}

Details

In response to this:

/cherry-pick release-v1.64

@akalenyu FYI following our conversation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

RamLavi · 2026-05-12T18:12:16Z

/cherry-pick release-v1.64

@akalenyu FYI following our conversation

oh sorry was not aware you already cherry-picked. Thanks!!

* Extract upload server port into a common constant Replace hardcoded 8443 references in the upload controller with common.UploadServerPort to prevent silent breakage if the port ever changes. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]> * Add open-default-ports annotation to upload server pod Since the upload service became headless (#4052), DNS resolves directly to the upload server pod IP. In OVN-Kubernetes clusters where the target namespace uses network segmentation, the pod's default network interface is locked down. The k8s.ovn.org/open-default-ports annotation opens port 8443 on the default interface, allowing the upload proxy to connect. The annotation is harmless on non-OVN clusters where it is simply ignored. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]> --------- Signed-off-by: Ram Lavi <[email protected]>

#4135) * Extract upload server port into a common constant Replace hardcoded 8443 references in the upload controller with common.UploadServerPort to prevent silent breakage if the port ever changes. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]> * Add open-default-ports annotation to upload server pod Since the upload service became headless (#4052), DNS resolves directly to the upload server pod IP. In OVN-Kubernetes clusters where the target namespace uses network segmentation, the pod's default network interface is locked down. The k8s.ovn.org/open-default-ports annotation opens port 8443 on the default interface, allowing the upload proxy to connect. The annotation is harmless on non-OVN clusters where it is simply ignored. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]> --------- Signed-off-by: Ram Lavi <[email protected]> Co-authored-by: Ram Lavi <[email protected]>

#4136) * Extract upload server port into a common constant Replace hardcoded 8443 references in the upload controller with common.UploadServerPort to prevent silent breakage if the port ever changes. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]> * Add open-default-ports annotation to upload server pod Since the upload service became headless (#4052), DNS resolves directly to the upload server pod IP. In OVN-Kubernetes clusters where the target namespace uses network segmentation, the pod's default network interface is locked down. The k8s.ovn.org/open-default-ports annotation opens port 8443 on the default interface, allowing the upload proxy to connect. The annotation is harmless on non-OVN clusters where it is simply ignored. Assisted-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Ram Lavi <[email protected]> --------- Signed-off-by: Ram Lavi <[email protected]> Co-authored-by: Ram Lavi <[email protected]>

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Mar 9, 2026

kubevirt-bot requested review from aglitke and arnongilboa March 9, 2026 16:41

kubevirt-bot added the size/S label Mar 9, 2026

kubevirt-bot requested review from Acedus and mhenriks March 9, 2026 16:45

Acedus reviewed Mar 9, 2026

View reviewed changes

Acedus approved these changes Mar 10, 2026

View reviewed changes

kubevirt-bot assigned Acedus Mar 10, 2026

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 10, 2026

kubevirt-bot requested a review from 0xFelix March 10, 2026 16:09

akalenyu mentioned this pull request Mar 15, 2026

Update kubevirtci, 1.35 lanes #4047

Merged

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 16, 2026

kubevirt-bot merged commit 2924a9b into kubevirt:main Mar 16, 2026
21 checks passed

akalenyu mentioned this pull request Apr 12, 2026

imageupload tests: run in parallel kubevirt/kubevirt#17468

Merged

9 tasks

kubevirt-bot mentioned this pull request May 11, 2026

[release-v1.64] use headless service for upload server #4122

Merged

RamLavi mentioned this pull request May 12, 2026

Add open-default-ports annotation to upload server pod #4124

Merged

Conversation

akalenyu commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akalenyu commented Mar 9, 2026

Uh oh!

coveralls commented Mar 9, 2026

Uh oh!

Acedus Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

akalenyu Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Acedus Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

akalenyu Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Acedus commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Acedus left a comment

Choose a reason for hiding this comment

Uh oh!

akalenyu commented Mar 10, 2026

Uh oh!

0xFelix commented Mar 12, 2026

Uh oh!

mhenriks commented Mar 12, 2026

Uh oh!

akalenyu commented Mar 15, 2026

Uh oh!

mhenriks commented Mar 16, 2026

Uh oh!

kubevirt-bot commented Mar 16, 2026

Uh oh!

Uh oh!

akalenyu commented May 11, 2026

Uh oh!

kubevirt-bot commented May 11, 2026

Uh oh!

RamLavi commented May 12, 2026

Uh oh!

kubevirt-bot commented May 12, 2026

Uh oh!

RamLavi commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

akalenyu commented Mar 9, 2026 •

edited

Loading

Acedus commented Mar 9, 2026 •

edited

Loading