Added logic to block conformance report if support features source is manual for non-gateway features. #3927

bexxmodd · 2025-07-16T21:22:27Z

What type of PR is this?
/area conformance-test

What this PR does / why we need it:
To guide conformance test users in a right direction, we want to discourage manually supplying supported features for conformance tests, instead those features should be inferred from GWC status. This change will block report generation if above mentioned logic is not followed.

Does this PR introduce a user-facing change?:

For generating conformance report for Gateway features, there will be no need to use flags like `--supported-features` or `--exempt-features`, instead conformance suite will automatically read them from GWC status. This is the behavior we want to encourage to reduce manual intervention in the report generation process, those manually supplied (or excluded) features should only be used for local development/testing.

… manual for non-gateway features.

k8s-ci-robot · 2025-07-16T21:22:36Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bexxmodd
Once this PR has been reviewed and has the lgtm label, please assign liorlieberman for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

conformance/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-07-16T21:22:37Z

Hi @bexxmodd. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bexxmodd · 2025-07-16T21:33:53Z

/cc @LiorLieberman @robscott

howardjohn · 2025-07-16T23:04:16Z

conformance/utils/suite/suite.go

+	if suite.supportedFeaturesSource == supportedFeaturesSourceManual &&
+		!hasMeshFeatures(suite.SupportedFeatures) &&
+		!suite.conformanceProfiles.HasAny(MeshHTTPConformanceProfileName, MeshGRPCConformanceProfileName) {
+		return nil, fmt.Errorf("can't generate report: Gateway's supported features should be read from Status and not supplied through flags")


This means its impossible to run the report if your gateway doesn't support the SupportFeatures feature. Which is a brand new feature. This seems far to early to do, if ever

Agree that we should not block in v1.4. I think the corresponding GEP said that while this would be the default/encouraged path for v1.4+, it would only become required in v1.5. Of course mesh features will also have a longer extension here because there's currently no alternative way of listing supported features.

Given that we decided to give grace period for Supported Features until v1.5 release, my intention with this PR was to merge it after Jul 22 so it falls under v1.5, when we'll start proactively blocking manually specified reports.

robscott

Thanks @bexxmodd!

robscott · 2025-07-17T00:04:37Z

conformance/utils/suite/suite.go

+	if suite.supportedFeaturesSource == supportedFeaturesSourceManual &&
+		!hasMeshFeatures(suite.SupportedFeatures) &&
+		!suite.conformanceProfiles.HasAny(MeshHTTPConformanceProfileName, MeshGRPCConformanceProfileName) {
+		return nil, fmt.Errorf("can't generate report: Gateway's supported features should be read from Status and not supplied through flags")


Agree that we should not block in v1.4. I think the corresponding GEP said that while this would be the default/encouraged path for v1.4+, it would only become required in v1.5. Of course mesh features will also have a longer extension here because there's currently no alternative way of listing supported features.

robscott · 2025-07-17T00:06:21Z

conformance/utils/suite/suite.go

 	defer suite.lock.RUnlock()

+	if suite.supportedFeaturesSource == supportedFeaturesSourceManual &&
+		!hasMeshFeatures(suite.SupportedFeatures) &&


A bit orthogonal to this, but I think it would be useful to proactively ignore mesh features published in GatewayClass status so that it's clear that those should be reported separately and aren't really tied to the GatewayClass. Since it seems like a Mesh resource is ~imminent, it seems cleaner for Mesh features to be reported there in the future instead of both resources reporting all features.

Since I think this would exclusively affect Istio, would appreciate some feedback from that side on this idea:

/cc @howardjohn @mikemorris

That makes sense to me, agreed clearing this up sooner sounds good.

@robscott currently if Mesh features are read from GWC we throw an error

Are you implying to have some extra check here?

That impl seems good to me (and easy to catch/fix as an implementer) @bexxmodd

robscott · 2025-07-17T00:34:52Z

/ok-to-test

k8s-ci-robot · 2025-09-20T11:46:32Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kflynn · 2025-10-07T16:35:20Z

There was a lot of discussion about this on Slack back in July that never made it to this PR. Summarizing from that:

I'm very much in favor of publishing supportedFeatures on Gateway or Mesh resources, depending on what the implementation supports.
I'm very much in favor of having the conformance code default to reading which features to test from supportedFeatures for whatever profiles are being requested.
I'm in favor of letting conformance detect mismatches between supportedFeatures and what tests pass or fail, and I'm in favor of somehow telling Ana when she tries to use a feature not in supportedFeatures. (I don't know that either of these is really practical to implement, but I'd be in favor of them.)
I'm very, very strongly opposed to trying to block reports that use manually-specified features. There's no such thing as zero-cost code, nor zero-error code, and in every case I've seen similar things in the past, the costs have outweighed any benefits.

In this particular case, the benefit is particularly unclear, but the cost is not, so I'm opposed to including this at all. (I might be OK, much much later, with relying completely on features read from supportedFeatures -- but not yet.)

howardjohn

I strongly agree with Flynn that this is not the right approach for the project

robscott · 2025-10-07T18:26:48Z

I'm very, very strongly opposed to trying to block reports that use manually-specified features. There's no such thing as zero-cost code, nor zero-error code, and in every case I've seen similar things in the past, the costs have outweighed any benefits.

This seems to be the crux of the case against this change, but I don't think I fully understand it. It seems like we're saying that code can have errors, and therefore we shouldn't make changes to our code?

In this particular case, the benefit is particularly unclear, but the cost is not, so I'm opposed to including this at all

I think the benefit is quite clear - we ensure that all conformant implementations are consistently publishing the features they support. I'm not really sure what the cost is here other than that there may be bugs in our implementation, which seems true of anything we could possibly do?

howardjohn · 2025-10-07T18:34:14Z

Would you think it was acceptable to block an entire conformance report if some single feature wasnt supported? If not, why should this one do that?

we already have only like 8/40 implementations not submitting any conformance reports , why artificially punish some of those 8 just because they don't support 1 features (that is, IMO, the least useful feature in the API)?

additionally, requiring this on Mesh which is an experimental feature, would be unacceptable as well

bexxmodd · 2025-10-07T18:47:20Z

additionally, requiring this on Mesh which is an experimental feature, would be unacceptable as well

This one is specific for GW features.

kflynn · 2025-10-07T19:31:49Z

@robscott, if you default to reading supported features from supportedFeatures and leave everything else the same, you'll get to just about the same place as this idea that we need to police manually-supplied features. We simply don't need to solve this "problem" with code: the pressure that will push implementations to be accurate in what they claim to support is that the users of the implementation will either demand features that they don't see listed as supported, or yell if features that they do see listed as supported don't work.

Furthermore, the proposed code won't solve this problem (that we don't need to solve) anyway, since reports can still be edited by hand before their PR goes up. But it will require maintenance -- and adding code that has to be maintained in order to not solve a problem that doesn't need solving anyway is 100% incurring costs for no tangible benefit.

Neither is it hypothetical to talk about maintenance costs, because right now at this very moment, code introduced to (not) solve this problem (that we don't need to solve) is preventing me from running conformance at all for Linkerd... which is a thing that was never supposed to be affected anyway, but was.

At this point, Beka and I have already both separately spent time on this, so we are already racking up more costs for no wins -- and it's not fixed yet, either, so the costs will go higher still. If that doesn't resonate with you personally, I can start billing you per my normal hourly consulting rate for time I spend on this... 🙂

kflynn · 2025-10-07T19:33:23Z

/hold

just to make sure this isn't accidentally merged while we're discussing it.

bexxmodd added 2 commits July 16, 2025 14:15

Added logic to block conformance report if support features source is…

57450f0

… manual for non-gateway features.

Updated error text.:

f9382e0

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. area/conformance-test Issues or PRs related to Conformance tests. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 16, 2025

k8s-ci-robot requested review from kflynn and sunjayBhatia July 16, 2025 21:22

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 16, 2025

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 16, 2025

k8s-ci-robot requested review from LiorLieberman and robscott July 16, 2025 21:33

howardjohn reviewed Jul 16, 2025

View reviewed changes

robscott reviewed Jul 17, 2025

View reviewed changes

k8s-ci-robot requested review from howardjohn and mikemorris July 17, 2025 00:07

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 17, 2025

bexxmodd mentioned this pull request Aug 25, 2025

GEP: SupportedFeatures #3164

Open

shaneutt added this to Release v1.4.0 Aug 25, 2025

shaneutt self-assigned this Aug 25, 2025

shaneutt moved this to Review in Release v1.4.0 Aug 25, 2025

shaneutt added this to the v1.4.0 milestone Aug 25, 2025

shaneutt linked an issue Aug 25, 2025 that may be closed by this pull request

GEP: SupportedFeatures #3164

Open

shaneutt requested review from mlavacca, shaneutt and youngnick August 25, 2025 18:00

shaneutt removed this from Release v1.4.0 Aug 25, 2025

shaneutt modified the milestones: v1.4.0, v1.5.0 Aug 25, 2025

shaneutt removed their assignment Aug 25, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 20, 2025

kflynn mentioned this pull request Oct 7, 2025

Fix that allows to --all-features and --exempt-features flags work together #4149

Merged

howardjohn suggested changes Oct 7, 2025

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 7, 2025

Added logic to block conformance report if support features source is manual for non-gateway features. #3927

Are you sure you want to change the base?

Added logic to block conformance report if support features source is manual for non-gateway features. #3927

Uh oh!

Conversation

bexxmodd commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 16, 2025

Uh oh!

k8s-ci-robot commented Jul 16, 2025

Uh oh!

bexxmodd commented Jul 16, 2025

Uh oh!

howardjohn Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

bexxmodd Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robscott left a comment

Choose a reason for hiding this comment

Uh oh!

robscott Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

robscott Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

mikemorris Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

bexxmodd Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

mikemorris Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

robscott commented Jul 17, 2025

Uh oh!

k8s-ci-robot commented Sep 20, 2025

Uh oh!

kflynn commented Oct 7, 2025

Uh oh!

howardjohn left a comment

Choose a reason for hiding this comment

Uh oh!

robscott commented Oct 7, 2025

Uh oh!

howardjohn commented Oct 7, 2025

Uh oh!

bexxmodd commented Oct 7, 2025

Uh oh!

kflynn commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kflynn commented Oct 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

bexxmodd commented Jul 16, 2025 •

edited

Loading

bexxmodd Jul 17, 2025 •

edited

Loading

kflynn commented Oct 7, 2025 •

edited

Loading