Conformance tests can be flaky in some edge cases

[robscott]

What would you like to be added:
Some changes to conformance framework to reduce potential flakiness. Solutions might include:

1. A configurable timeout between tests to account for this kind of flakiness (depends on how long it takes for config to propagate in underlying implementation)
2. Reusing Gateways less across different tests
3. Encouraging each test to have unique path matchers (or any other kind of matcher)

Also open to any other alternatives.

Why this is needed:
As we're submitting a conformance report for GKE (#3230), we found that the simplest reproduction steps could be flaky. This is because the features we support result in a unique and somewhat problematic sequence of tests running. We go from simple-same-namespace:

gateway-api/conformance/tests/httproute-simple-same-namespace.yaml

Lines 7 to 12 in 58d5098

	parentRefs:
	- name: same-namespace
	rules:
	- backendRefs:
	- name: infra-backend-v1
	port: 8080

to weighted backends:

gateway-api/conformance/tests/httproute-weight.yaml

Lines 7 to 19 in 58d5098

	parentRefs:
	- name: same-namespace
	rules:
	- backendRefs:
	- name: infra-backend-v1
	port: 8080
	weight: 70
	- name: infra-backend-v2
	port: 8080
	weight: 30
	- name: infra-backend-v3
	port: 8080
	weight: 0

Importantly both tests are using the same Gateway, matching criteria (any), and primary Service. This means that if the routing configuration hasn't propagated quite yet, it will just look like we're not traffic splitting, and thus result in a flaky failure.

[wstcliyu]

It would be good if we can add the opinion for long term solution to this issue.

#3243 (comment)

[mlavacca]

I think the long-term solution would be to improve isolation between tests. Every test should create its own resources and clean them up once the test finishes. A side effect of this approach would be that we can control how many resources are required by the suite: At the moment, there are 3-4 long-living gateways that are provisioned upfront + gateways that are created by specific tests. This means that a cluster has to tackle at least 4-5 gateways at a time. Providing test isolation gives the implementation control over the cardinality of the concurrent Gateways created and provisioned.

[youngnick]

I don't think we ever got around to implementing it, but I remember discussing at some point that having a smaller number of created Gateways in the tests meant that it would be possible to pre-create the Gateways, and have the conformance tests check that each Gateway matched, allowing for implementations that need a long time to provision Gateways ( or who want to limit the number of Gateways provisioned) to still pass conformance.

Just wanted to make sure we don't lose that context when we solve this issue - which I absolutely agree we should.

[shaneutt]

/triage needs-information

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[candita]

/remove-lifecycle stale

@robscott @mlavacca @wstcliyu Does the code in #3243 actually run without a command line flag to set the sleep duration?

Many of the tests wait for the test to return a consistent response ,and this is the best solution for the problem.

@youngnick Pre-creating objects is already a minor pain point when we are just running a subset of conformance tests - unneeded objects may be created, and it can add time to the wait for test results.