Skip to content

Add timeout between conformance test cases#3243

Merged
k8s-ci-robot merged 1 commit into
kubernetes-sigs:mainfrom
wstcliyu:timeout
Aug 6, 2024
Merged

Add timeout between conformance test cases#3243
k8s-ci-robot merged 1 commit into
kubernetes-sigs:mainfrom
wstcliyu:timeout

Conversation

@wstcliyu
Copy link
Copy Markdown
Contributor

@wstcliyu wstcliyu commented Aug 2, 2024

What type of PR is this?

/area conformance

What this PR does / why we need it:
It reduces the flakiness of running conformance tests sequentially. Because some test cases reuse the same Gateway and the same Service, and have the similar path matcher, it may need longer time to fully clean up the resources.

Which issue(s) this PR fixes:

N/A

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. area/conformance-test Issues or PRs related to Conformance tests. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 2, 2024
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Aug 2, 2024

Hi @wstcliyu. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 2, 2024
robscott
robscott reviewed Aug 2, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@robscott robscott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wstcliyu!

Comment thread conformance/utils/config/timeout.go Outdated
Comment thread conformance/utils/suite/suite.go
test: test,
result: res,
}
if res == testSucceeded || res == testFailed {
Copy link
Copy Markdown
Member

@robscott robscott Aug 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Would be good to skip this step when you're finishing the last step - presumably it's only useful in between tests and not after the last one.

Copy link
Copy Markdown
Contributor Author

@wstcliyu wstcliyu Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Copy Markdown
Member

@mlavacca mlavacca Aug 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. The sleep is performed for the last test as well. We should perform some checks on the test index.

Copy link
Copy Markdown
Contributor Author

@wstcliyu wstcliyu Aug 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments. I've updated the code here to sleep before running each test case that is not skipped except the first test case. Please review again

@robscott
Copy link
Copy Markdown
Member

robscott commented Aug 2, 2024

x-ref #3233

@robscott
Copy link
Copy Markdown
Member

robscott commented Aug 2, 2024

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 2, 2024
shaneutt
shaneutt reviewed Aug 5, 2024
View reviewed changes
Comment thread conformance/utils/suite/suite.go Outdated
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 5, 2024
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 5, 2024
@wstcliyu wstcliyu mentioned this pull request Aug 5, 2024
Closed
@mlavacca
Copy link
Copy Markdown
Member

mlavacca commented Aug 6, 2024

/cc

@k8s-ci-robot k8s-ci-robot requested a review from mlavacca August 6, 2024 10:05
shaneutt
shaneutt approved these changes Aug 6, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@shaneutt shaneutt left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this @wstcliyu. I'm OK with this as an interim solution. @robscott / @mlavacca for final LGTM when ready 🖖

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Aug 6, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: shaneutt, wstcliyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 6, 2024
mlavacca
mlavacca reviewed Aug 6, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@mlavacca mlavacca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @wstcliyu!

This sleep is a bit hacky and I think we should really focus on solving the root issue. Since this is something that can produce immediate relief, let's go this way. I provided my point of view in the original issue and I really think we should invest ourselves in improving the test isolation in a robust manner.

Comment thread conformance/utils/suite/suite.go
test: test,
result: res,
}
if res == testSucceeded || res == testFailed {
Copy link
Copy Markdown
Member

@mlavacca mlavacca Aug 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. The sleep is performed for the last test as well. We should perform some checks on the test index.

@robscott
Copy link
Copy Markdown
Member

robscott commented Aug 6, 2024

Thanks @wstcliyu!

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2024
@k8s-ci-robot k8s-ci-robot merged commit 47930fb into kubernetes-sigs:main Aug 6, 2024
xtineskim pushed a commit to xtineskim/gateway-api that referenced this pull request Aug 8, 2024
@wstcliyu @xtineskim
Add timeout between test cases (kubernetes-sigs#3243)
7fcc041
This was referenced Nov 21, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robscott robscott robscott left review comments

@michaelbeaumont michaelbeaumont Awaiting requested review from michaelbeaumont

@sunjayBhatia sunjayBhatia Awaiting requested review from sunjayBhatia

+2 more reviewers

@mlavacca mlavacca mlavacca left review comments

@shaneutt shaneutt shaneutt approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@robscott robscott

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/conformance-test Issues or PRs related to Conformance tests. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wstcliyu @k8s-ci-robot @robscott @mlavacca @shaneutt