feat(test): adding support for backwards compatibility testing by chatton · Pull Request #1912 · cosmos/ibc-go

chatton · 2022-08-08T14:41:32Z

Description

This PR adds support for backwards compatibility testing in the form of allowing us to specify the images to be used for chain A and chain B by running a workflow_dispatch workflow via the Github UI or a tool like gh

E.g. we can run FeeMiddlewareTestSuite with chain A on main and chain B on v4.0.0-r2

We will have UI access to this workflow once it is merged to main, in order to test it, the best option is to use gh. You can test it with this command once you have gh installed.

gh workflow run "Manual E2E" --ref "cian/issue#1911-support-backwards-compatibility-tests" -f test-entry-point=TestFeeMiddlewareTestSuite -f chain-b-image="ghcr.io/cosmos/ibc-go-simd" -f chain-b-tag="v4.0.0-rc3" -f chain-a-image="ghcr.io/cosmos/ibc-go-simd-e2e" -f chain-a-tag="main" -f relayer-tag="v2.0.0-rc2"

Changing values as appropriate.

closes: #1911

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer
Review Codecov Report in the comment section below once CI passes

chatton · 2022-08-08T14:42:13Z

+	sigs.k8s.io/yaml v1.3.0 // indirect
 )
+
+replace github.com/strangelove-ventures/ibctest => ../../forks/ibctest


this must be removed before the PR can be merged.

…ity-tests

https://github.com/cosmos/ibc-go into cian/issue#1911-support-backwards-compatibility-tests

…ity-tests

codecov-commenter · 2022-08-09T10:55:35Z

Codecov Report

❌ Patch coverage is 36.36364% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.09%. Comparing base (f054208) to head (c4bca40).
⚠️ Report is 2306 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/build_test_matrix/main.go	36.36%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1912      +/-   ##
==========================================
- Coverage   80.12%   80.09%   -0.04%     
==========================================
  Files         167      167              
  Lines       11766    11773       +7     
==========================================
+ Hits         9428     9429       +1     
- Misses       1923     1929       +6     
  Partials      415      415

Files with missing lines	Coverage Δ
cmd/build_test_matrix/main.go	`65.82% <36.36%> (-5.02%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

https://github.com/cosmos/ibc-go into cian/issue#1911-support-backwards-compatibility-tests

…ity-tests

colin-axner

wahoo!!

colin-axner · 2022-08-10T10:16:59Z

-	rlyTag, ok := os.LookupEnv(GoRelayerTag)
+	chainBSimdTag, ok := os.LookupEnv(ChainBSimdTagEnv)
+	if !ok {
+		chainBSimdTag = chainASimdTag


Should we document the default behaviour somewhere? That is when the chainB image/tag isn't specified. It might be a little odd if the image is specified but that tag isn't? or vice versa?

colin-axner · 2022-08-10T10:18:49Z

 // field in a github action workflow. This string can be used with `fromJSON(str)` to dynamically build
 // the workflow matrix to include all E2E tests under the e2eRootDirectory directory.
-func getGithubActionMatrixForTests(e2eRootDirectory string) (GithubActionTestMatrix, error) {
+func getGithubActionMatrixForTests(e2eRootDirectory, suite string) (GithubActionTestMatrix, error) {


Should we add godoc to specify the usage of an empty string to indicate runAllTests?

colin-axner · 2022-08-10T10:21:04Z

+name: Manual E2E
+on:
+  workflow_dispatch:
+    inputs:
+      test-suite:
+        description: 'The Test Suite To Run'
+        required: true
+        type: string
+        default: ""
+      chain-a-image:
+        description: 'The image to use for chain A'
+        required: true
+        type: string
+        default: "ghcr.io/cosmos/ibc-go-simd-e2e"
+      chain-a-tag:
+        description: 'The tag to use for chain A'
+        required: true
+        type: string
+        default: "main"
+      chain-b-image:
+        description: 'The image to use for chain B'
+        required: true
+        type: string
+        default: "ghcr.io/cosmos/ibc-go-simd"
+      chain-b-tag:
+        description: 'The tag to use for chain B'
+        required: true
+        default: "v4.0.0-rc2"
+        type: string
+      relayer-tag:
+        description: 'The tag to use for the relayer'
+        required: true
+        default: "v2.0.0-rc2"
+        type: string


nice! Is the idea to add multiple files, one for each previous version?

the way inputs work is that you can specify a value when you run the workflow (vi UI or using a cli tool like gh). This one workflow should let us run any test suite using any combination of versions.

Got it. So we would run this workflow manually rather than it being run on each pr. That makes sense. Mostly to save execution time?

yes this is a manual task, we can run the permutations we care about when we are doing QA. I think it might be overkill to run all these permutations on every PR.

We can also pick and choose which test suites are important for which versions, I.e. no fee middleware on v2

Sounds good. Could you add instructions on how to run manually? I'd like to test it out myself

@colin-axner I added instructions on how to execute these tests with gh. Once they are merged into main, we will be able to use the UI to trigger the workflows.

Adding a sample run here for reference: https://github.com/cosmos/ibc-go/runs/7766177908?check_suite_focus=true

…ity-tests

https://github.com/cosmos/ibc-go into cian/issue#1911-support-backwards-compatibility-tests

…ity-tests

colin-axner

LGTM! This is so cool! I am very excited :)

colin-axner · 2022-08-10T12:52:58Z

+        type: choice
+        options:
+          - v4.0.0-rc3
+          - v3.0.0-rc2


Suggested change

- v3.0.0-rc2

- v3.0.0

How do we get v3.1, v2.3 etc?

to start with I just added a few basic entries. This are just images that exist already. All this list is doing is giving us a drop down menu. We can edit it as we see fit! We just need to make sure the images exist.

damiannolan

Awesome job

damiannolan · 2022-08-10T17:17:28Z

+        options:
+          - main
+          - v4.0.0-rc3
+          - v3.0.0
+          - v2.2.0
+          - v2.1.0
+          - v2.0.0


Are we missing v3.1.0 here?

we are, but that image hasn't actually been built yet. That will have to come in a follow up!

feat(test): adding support for backwards compatibility testing

8f44301

chatton commented Aug 8, 2022

View reviewed changes

chatton and others added 4 commits August 8, 2022 15:45

Merge branch 'main' into cian/issue#1911-support-backwards-compatibil…

f470bcf

…ity-tests

chore: made names of variables more descriptive

2297280

Merge branch 'cian/issue#1911-support-backwards-compatibility-tests' of

4c30485

https://github.com/cosmos/ibc-go into cian/issue#1911-support-backwards-compatibility-tests

Merge branch 'main' into cian/issue#1911-support-backwards-compatibil…

f7136fe

…ity-tests

chatton and others added 9 commits August 9, 2022 16:34

ibctest upgraded to 0.46

cebb54b

Merge branch 'cian/issue#1911-support-backwards-compatibility-tests' of

44372fa

https://github.com/cosmos/ibc-go into cian/issue#1911-support-backwards-compatibility-tests

merge main

4f32582

merge main

02a4650

chore: adding e2e-manual workflow

a1242ec

Merge branch 'main' into cian/issue#1911-support-backwards-compatibil…

97cc9e3

…ity-tests

chore: specify test suite with an env var

d100cc8

chore: correcting condition in build_test_matrix_main.go

e40eb94

chore: referencing inputs correctly through github action

b4b6e0d

colin-axner reviewed Aug 10, 2022

View reviewed changes

chatton and others added 4 commits August 10, 2022 11:28

chore: adding multiple choice to the workflow

4f5dc37

Merge branch 'main' into cian/issue#1911-support-backwards-compatibil…

7dbd60d

…ity-tests

chore: addign docstrings based on PR feedback

dc545df

Merge branch 'cian/issue#1911-support-backwards-compatibility-tests' of

3850159

https://github.com/cosmos/ibc-go into cian/issue#1911-support-backwards-compatibility-tests

chatton marked this pull request as ready for review August 10, 2022 11:12

chatton requested review from AdityaSripal, charleenfei, crodriguezvega, damiannolan and seantking as code owners August 10, 2022 11:12

trigger workflow in order to be able to use gh to run it

c155674

chatton marked this pull request as draft August 10, 2022 11:22

chore: remove login to docker registry

e7856f2

chatton and others added 6 commits August 10, 2022 12:30

chore: removing pull_request as target

53189f7

chore: adding default value for chain-b-tag

e8efc01

chore: changed events to event

a2c1ae0

chore: updated task to use test entrypoint instead of test suite name

85452f5

chore: fixed relayer tag input

107aeca

Merge branch 'main' into cian/issue#1911-support-backwards-compatibil…

fc0dc1f

…ity-tests

chatton marked this pull request as ready for review August 10, 2022 12:31

colin-axner approved these changes Aug 10, 2022

View reviewed changes

chore: changing from rc to v3.0.0

c4bca40

damiannolan approved these changes Aug 10, 2022

View reviewed changes

chatton merged commit 2bd397b into main Aug 11, 2022

chatton deleted the cian/issue#1911-support-backwards-compatibility-tests branch August 11, 2022 08:05

chatton mentioned this pull request Aug 24, 2022

Consider using git-cliff to generate CHANGELOG.md #2106

Closed

3 tasks

chatton mentioned this pull request Nov 7, 2022

Add git cliff toml to support generation of changelog #2697

Merged

9 tasks

Conversation

chatton commented Aug 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 9, 2022 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

colin-axner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatton Aug 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

colin-axner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damiannolan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chatton commented Aug 8, 2022 •

edited

Loading

codecov-commenter commented Aug 9, 2022 •

edited by codecov Bot

Loading

chatton Aug 10, 2022 •

edited

Loading