Skip to content

Refactor ci #2137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
davidcassany merged 7 commits into from
Jul 11, 2024
Merged

Refactor ci #2137

davidcassany merged 7 commits into from
Jul 11, 2024

Conversation

@davidcassany
Copy link
Contributor

@davidcassany davidcassany commented Jul 9, 2024

Fixes #2132

@davidcassany davidcassany requested a review from a team as a code owner July 9, 2024 23:28
@davidcassany davidcassany force-pushed the refactor_ci branch 4 times, most recently from a32ef58 to 1215a59 Compare July 10, 2024 09:14
@davidcassany
Improve cache cleanup
edf8382 
Signed-off-by: David Cassany <[email protected]>
davidcassany
davidcassany commented Jul 10, 2024
View reviewed changes
Copy link
Contributor Author

@davidcassany davidcassany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR allows to run the upgrade test locally too. In fact, that's the thing, if it is runnable locally without having to configure infraestructure, it should also be possible to run it in GHA without needing external services like a registry.

In my env the following sequences passes:

make build && make build-os
make test-upgrade

.github/workflows/build.yaml
git fetch --prune --unshallow
- name: Log in to ghcr.io
uses: docker/login-action@v3
- name: Define version
Copy link
Contributor Author

@davidcassany davidcassany Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use version identifier and tag base on a hash from certain paths. This allows to re-use cache in case of PRs that are only touching parts that do not affect the build (e.g. tests code, workflows, etc.)

scripts/run_registry.sh
@@ -0,0 +1,78 @@
#!/bin/bash
Copy link
Contributor Author

@davidcassany davidcassany Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a script that handles the local registry instance with a variant of docker run registry:2. Challenges here were:

  • Need to compute a valid IP of the host that is visible from the qemu VM in "user mode" network setup.
  • Need to push to an insecure registry (I could not figure out how work around that, a self signed certificate would still be insecure unless the host is configured to trust it)

scripts/run_upgradetest.sh
pushd "${ROOT_PATH}" > /dev/null
go run ${ginkgo} ${ginkgo_args} ./tests/wait-active
go run ${ginkgo} ${ginkgo_args} ./tests/upgrade -- \
--toolkit-image=docker://${reg_url}/${toolkit_img} --upgrade-image=docker://${reg_url}/${upgrade_img}
Copy link
Contributor Author

@davidcassany davidcassany Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scripts is essentially existing because it was so cumbersome to compose this arguments from variables in the Makefile directly

@davidcassany davidcassany self-assigned this Jul 10, 2024
davidcassany
davidcassany commented Jul 10, 2024
View reviewed changes
.github/workflows/build_and_test_x86.yaml
jobs:
build-os:
permissions:
packages: write
Copy link
Contributor Author

@davidcassany davidcassany Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No more packages: write in this workflow. Only contents: read.

davidcassany
davidcassany commented Jul 10, 2024
View reviewed changes
.github/workflows/cache-cleanup.yaml
on:
pull_request_target:
types:
- synchronize
Copy link
Contributor Author

@davidcassany davidcassany Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To remove unused cache on pushing commits to an opened PR

anmazzotti
anmazzotti reviewed Jul 11, 2024
View reviewed changes
anmazzotti
anmazzotti approved these changes Jul 11, 2024
View reviewed changes
Copy link
Contributor

@anmazzotti anmazzotti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work. Left a couple of nitpicking comments, but nothing essential.
LGTM

@davidcassany davidcassany force-pushed the refactor_ci branch 2 times, most recently from 67bbc24 to f7c4f7f Compare July 11, 2024 09:16
@davidcassany
Improve ip detection
dbb8983 
Signed-off-by: David Cassany <[email protected]>
@davidcassany davidcassany enabled auto-merge (squash) July 11, 2024 09:25
@davidcassany davidcassany merged commit 644cdc7 into rancher:main Jul 11, 2024
@davidcassany davidcassany deleted the refactor_ci branch July 11, 2024 14:55
@davidcassany davidcassany mentioned this pull request Jul 15, 2024
Merged
davidcassany added a commit to davidcassany/elemental-toolkit that referenced this pull request Jul 18, 2024
@davidcassany
Refactor ci (rancher#2137)
1915158 
* Refactor CI to use a local registry
* Cache image and fetch image tarballs
* Composite fetch images
* Push to insecure registry with podman
* Simplify workflow runs
* Improve cache cleanup
* Improve ip detection

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 644cdc7)
davidcassany added a commit to davidcassany/elemental-toolkit that referenced this pull request Jul 18, 2024
@davidcassany
Refactor ci (rancher#2137)
423c069 
* Refactor CI to use a local registry
* Cache image and fetch image tarballs
* Composite fetch images
* Push to insecure registry with podman
* Simplify workflow runs
* Improve cache cleanup
* Improve ip detection

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 644cdc7)
davidcassany added a commit to davidcassany/elemental-toolkit that referenced this pull request Jul 18, 2024
@davidcassany
Refactor ci (rancher#2137)
c6e64a6 
* Refactor CI to use a local registry
* Cache image and fetch image tarballs
* Composite fetch images
* Push to insecure registry with podman
* Simplify workflow runs
* Improve cache cleanup
* Improve ip detection

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 644cdc7)
davidcassany added a commit that referenced this pull request Jul 29, 2024
@davidcassany @fgiudici
[backport] CI refactor to v2.1.x branch (#2146)
216ab93 
* Fix stackable sources for build-iso (#2061)

* Fix stackable sources for build-iso
* Move syncFunc as a parameter of DumpSource

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 22baf14)

* Fix error return

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit f9934d7)

* Change default root password

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 46a88e1)

* Allow key pair of ssh connection in tests

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit a5f526d)

* Set legacy password for upgrade test

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 6f0b28e)

* Allow insecure registries (#2131)

* Allow skipping TLS verification on registries

Signed-off-by: David Cassany <[email protected]>

* Improve error logging

Signed-off-by: David Cassany <[email protected]>

* Fix flag default value

Signed-off-by: David Cassany <[email protected]>

* Fix image extractor mock and use default TLS verification for tests

Signed-off-by: David Cassany <[email protected]>

---------

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 22996ef)

* Fix permissions

* Set token to explicit content read permissions
  (cherry picked from commit 3d657c9)

* Move permissions at top level
  (cherry picked from commit 8b8ae12)

* Fix workflows, permissions at top level
  (cherry picked from commit d3da1cf)

* Leftover from d3da1cf
  (cherry picked from commit 3206436)

* Fix nested permissions definition

  Top level permissions can't be increased for specific jobs,
  hence setting the permission on each specific job on workflows
  that require more fine grain approach.

  In addition it removes the pull-request permission as this is mostly
  required for PR decorators, read/write labels, etc.
  (cherry picked from commit 1a0903a)

* Refactor ci (#2137)

* Refactor CI to use a local registry
* Cache image and fetch image tarballs
* Composite fetch images
* Push to insecure registry with podman
* Simplify workflow runs
* Improve cache cleanup
* Improve ip detection

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 644cdc7)

* Fix cache cleaning for PRs and main pushes (#2144)

* Fix cache cleaning for PRs and main pushes
* Update .github/workflows/cache-cleanup.yaml

Co-authored-by: Francesco Giudici <[email protected]>
Signed-off-by: David Cassany Viladomat <[email protected]>
(cherry picked from commit f54f683)

* Remove unused steps in workflow

Signed-off-by: David Cassany <[email protected]>
(cherry picked from commit 51a85a7)
(cherry picked from commit 9f8d14fea58c2e458f8840d274422352f0b0b1c2)

* Do not fetch upstream go dependencies

Signed-off-by: David Cassany <[email protected]>

* Update cmd/cloud-init.go

Missing err

Co-authored-by: Francesco Giudici <[email protected]>
Signed-off-by: David Cassany Viladomat <[email protected]>

---------

Signed-off-by: David Cassany <[email protected]>
Signed-off-by: David Cassany Viladomat <[email protected]>
Co-authored-by: Francesco Giudici <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anmazzotti anmazzotti anmazzotti approved these changes

Assignees

@davidcassany davidcassany

Labels

None yet

Projects

Elemental
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Refactor Toolkit CI to use a private local registry

2 participants

@davidcassany @anmazzotti