Skip to content

Automatically run release validation tests in CI against multiple base systems #307

Merged
knuton merged 8 commits intomainfrom
release-validation-matrix
Jan 21, 2026
Merged

Automatically run release validation tests in CI against multiple base systems #307
knuton merged 8 commits intomainfrom
release-validation-matrix

Conversation

@yfyf
Copy link
Collaborator

@yfyf yfyf commented Jan 20, 2026

Currently we manually run release-validation tests, but this is easy to forget and there is no visibility to the team about the outcomes.

Since they are "expensive" to run (take a long time), running them on each push/PR to main is maybe too extreme, but running them on each push to release/* is warranted since it indicates we intend to deploy the tree in the somewhat near future.

CI also allows us to easily run the release-validation tests against multiple base system images. Since we have not yet decided on the "archetypal skeletons", for now this just hard-codes base system versions to some "ancient" ones + the previous (latest released) version.

Manually started CI run: https://github.com/dividat/playos/actions/runs/21177566028 🟢

yfyf added 2 commits January 20, 2026 10:25
This is a first step / exercise for testing against multiple "skeleton"
versions.
@yfyf yfyf requested a review from knuton January 20, 2026 08:51
@yfyf yfyf added the reviewable Ready for initial or iterative review label Jan 20, 2026
yfyf added 2 commits January 20, 2026 13:14
We exploit the fact that release disks are built with passwordless root
(for debugging purposes).
@yfyf yfyf force-pushed the release-validation-matrix branch from 9bbc9eb to 606eaaf Compare January 20, 2026 12:41
Cannot reproduce it locally, but maybe it is "too fast" on CI
@yfyf
Copy link
Collaborator Author

yfyf commented Jan 20, 2026

Discovered some flakiness in release-validation tests (mostly in CI), which seems to be fixed with
776f427 and 606eaaf, except for a strange navigation failures that seem to only happen on 2024.7.0 and which I cannot reproduce locally unfortunately. Trying to find the root cause, probably also timing related. Can open a separate PR for these.

yfyf added 3 commits January 20, 2026 15:54
Trying to fix strange CI issues
For weird reasons and only on CI the 2024.7.0-DISK version seems to take
very long until controller GUI pages are fully loaded on the screen.
Worse, sometimes the OCR results indicate that parts of two different
controller pages are visible at the same time.

While a long sleep after navigate_to_status_page() seems to resolve
this, it is unreasonable to slow down all other versions because of
this.
@yfyf yfyf force-pushed the release-validation-matrix branch from d1d70d6 to f01ad6f Compare January 20, 2026 15:38
@yfyf
Copy link
Collaborator Author

yfyf commented Jan 20, 2026

There's probably some video driver compatibility with QEMU issue in 2024.7.0, causing partial/very slow screen refresh.

The OCR texts indicate that both bits from the both the Controller main (Information) page and the Status Page are simultaneously visible on the screen - something that should not be possible.

For example: https://github.com/dividat/playos/actions/runs/21176096534/job/60905229495#step:4:4611

Update State  # <- this is only visible in System Status page
RebootRequired

Machine ID # <- this is only visible in Information page
f414-cca8-3125-48d2-9689-ebf2-87fb-67e0

Possible solutions:

  1. Wait even longer after navigate_to_system_status() for the screen to refresh
  2. Exclude 2024.7.0 from the testing matrix, since this seems to only affect this release

I tried option 1 in d1d70d6 and it seems to be sufficient, which amounts to an extra ~5 second sleep, but I don't think it's reasonable to penalize all other versions because of one flaky one.

So instead for now applying option 2 in f01ad6f, if we discover the same issues for some skeleton, we can go back to option 1.

@knuton knuton merged commit 5f3eae1 into main Jan 21, 2026
90 of 91 checks passed
@knuton knuton deleted the release-validation-matrix branch January 21, 2026 17:18
@knuton knuton removed the reviewable Ready for initial or iterative review label Jan 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants