Use skip markers in tests #4188

Flamefire · 2023-01-19T08:05:18Z

We can use the more conventional @unittest.skipIf or self.skipTest markers to skip tests instead of C&P of conditions which might be wrong (some were)
An approach taken from IIRC PyTorch is to define own decorators requiresFoo as a wrapper around a specific skipIf which is reusable and easier to understand.

Only issue I can see is with the change to CI to ignore all skipped tests as now that I look at it previously we were able to selectively ignore some skipped tests.
So maybe we should not allow any skipped tests on CI which usually means installing additional Python packages. And for the tests requiring a github token we then need to resort back to the old approach but maybe we can still use a decorator here

Requires silence distutils deprecation warnings #4204

.github/workflows/unit_tests.yml

boegel · 2023-02-01T07:38:29Z

Only issue I can see is with the change to CI to ignore all skipped tests as now that I look at it previously we were able to selectively ignore some skipped tests. So maybe we should not allow any skipped tests on CI which usually means installing additional Python packages. And for the tests requiring a github token we then need to resort back to the old approach but maybe we can still use a decorator here

@Flamefire How about only allowing \.s+ in the output when running the test suite in a PR?
Because only then should we allow skipped tests (because the GitHub token is missing)...

So we should conditionally extend IGNORE_PATTERNS with |\.s+ if the test suite is being triggered by a push to develop branch (I think)...

Flamefire · 2023-02-01T12:07:44Z

So we should conditionally extend IGNORE_PATTERNS with |\.s+ if the test suite is being triggered by a push to develop branch (I think)...

I guess you meant to insert a NOT somewhere as no tests should be skipped on push to develop.

How about only allowing \.s+ in the output when running the test suite in a PR?
Because only then should we allow skipped tests (because the GitHub token is missing)...

Makes sense, however we should be more selective here. I hence added a dedicated decorator for that which doesn't produce output for PRs (i.e. silently succeeds) and make use of $FORCE_EB_GITHUB_TESTS to ensure github tests are run when the token is expected to be there.

Flamefire · 2025-09-04T08:16:22Z

Resolved

boegel · 2025-11-19T15:08:09Z

@Flamefire tests are failing now because some tests produces some unexpected (trace) output:

ERROR: Found printed messages in output of test suite
.................................................................................................................................................................................................................................................................................................................................s...........................................................................................................................s.................................................................s........................  >> loading toolchain module: intel-compilers/2024.0.0

Flamefire · 2025-11-20T12:17:50Z

That's actually a side effect of the changed default for --trace in EB 5.

As we do a lot of stdout/stderr mocking and/or disabling it again in the tests I disabled it by default now.

If you want that separately: #5053

Flamefire · 2025-11-21T08:57:14Z

Ok, the other skip sign was caused by an incomplete conflict resolution which was impossible to find with the current approach.
I force-pushed to fixup the conflict resolution failure directly

I added a list of skipped tests at the end. The expected ones should be marked with skip_silentCI* (used by e.g. requires_github_access). Unexpected ones will show up in this list in addition to the "s" chars in the short output.

We could on CI completely ignore ".s." (already done) and (instead of skip_silentCI) filter the "sometimes expected" failures in the CI config ($IGNORE_PATTERNS)

Flamefire · 2025-11-21T12:12:23Z

Rebased due to conflict by other PR

boegel · 2025-12-03T08:00:34Z

@Flamefire Needs more conflict resolving.

Sorry for dragging this on so long, it's a huge PR to review...

Flamefire · 2025-12-03T08:25:44Z

Rebased to fix

Sorry for dragging this on so long, it's a huge PR to review...

I know sorry for that. If you view the diff without whitespace changes it gets better at least

The pip-installable pysvn is incomplete and the full one is not easy to install.

Simply return if `$TEST_EASYBUILD_MODULES_TOOL` is not set to Lmod. This avoids any "skip" output on CI

For PRs silently skip the tests when no token is available else use a skip. On GHA force enable the tests when the source repo is 'easybuilds' so a token is expected to be available.

As that is used a lot use that for better describing the purpose.

GC3Pie on Python 3.11 does not work: gc3pie/gc3pie#674 So avoid the skip mark for that.

On CI we don't want the full output of --verbose but fail when tests are skipped. This is only indicated by a "s" inbetween many dots. So show a list of skipped tests and their reason at the end. This also allows easy filtering.

Successful output looks like this > ...............................................s...................................................................................... > ................................................................................................................... > ---------------------------------------------------------------------- > Ran 946 tests in 714.512s > > OK (skipped=3) We only care about the part until "Ran xxx tests" not including the divider (dashes). Also we can ignore any lines with just "s" and ".".

boegel added the tests label Jan 21, 2023

boegel added this to the next release (4.7.1?) milestone Jan 21, 2023

boegel reviewed Jan 21, 2023

View reviewed changes

.github/workflows/unit_tests.yml Outdated Show resolved Hide resolved

Flamefire force-pushed the skip-markers branch from d5fb240 to fa05f50 Compare February 1, 2023 12:05

Flamefire force-pushed the skip-markers branch 5 times, most recently from f76b45c to 65320fb Compare February 2, 2023 12:20

Flamefire mentioned this pull request Feb 2, 2023

add support for passing valid unittest options to test suite #3790

Merged

Flamefire force-pushed the skip-markers branch from 65320fb to c7f4ee9 Compare February 2, 2023 16:03

Flamefire changed the title ~~Use skip markers and skipTest in tests~~ Use skip markers in tests Feb 2, 2023

Flamefire force-pushed the skip-markers branch 7 times, most recently from 05c14fe to 2a38d1e Compare February 7, 2023 15:08

easybuilders deleted a comment from boegelbot Feb 15, 2023

boegel modified the milestones: next release (4.7.1), release after 4.7.1 Feb 25, 2023

boegel modified the milestones: next release (4.7.2), release after 4.7.2 May 10, 2023

Flamefire force-pushed the skip-markers branch 2 times, most recently from 4040414 to 149f7ec Compare May 11, 2023 08:56

boegel added this to the release after 5.1.2 (5.2.0?) milestone Sep 3, 2025

Flamefire force-pushed the skip-markers branch 3 times, most recently from 27abdeb to dc77998 Compare November 18, 2025 14:37

Flamefire force-pushed the skip-markers branch 2 times, most recently from 883121b to 54969a1 Compare November 21, 2025 08:50

Flamefire force-pushed the skip-markers branch 2 times, most recently from 65fd048 to 480fcaa Compare November 21, 2025 12:11

boegel modified the milestones: next release (5.2.0), release after 5.2.0 Dec 3, 2025

Flamefire force-pushed the skip-markers branch from 480fcaa to a41450b Compare December 3, 2025 08:24

Flamefire force-pushed the skip-markers branch from a41450b to 0c9090e Compare December 3, 2025 08:55

Flamefire added 10 commits December 8, 2025 11:32

Use skipTest than faulty C&P of return conditions

2bcb9d8

Silently skip SVN test on CI

77a2ab8

The pip-installable pysvn is incomplete and the full one is not easy to install.

Avoid output of expected skip of LMod tests

c4dfe47

Simply return if `$TEST_EASYBUILD_MODULES_TOOL` is not set to Lmod. This avoids any "skip" output on CI

Move GC3Pie check to skip decorator

c44cd40

Introduce skip marker for github token-using tests

7e715e7

For PRs silently skip the tests when no token is available else use a skip. On GHA force enable the tests when the source repo is 'easybuilds' so a token is expected to be available.

Introduce skip_silent* decorators

66bddb6

As that is used a lot use that for better describing the purpose.

Silently skip GC3PIe test w/ Python 3.11

cd2db49

GC3Pie on Python 3.11 does not work: gc3pie/gc3pie#674 So avoid the skip mark for that.

Show skipped tests at end of CI run with reason.

4230309

On CI we don't want the full output of --verbose but fail when tests are skipped. This is only indicated by a "s" inbetween many dots. So show a list of skipped tests and their reason at the end. This also allows easy filtering.

Update IGNORE_PATTERNS

a012fba

Flamefire force-pushed the skip-markers branch from 0c9090e to 13f8f11 Compare December 8, 2025 10:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use skip markers in tests #4188

Use skip markers in tests #4188

Uh oh!

Flamefire commented Jan 19, 2023 •

edited by branfosj

Loading

Uh oh!

Uh oh!

boegel commented Feb 1, 2023

Uh oh!

Flamefire commented Feb 1, 2023

Uh oh!

Flamefire commented Sep 4, 2025

Uh oh!

boegel commented Nov 19, 2025

Uh oh!

Flamefire commented Nov 20, 2025

Uh oh!

Flamefire commented Nov 21, 2025 •

edited

Loading

Uh oh!

Flamefire commented Nov 21, 2025

Uh oh!

boegel commented Dec 3, 2025

Uh oh!

Flamefire commented Dec 3, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Use skip markers in tests #4188

Are you sure you want to change the base?

Use skip markers in tests #4188

Uh oh!

Conversation

Flamefire commented Jan 19, 2023 • edited by branfosj Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

boegel commented Feb 1, 2023

Uh oh!

Flamefire commented Feb 1, 2023

Uh oh!

Flamefire commented Sep 4, 2025

Uh oh!

boegel commented Nov 19, 2025

Uh oh!

Flamefire commented Nov 20, 2025

Uh oh!

Flamefire commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented Nov 21, 2025

Uh oh!

boegel commented Dec 3, 2025

Uh oh!

Flamefire commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Flamefire commented Jan 19, 2023 •

edited by branfosj

Loading

Flamefire commented Nov 21, 2025 •

edited

Loading

Flamefire commented Dec 3, 2025 •

edited

Loading