Add --lf/--failed to re-run only previously-failed tests#1653
Open
mvdbeek wants to merge 1 commit into
Open
Conversation
Wire up the long-defined but unimplemented --failed flag in BaseEngine.test() so planemo skips test cases that passed in the prior run. Also adds: - --lf as a pytest-compatible alias for --failed - --failed_json to separate the input JSON (previous results) from the output --test_output_json, which is needed for the planemo-ci-action per-tool-group loop where multiple groups share one previous results file
Open
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
--failedflag has been defined inplanemo testsince 2015 but was neverwired into the modern execution pipeline — the only code that referenced it
(
GalaxyTestCommand.build()) is dead code left over from the oldrun_tests.shapproach.
This PR implements the flag properly:
--failed/--lf: filters test cases before execution to only thosewhose IDs appear as non-success in a previous
tool_test_output.json.--lfis added as a pytest-compatible alias.--failed_json: optional separate path to read the previous results from,distinct from
--test_output_json. This is needed when the planemo-ci-actiontest loop runs multiple tool groups, each writing their own output JSON, while
all reading from the same previous chunk-level JSON for filtering.
How the filter works
Test case IDs follow the pattern
{tool_id}_{index}(e.g.bwa_mem_0).StructuredData.failed_idsalready returns these from the JSON; the new codein
BaseEngine.test()filters the gathered test cases against that set beforehanding them to the engine.
planemo-ci-action changes
A companion PR to planemo-ci-action
adds a
retestmode and aprevious-run-idinput. Inretestmode the action:Tool test output {chunk}artifact from the specified previousrun via
gh run downloadplanemo test --failed --failed_json previous_run_results/tool_test_output.jsonfor each group
This lets IUC weekly test maintainers re-run only the failing tests from a
previous scheduled run by triggering a
workflow_dispatchwith the failed run'sID, without re-testing everything that already passed.
Usage
Test plan
tests/test_last_failed.pycovering:StructuredData.failed_idsincludes failure/error/skip, excludes success--failed, all test cases pass through unchanged--failed, only previously-failed cases are submitted to the engine--failed_jsontakes precedence over--test_output_jsonas input sourceClickException