Skip to content

Add --lf/--failed to re-run only previously-failed tests#1653

Open
mvdbeek wants to merge 1 commit into
galaxyproject:masterfrom
mvdbeek:lf-style-tests
Open

Add --lf/--failed to re-run only previously-failed tests#1653
mvdbeek wants to merge 1 commit into
galaxyproject:masterfrom
mvdbeek:lf-style-tests

Conversation

@mvdbeek

@mvdbeek mvdbeek commented Jun 14, 2026

Copy link
Copy Markdown
Member

Summary

The --failed flag has been defined in planemo test since 2015 but was never
wired into the modern execution pipeline — the only code that referenced it
(GalaxyTestCommand.build()) is dead code left over from the old run_tests.sh
approach.

This PR implements the flag properly:

  • --failed / --lf: filters test cases before execution to only those
    whose IDs appear as non-success in a previous tool_test_output.json.
    --lf is added as a pytest-compatible alias.
  • --failed_json: optional separate path to read the previous results from,
    distinct from --test_output_json. This is needed when the planemo-ci-action
    test loop runs multiple tool groups, each writing their own output JSON, while
    all reading from the same previous chunk-level JSON for filtering.

How the filter works

Test case IDs follow the pattern {tool_id}_{index} (e.g. bwa_mem_0).
StructuredData.failed_ids already returns these from the JSON; the new code
in BaseEngine.test() filters the gathered test cases against that set before
handing them to the engine.

planemo-ci-action changes

A companion PR to planemo-ci-action
adds a retest mode and a previous-run-id input. In retest mode the action:

  1. Downloads the Tool test output {chunk} artifact from the specified previous
    run via gh run download
  2. Finds the same tool groups for this chunk
  3. Runs planemo test --failed --failed_json previous_run_results/tool_test_output.json
    for each group

This lets IUC weekly test maintainers re-run only the failing tests from a
previous scheduled run by triggering a workflow_dispatch with the failed run's
ID, without re-testing everything that already passed.

Usage

# normal run
planemo test tool.xml --test_output_json results.json

# re-run only failures (reads results.json, overwrites it)
planemo test tool.xml --lf --test_output_json results.json

# separate input and output (useful in CI loops)
planemo test tool.xml --lf --failed_json prev.json --test_output_json rerun.json

Test plan

  • Unit tests in tests/test_last_failed.py covering:
    • StructuredData.failed_ids includes failure/error/skip, excludes success
    • Without --failed, all test cases pass through unchanged
    • With --failed, only previously-failed cases are submitted to the engine
    • --failed_json takes precedence over --test_output_json as input source
    • Missing JSON raises a clear ClickException
    • All-passed previous run returns empty result immediately (no engine calls)

Wire up the long-defined but unimplemented --failed flag in BaseEngine.test()
so planemo skips test cases that passed in the prior run.  Also adds:
- --lf as a pytest-compatible alias for --failed
- --failed_json to separate the input JSON (previous results) from the
  output --test_output_json, which is needed for the planemo-ci-action
  per-tool-group loop where multiple groups share one previous results file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant