pytest.skip() results in UnexpectedException error

[effigies]

In pytest-dev/pytest#4927, it became possible to use pytest.skip() within a doctest.

Passing/skipping pytest output

$ pytest -sv --doctest-modules --pyargs niworkflows -k niworkflows.reports.core    
==================================================== test session starts =====================================================
platform linux -- Python 3.10.10, pytest-7.2.2, pluggy-1.0.0 -- /home/chris/mambaforge/envs/fmriprep-dev/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/chris
plugins: doctestplus-0.12.1
collected 391 items / 387 deselected / 4 selected                                                                            

mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Report SKIPPED
mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Reportlet SKIPPED
mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Smallest PASSED
mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.run_reports SKIPPED

======================================== 1 passed, 3 skipped, 387 deselected in 3.32s ========================================

Adding --doctest-plus causes these skips to be reported as UNEXPECTED EXCEPTION:

Failing pytest call with `--doctest-plus`

$ pytest -sv --doctest-modules --pyargs niworkflows -k niworkflows.reports.core --doctest-plus
==================================================== test session starts =====================================================
platform linux -- Python 3.10.10, pytest-7.2.2, pluggy-1.0.0 -- /home/chris/mambaforge/envs/fmriprep-dev/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/chris
plugins: doctestplus-0.12.1
collected 391 items / 387 deselected / 4 selected                                                                            

mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Report FAILED
mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Reportlet FAILED
mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Smallest PASSED
mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.run_reports FAILED

========================================================== FAILURES ==========================================================
_________________________________________ [doctest] niworkflows.reports.core.Report __________________________________________
248 
249 
250     .. testsetup::
251 
252         >>> cwd = os.getcwd()
253         >>> os.chdir(tmpdir)
254 
255         >>> from shutil import copytree
256         >>> from bids.layout import BIDSLayout
257         >>> test_data_path = find_resource_or_skip('data/tests/work')
UNEXPECTED EXCEPTION: Missing resource data/tests/work; run this test from a source repository
Traceback (most recent call last):
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest niworkflows.reports.core.Report[4]>", line 1, in <module>
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/conftest.py", line 45, in find_resource_or_skip
    pytest.skip(f"Missing resource {resource}; run this test from a source repository")
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/_pytest/outcomes.py", line 179, in skip
    raise Skipped(msg=reason, allow_module_level=allow_module_level)
Skipped: Missing resource data/tests/work; run this test from a source repository
/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py:257: UnexpectedException
________________________________________ [doctest] niworkflows.reports.core.Reportlet ________________________________________
099     object.
100 
101     .. testsetup::
102 
103         >>> cwd = os.getcwd()
104         >>> os.chdir(tmpdir)
105 
106         >>> from shutil import copytree
107         >>> from bids.layout import BIDSLayout
108         >>> test_data_path = find_resource_or_skip('data/tests/work')
UNEXPECTED EXCEPTION: Missing resource data/tests/work; run this test from a source repository
Traceback (most recent call last):
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest niworkflows.reports.core.Reportlet[4]>", line 1, in <module>
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/conftest.py", line 45, in find_resource_or_skip
    pytest.skip(f"Missing resource {resource}; run this test from a source repository")
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/_pytest/outcomes.py", line 179, in skip
    raise Skipped(msg=reason, allow_module_level=allow_module_level)
Skipped: Missing resource data/tests/work; run this test from a source repository
/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py:108: UnexpectedException
_______________________________________ [doctest] niworkflows.reports.core.run_reports _______________________________________
517 
518     Run the reports.
519 
520     >>> pytest.skip()
UNEXPECTED EXCEPTION: 
Traceback (most recent call last):
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest niworkflows.reports.core.run_reports[0]>", line 1, in <module>
  File "/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/_pytest/outcomes.py", line 179, in skip
    raise Skipped(msg=reason, allow_module_level=allow_module_level)
Skipped
/home/chris/mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py:520: UnexpectedException
================================================== short test summary info ===================================================
FAILED mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Report
FAILED mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.Reportlet
FAILED mambaforge/envs/fmriprep-dev/lib/python3.10/site-packages/niworkflows/reports/core.py::niworkflows.reports.core.run_reports
======================================== 3 failed, 1 passed, 387 deselected in 3.31s =========================================

My particular use-case is that I have some large test files in my source repository. I exclude them from the wheel/installed package to avoid excessively large site-packages/ directories, so I've written fixtures/helper functions to call pytest.skip() if an excluded test file/directory is requested.

I had a quick look through your source code, and couldn't see a place to make a similar patch to the upstream one. Assuming this is something you'd be open to supporting, if you can point me in the right direction, I'm happy to submit a PR.

[pllim]

If the logic to fix this is indeed here and not upstream, then it would have to be in https://github.com/astropy/pytest-doctestplus/blob/main/pytest_doctestplus/plugin.py .

I am not very sure how to fix either. The upstream change you pointed out touched a method in PytestDoctestRunner that inherits from doctest.DebugRunner. Here, we use an instance of DebugRunner directly.

Feel free to propose a fix. Otherwise maybe we would have to ping pytest developers for hints.

[effigies]

Okay, I went ahead and read their subclass.

They've made it pretty much impossible to subclass that, but I can replicate what they did here by subclassing DebugRunner. Would you be interested in the continue-on-error code they've got in there, as well?

[pllim]

I think @bsipocz had asked for a continue-on-error feature before but I forgot if it is this plugin or some other one. Though if it is not technically needed to fix your particular problem here, it is out of scope. 🤷

[bsipocz]

yeap, it's here: #188

Makes no sense in some cases, when the context is not restarted, but sometimes when the actual doctests sections are independent it could be a super useful.

[effigies]

I'm happy to include something like #188/#44 in #196 or open a new PR. Whichever approach suits y'all.

[pllim]

Re: #196 -- Let's keep one about skip only, so it is easier to review. Thanks!

[bsipocz]

Agree with @pllim, it's much easier to handle multiple small PR than an octopus-like big one. If you need to rely on the content of an earlier PR before it's merged, it's fine to build on top of that, we can always rebased once things end up in main, yet in that approach, the additional features/fixes are not holding up the earlier ones.