mark/fixture to turn coverage recording off

[The-Compiler]

I don't know how feasible this is, but it'd be nice to have a mark and/or contextmanager-fixture to turn coverage collection off:

@pytest.mark.coverage_disabled
def test_foo():
    # do some integration test or whatever
    pass

def test_bar(coverage):
    with coverage.disabled():
        pass

[ionelmc]

Definitely doable but just to understand your use case: does pytest-cov make your integration much slower?

Also, you can always have a no_coverage fixture:

@pytest.yield_fixture
def no_coverage():
    tracer = sys.gettrace()
    sys.settrace(None)
    yield
    sys.settrace(tracer)

[The-Compiler]

My use case is that I use coverage as a "todo-list" for the things I still need to write tests for (a lot...) - but when writing a broader integration test, I'd like that test to not "distort" my coverage.

[ionelmc]

In my projects integration tests tend to be the best kind of tests to measure feature coverage.

Unit/component tests have the nasty tendency to rely on internals so even if they do cover code they don't really cover the feature. Their only advantage is speed.

Would the fixture solve your usecase?

PS. The pytest_runtest_call hook can be used to get the same effect, eg:

def pytest_runtest_call(item, __multicall__): # TODO: use hookwrapper
    if node.get_marker("no_coverage"):
        tracer = sys.gettrace()
        sys.settrace(None)
        __multicall__.execute()
        sys.settrace(tracer)
    else:
        __multicall__.execute()

[ionelmc]

Alternatively, you can put integration tests in a different directory (that's what I do for one large internal project).

Here's another example: https://github.com/ionelmc/virtualenv/tree/develop/tests (but there Donald did the layout and automarker thing in conftest).

[ionelmc]

Do you think other people would make use of such a marker? How would you name it? no_coverage? uncovered?

[The-Compiler]

Let me try to explain one of my usecases in more detail - I have a module configtypes which define types which values in a config file can have. Those have a transform and a validate method, and could look somewhat like this:

class Bool(BaseType):

    def validate(self, value):
        if value not in ['true', 'false']:
            raise ValidationError

    def transform(self, value):
        ...

Now I have a default config, and I want to make sure the default config is valid, so I validate it which runs all the config type's validate functions - yet those are not actually tested. Here for example I might want to test True and discover I should probably use value.lower() instead. But the default config test marks it as covered.

I have no idea if people would use it in general - I asked in #pylib, let's see if/how people respond.

[ionelmc]

Just a note here: if we reinstall the tracer it's going to be way slower on reinstall. More details: https://bitbucket.org/ned/coveragepy/issues/397/stopping-and-resuming-coverage-with

@nedbat won't release a fix for this very soon so best to wait on this for a while?

[The-Compiler]

This was fixed in coverage 4.0.1 - thanks @nedbat!

[ionelmc]

Ok lets find a good name for this marker.

Which one?

• pytest.mark.nocover
• pytest.mark.nocoverage
• pytest.mark.skipcover
• pytest.mark.skipcoverage
• pytest.mark.no_covererage
• pytest.mark.skip_coverage

[nicoddemus]

Since the special comments to exclude lines is # pragma: no cover, I would vote for pytest.mark.no_cover, and mention it in the documentation to help people remember by association. 😁

[The-Compiler]

I agree with @nicoddemus - pytest.mark.no_cover makes the most sense IMHO.