Using coverage with Cython file

Python package Coverage allows for collecting coverage of Python codebase exercosed by running a test suite.

Cython provides Coverage plug-in, allowing one to collect coverage of Cython codebase as well.

Sample project cython-coverage-setuptools was created following instructions from Stefan Behnel's 2015 blog "Line Coverage Analysis for Cython Modules" for projects that use setuptools to packaging.

It works as advertised:

cd cython-coverage-setuptools
python setup.py develop
coverage run -m pytest tests
coverage report
pip uninstall -y tasket && cd ..

The coverage report output indeed shows both the Python and the Cython files:

[cython-coverage-setuptools] $ coverage report
Name                 Stmts   Miss Branch BrPart  Cover
------------------------------------------------------
tasket/__init__.py       3      0      0      0   100%
tasket/_comp.pyx         5      0      0      0   100%
tasket/_hello.py         8      0      2      0   100%
------------------------------------------------------
TOTAL                   16      0      2      0   100%

Recently I worked on a transitioning a project, IntelPython/dpctl, from using setuptools to scikit-build. After the change has been merged I noticed that all Cython files disappeared from the coverage report.

This project was started to figure out the underlying cause and to find a fix.

This README.md serves to document the journey and the fix.

The reproducer project can be found in cython-coverage-scikit-build folder:

cd cython-coverage-scikit-build
python setup.py develop -- -G 'Unix Makefiles' -DCMAKE_C_COMPILER:PATH=$(which gcc) -DCMAKE_CXX_COMPILER:PATH=$(which g++)
coverage run -m pytest tests
coverage report
pip uninstall -y tasket && cd ..

This time the output of coverage report did not list the Cython file:

[cython-coverage-scikit-build] $ coverage report
Name                 Stmts   Miss Branch BrPart  Cover
------------------------------------------------------
tasket/__init__.py       3      0      0      0   100%
tasket/_hello.py         8      0      2      0   100%
------------------------------------------------------
TOTAL                   11      0      2      0   100%

For people interested in the solution, the fix can be found in cython-coverage-fix.patch:

# apply the fox
git apply cython-coverage-fix.patch
cd cython-coverage-scikit-build
git clean -dfx
python setup.py develop -- -G 'Unix Makefiles' -DCMAKE_C_COMPILER:PATH=$(which gcc) -DCMAKE_CXX_COMPILER:PATH=$(which g++)
coverage run -m pytest tests
coverage report

which now reports the coverage of the Cython file:

[cython-coverage-scikit-build] $ coverage report
Name                 Stmts   Miss Branch BrPart  Cover
------------------------------------------------------
tasket/__init__.py       3      0      0      0   100%
tasket/_comp.pyx         5      0      0      0   100%
tasket/_hello.py         8      0      2      0   100%
------------------------------------------------------
TOTAL                   16      0      2      0   100%

Now for the explanation (to the base of my understanding).

As the Stefan's blog points out the Cython.Coverage plugin expects to find the generated C++ source files next to their respective .pyx files, and hence the CMake scripts needs to perform that step, since scikit-build saves the generated sources in _skbuild/*/cmake-build folder.

Additionally, scikit-build invokes ${CYTHON_EXECUTABLE} command without explicitly setting its --working-directory. In absence of this setting the Cython is unable to infer the relative path of the *.pyx file in the project layout and simply inserts the filename in __pyx_f array used in line tracing calls.

For example, before the fix is applied, _comp.cxx generated

static const char *__pyx_f[] = {
  "_comp.pyx",
};

while in the _comp.cpp generated by setuptools-driven cythonize, and in the _comp.cxx after the fix was applied, the array is as follows:

static const char *__pyx_f[] = {
  "tasket/_comp.pyx",
};