Code coverage

[ElliottKasoar]

As discussed elsewhere, and temporarily fixed by #610, coveralls has been having frequent outages over the past month or so.

We are therefore reevaluating how/whether we track coverage. There are a few options:

• Replace coveralls with another provider, which may or may not be more reliable
• Remove code coverage as a metric we track (how useful is it?)
• Keep coveralls, and accept some failures, now that it doesn't show up as a failure in the CI
  • This of course does not preclude revisiting this decision in a few months, depending on the ongoing service

I think having coverage for janus-core is a useful distinction compared to some alternatives, as it's quite a simple way to show we take the software engineering seriously, which includes testing.

For other projects that are more UI focused e.g. ML-PEG, ALC-UX, coverage is significantly harder, so perhaps in other cases we drop it.

If we do keep some measure for janus-core, I'm also still inclined to give it some time to see the direction things move, but I'm not against swapping if anyone suggests a reliable alternative.

[oerc0122]

I have doubts about the utility of coverage as a metric, particularly as a target. It can lead to useless tests just to up the metric which then eliminate the utility of knowing whether a region of code is appropriately tested, i.e. people go "Oh, well it's clearly covered."

Possibly more useful is just the published coverage report that you can ignore as a comment.

There are, however, other actions available for coverage (e.g. https://github.com/orgoro/coverage) if we desire it.

[ElliottKasoar]

From the dev perspective, we don't enforce any requirements on the coverage; it is purely informational. I don't think our current review process has ever led to lines of code being covered for that reason, rather than because we wanted to test the actual functionality, so I'm not too concerned by that aspect.

On the other hand, it is true that if lines aren't covered at all, we are potentially missing bugs etc, so seeing that we don't cover some parts of post_process.py, for example, doesn't hurt?

You could maybe argue there's a risk of users assuming there shouldn't be a bug because of coverage, but I don't think that's particularly realistic either.

[ElliottKasoar]

If we do keep coverage, I think having the actual code breakdown by line is useful (https://coveralls.io/github/stfc/janus-core), which I don't see in that alternative?