Rerun failed tests up to 5 times before complete fail

[Marilyth]

Band aid fix for all the random failing tests on GitHub.
If tests fail, give them 5 more chances to pass before declaring the action failed.
Does not get rid of the occasional xdist hang up.

[joernu76]

I run sometimes into timeouts already now (at least with my modified time check. NFS??). How does this relate to our maximal runtime?

[joernu76]

What about
https://docs.pytest.org/en/6.2.x/flaky.html

via pytest.mark.xfail?

Especially this deeper link

https://github.com/pytest-dev/pytest-rerunfailures
Seems to do something similar, but allows more fine grained control via

@pytest.mark.flaky(reruns=5)

[Marilyth]

Some testing with pytest-rerunfailures caused me a lot of headache and segmentation faults after x reruns.
Considering pytest comes with failure reruns already, I don't see the need for it.

The runtime increases with each failed test. Running them again 5 times will take maybe 10 seconds of overhead + the time needed for the tests * 5. If it passes after 1 retry then the test finishes right there.
It isn't much of an increase.

[ReimarBauer]

see comment

[ReimarBauer]

looks good :)