remove default threshold in test

[AdriMarteau]

Setting default threshold like 1 for accuracy test can have some limitation or risks.

On the risk side, a model with 100% accuracy is most likely a spurious one and the test would pass.
Also, this make it less transparent as to the parameters reported when executing the test like in Suite.run.

Removing the default value would also raise an error when defining the test and force the developper to better think about a proper level.

[henchaves]

Hey @kevinmessiaen and @mattbit, what is your opinion on that?

[mattbit]

Sorry, this fell through the cracks. I agree with @AdriMarteau, not providing a default value would force the user to define their own threshold, which is a good thing for this type of test.

However, for backward compatibility it may be inappropriate to change that behavior now. I also think we have situations where the test params are passed at the suite level, and we want to instantiate the test without passing parameters — this needs to be verified.

In my opinion, the best tradeoff for now is to keep the threshold at 1.0, which would make the test (almost) always fail by default. The only case is the spurious 100% accuracy result, but even if the user sets a lower threshold this would pass undetected. Sadly there's not much we can do in that specific case except maybe logging a warning if measured accuracy is 1.