Make tests run faster

[lhoestq]

From 7min to 2min to run pytest.
Ideally we should keep the whole CI run time below 10min.

In this PR I removed the remote tests that were never used.
I also replaced nested parametrized tests with unit tests.
This makes me think that we could still add more high level tests to check for a few combinations of parameters (but not all of them since there are too many of them).
Let me know what you think

Finally in another PR we can also separate in two circleci jobs:

• the tests of the code code of the lib
• the tests of the all the dataset/metric scripts.

[albertvillanova]

LOL, I was also working on something similar 😅. I'm gonna have a look!!!

[albertvillanova]

I totally agree that test time MUST be small. Our test suite was starting to last too long and this must be fixed.

On the other hand, I think we should increase our test coverage: I think there are many parts of the code without being tested and this must be also addressed.

I would suggest dividing our test suite into real unit tests (super fast) and integration tests (slower tests):

• for real unit tests, we can be exhaustive and try to increase the coverage, as they are really fast
• for integration tests, we must be more cautious and we should select some general cases and only test those those combinations
• we could even keep some end-to-end tests (like load_dataset), but very few

I think this PR goes in the right direction, but we should continue improving real unit test (at high speed) coverage.

Feel free to tell me your opinion and how we could keep delving into this matter.

[lhoestq]

Sorry I didn't know you were also working on it ^^'
And yes I 100% agree with you on the points you mentioned. We should definitely improve the coverage. It would be nice to have a clearer separation to know which tests in the suite are unit tests and which ones are integration tests

[albertvillanova]

Never mind: we both noticed tests can be improved. More PRs to come... 😉

According to the literature, unit tests are those that test a behavior unit, isolated from the other components and must be very fast: for me, this last requirement implies that they must be performed completely in memory.

As opposed, integration tests are those which also test interactions with external components, like web services, databases, file system, etc.

The problem I see is that our code is still too coupled and it is difficult to isolate components for testing. Therefore, I would suggest acting iteratively, by refactoring to decouple components and then implement unit tests for each component in isolation.