-
Notifications
You must be signed in to change notification settings - Fork 12
Thorough rewrite and optimization of integration tests #82
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
||
| @pytest.fixture | ||
| def vector_store_d2( | ||
| empty_collection_d2: Collection, # noqa: ARG001 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NIT: Is it possible to use @pytest.mark.usefixtures("empty_collection_d2") and get rid of the noqa ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch - filing for a later improvement since it's merged now (saw the comment only now, sorry)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I merged since it was NIT.
I'm doing a PR rn.
This PR completely restructures the integration tests for this project.
This was a much-needed improvement: the test suite had grown in a scarcely-controlled way, becoming bloated, very long-lasting and unstable (mostly due to the unnecessarily high number of collections being created and dropped during testing). Additionally, some of the tests were using obsolete constructs and duplicated assets.
The gist
This PR implements an extremely parsimonious usage of collections thanks to a hierarchy of fixtures (and ample usage of
setup_mode). Whenever possible, collections are simply truncated and re-used between tests. Some collections, which remain unavoidably per-test-function, are marked as such ("ephemeral..."). This is the case mostly for "DDL-related" tests, for instance when testing deprecations and errors due to indexing mismatches and such. For the case of vector-store tests, these are grouped in a separate module to skip them easily while developing.Numbers
The general idea is that there should never be more than 3 collections in the database, at any given time. (nevertheless, the collection names involved are way more, to avoid running into metadata-cache issues when dropping and re-creating them.)
Before this PR, the whole suite took about 40+ minutes to run (... and often fail). Now it takes 13 minutes. Specifically, the vector store tests alone went from 19 minutes down to about 7.
Additional notes
setup_modeis tested scarcely in itself (perhaps the only "loss of coverage" of this PR); however, it is implicitly tested essentially in all collection-related fixtures.