-
Notifications
You must be signed in to change notification settings - Fork 72
Support predicate-pushdown for isin operations
#1160
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## main #1160 +/- ##
==========================================
+ Coverage 81.30% 81.53% +0.22%
==========================================
Files 78 78
Lines 4414 4474 +60
Branches 799 817 +18
==========================================
+ Hits 3589 3648 +59
+ Misses 649 641 -8
- Partials 176 185 +9
... and 1 file with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
isin operationsisin operations
|
@ayushdg @randerzander - Note that I added |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes generally lgtm!
Doesn't need to be in this pr, but as a followup it might also make sense to add simpler unit tests that just use the attempt predicate pushdown method on dask dataframes and add them to https://github.com/dask-contrib/dask-sql/tree/main/tests/unit
…into isin-filtering
ayushdg
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for looking into this @rjzamora!
Closes #607
This PR adds predicate-pushdown support for
dd._Frame.isin(values), and uses a somewhat-ugly workaround to account for the fact thatdask.dataframewill produce a distinctMaterializedLayerobject for thevaluesargument toisin.Note that this PR also addresses
"not in"support. However,dask.dataframewill produce the wrong result for these filters until dask/dask#10320 is merged.