Consolidate limit/offset logic in partition func #598
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
charlesbluca
commented
Jun 24, 2022
| """ | ||
| # if no offset is specified we can use `head` to compute the window | ||
| if not offset: | ||
| return df.head(limit, npartitions=-1, compute=False) |
Collaborator
Author
There was a problem hiding this comment.
@ayushdg not sure if you'd prefer we check the first partition first to see if we can avoid the npartitions=-1 here?
Collaborator
There was a problem hiding this comment.
I think it's fine to keep it as is and revisit if we encounter perf issues.
Codecov Report
@@ Coverage Diff @@
## datafusion-sql-planner #598 +/- ##
=========================================================
Coverage ? 60.88%
=========================================================
Files ? 71
Lines ? 3566
Branches ? 730
=========================================================
Hits ? 2171
Misses ? 1246
Partials ? 149 📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
ayushdg
approved these changes
Jun 24, 2022
Collaborator
ayushdg
left a comment
ayushdg
left a comment
There was a problem hiding this comment.
Thanks a lot. Changes lgtm!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This restores the original consolidated computation of limit/offsets in
limit_partition_func, while also opting to keep the simplified computation of limits when an offset is not specified, which is what we opted for in #529.Closes #597