Add row_id and prefetch to parquet reader
#65
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pache#6041)" This reverts commit 741bbf6. # Conflicts: # arrow-flight/Cargo.toml # arrow-flight/gen/Cargo.toml # arrow-flight/src/arrow.flight.protocol.rs # arrow-integration-testing/Cargo.toml
This reverts commit 2983dc1.
This reverts commit 244d8bd.
…pache#6041)" This reverts commit 7750691.
nathanielc
approved these changes
May 6, 2025
nathanielc
left a comment
nathanielc
left a comment
There was a problem hiding this comment.
LGTM. I read through the tests and they are great. I don't have enough context on the implementation to give much feedback but it was pretty straightforward.
| batch_size, | ||
| array_reader, | ||
| apply_range(selection, reader.num_rows(), self.offset, self.limit), | ||
| // TODO what do we do here? |
There was a problem hiding this comment.
Did you figure out if None was fine here?
Collaborator
Author
There was a problem hiding this comment.
It is fine for us because we only use the sync reader but a challenge for upstreaming.
avantgardnerio
pushed a commit
that referenced
this pull request
Dec 17, 2025
avantgardnerio
pushed a commit
that referenced
this pull request
Dec 17, 2025
avantgardnerio
pushed a commit
that referenced
this pull request
Dec 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of topk optimizations in DQE:
row_idcolumn which will contain the absolute row offset of each returned row in the underlying file. This is what allows us to build a row selection once we have a topkprefetchcapability which allows us to prefetch columns when evaluating row filters. The idea being that it could be better to prefetch the columns we need to decode (if they are small) rather than wait for the row selection returned from our row filters. Did not end up using this but I think it may be useful if used more intelligently in DQE so leaving it in