Skip to content

parquet reader: move pruning predicate creation from ParquetSource to ParquetOpener #15561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 15 commits into from
Apr 6, 2025
Merged

parquet reader: move pruning predicate creation from ParquetSource to ParquetOpener #15561

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
718bed3
parquet reader: move pruning predicate creation from ParquetSource to…
adriangb Apr 3, 2025
4655adc
use file schema, avoid loading page index if unecessary
adriangb Apr 3, 2025
a30b631
Add comment
adriangb Apr 3, 2025
398e7a4
add comment
adriangb Apr 3, 2025
05bdcf8
Add comment
adriangb Apr 3, 2025
9a38aec
remove check
adriangb Apr 3, 2025
f886645
fix clippy
adriangb Apr 3, 2025
2d486d5
update sqllogictest
adriangb Apr 3, 2025
bc9af12
restore to explain plans
adriangb Apr 3, 2025
3fd4b8a
reverted
adriangb Apr 3, 2025
c193d2e
modify access
adriangb Apr 3, 2025
cd6d766
Fix ArrowReaderOptions should read with physical_file_schema so we do…
zhuqi-lucas Apr 4, 2025
ad627d8
Update opener.rs
adriangb Apr 4, 2025
34993f2
Always apply per-file schema during parquet read (#18)
alamb Apr 4, 2025
c9aaa3d
Update datafusion/datasource-parquet/src/opener.rs
adriangb Apr 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions datafusion/core/src/datasource/file_format/parquet.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,13 +67,13 @@ pub(crate) mod test_util {
.into_iter()
.zip(tmp_files.into_iter())
.map(|(batch, mut output)| {
let builder = parquet::file::properties::WriterProperties::builder();
let props = if multi_page {
builder.set_data_page_row_count_limit(ROWS_PER_PAGE)
} else {
builder
let mut builder = parquet::file::properties::WriterProperties::builder();
if multi_page {
builder = builder.set_data_page_row_count_limit(ROWS_PER_PAGE)
}
.build();
builder = builder.set_bloom_filter_enabled(true);
Copy link
Contributor Author

@adriangb adriangb Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I'm now using the actual explain plan to check that bloom filters are used it's easiest to just enable them and check that they pruned. It's also a more realistic test in that it's what I as a user would do to check if bloom filters are working.


let props = builder.build();

let mut writer = parquet::arrow::ArrowWriter::try_new(
&mut output,
Expand Down
Loading