Skip to content

Commit adcecf4

Browse files
mapleFUMottl
authored andcommitted
apacheGH-34785: [C++][Parquet] Add bloom filter write support (apache#37400)
### Rationale for this change Currently we allow reading bloom filter for specific column and row group. Now this patch allows writing bloom filters to Parquet files. ### What changes are included in this PR? Allow writing bf: * [x] Add WriterProperties config for writing bloom filter, including bf and (per-rowgroup) ndv estimation. * [x] Add BloomFilterBuilder for parquet * [x] From FileSerializer to ColumnWriter, adding bloomfilter * [x] Ensure Bloom Filter info is written to the file * [x] Testing logic for BloomFilterBuilder * [x] Testing logic for BloomFilter and ColumnWriter * [x] Testing whole roundtrip like `ParquetPageIndexRoundTripTest` ### Are these changes tested? Yes ### Are there any user-facing changes? User can create Bloom Filter in parquet with C++ api * Closes: apache#34785 * GitHub Issue: apache#34785 Lead-authored-by: Gang Wu <ustcwg@gmail.com> Co-authored-by: mwish <maplewish117@gmail.com> Co-authored-by: mwish <anmmscs_maple@qq.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
1 parent f98bf64 commit adcecf4

25 files changed

Lines changed: 1701 additions & 610 deletions

cpp/src/parquet/CMakeLists.txt

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ set(PARQUET_SRCS
161161
arrow/writer.cc
162162
bloom_filter.cc
163163
bloom_filter_reader.cc
164+
bloom_filter_writer.cc
164165
chunker_internal.cc
165166
column_reader.cc
166167
column_scanner.cc
@@ -373,7 +374,7 @@ set_source_files_properties(public_api_test.cc PROPERTIES SKIP_UNITY_BUILD_INCLU
373374

374375
add_parquet_test(internals-test
375376
SOURCES
376-
bloom_filter_reader_test.cc
377+
bloom_filter_reader_writer_test.cc
377378
bloom_filter_test.cc
378379
geospatial/statistics_test.cc
379380
geospatial/util_internal_test.cc
@@ -409,6 +410,8 @@ add_parquet_test(arrow-reader-writer-test
409410
arrow/arrow_statistics_test.cc
410411
arrow/variant_test.cc)
411412

413+
add_parquet_test(arrow-index-test SOURCES arrow/index_test.cc)
414+
412415
add_parquet_test(arrow-internals-test SOURCES arrow/path_internal_test.cc
413416
arrow/reconstruct_internal_test.cc)
414417

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

Lines changed: 0 additions & 381 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)