Skip to content

Improvements for parquet writing performance (25%-44%) #7824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Dandandan merged 8 commits into from
Jul 4, 2025
Merged

Improvements for parquet writing performance (25%-44%) #7824

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
990d49c
Optimize counting of values and nulls
jhorstmann Jun 29, 2025
f513036
Change asserts in BitWriter::put_value to debug_assert
jhorstmann Jun 29, 2025
e59c47b
Use slice iteration instead of indexing in flush_bit_packed_run
jhorstmann Jun 29, 2025
95b600e
Separate iteration for def_levels and non_null_indices
jhorstmann Jun 29, 2025
01c2507
Cache logical nulls of the array to avoid clones or repeated recomput…
jhorstmann Jun 29, 2025
d4c65a1
Change more asserts in RleEncoder to debug_assert
jhorstmann Jul 2, 2025
64d07d3
Optimize bloom filter initialization
jhorstmann Jul 2, 2025
e7f32e9
Write bloom filter as a single slice on little endian machines
jhorstmann Jul 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions arrow-buffer/src/util/bit_iterator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ impl<'a> BitIndexIterator<'a> {
impl Iterator for BitIndexIterator<'_> {
type Item = usize;

#[inline]
fn next(&mut self) -> Option<Self::Item> {
loop {
if self.current_chunk != 0 {
Expand Down
Loading
Loading