[HUDI-5534] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table#7642

Merged

alexeykudinkin merged 28 commits intoapache:masterfrom

onehouseinc:ak/blm-idx-dag-opt

Jan 27, 2023

Commits on Jan 26, 2023

Rebasing MT Bloom Index partition listing to do bulk-request to MT listing all partition
Alexey Kudinkin
committed
Revisit Bloom Index to avoid superfluous conversions
Alexey Kudinkin
committed
Adding missing utils to `HoodieData` hierarchy
Alexey Kudinkin
committed
Streamline Bloom Index to pass around `HoodiePairData` to avoid unnecessary (re-)mapping
Alexey Kudinkin
committed
Fixing `ClassCastException`
Alexey Kudinkin
committed
Fixed `BloomFilter` deserialization seq to avoid mutating source `ByteBuffer`
Alexey Kudinkin
committed
[WIP] Split Bloom Index lookup in MT and actual file probing to allow them to run w/ different parallelism levels;
Alexey Kudinkin
committed
Rebased Bloom Index to rely on full `HoodieFileGroupId` instead of just the `FileGroupId` (which ain't unique across partitions);
Alexey Kudinkin
committed
Added static FS view as broadcasted value (to avoid the need to fire HTTP request for every file probed, killing performance);
Alexey Kudinkin
committed
De-duplicated `HoodieBloomIndexCheckFunction` to be re-used for List/Spark helpers
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Use method rewriting the record that skips the validation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Reverting inadvertent changes
Alexey Kudinkin
committed
Fixing compilation for Scala 2.11
Alexey Kudinkin
committed
Scaffolded `HoodieRDDUtils` to handle incompatible change in Spark 3.x API
Alexey Kudinkin
committed
Fixed non-serializable lambdas
Alexey Kudinkin
committed
Killing dead code
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Introduced `Transient` to auto-reset after being serialized based on provided serializable initializer
Alexey Kudinkin
committed
Cleaned up `BaseTableMetadata` to avoid holding `SerializableConfiguration` explicitly;
Alexey Kudinkin
committed
Made `HoodieBackedTableMetadata` serializable even when reusing the underlying MT readers
Alexey Kudinkin
committed
Simplified and cleaned up `FlatteningIterator`;
Alexey Kudinkin
committed
Added workaround to avoid re-processing whole partition for every file-group looked up
Alexey Kudinkin
committed
Adding java-docs
Alexey Kudinkin
committed
[XXX] Avoid repartitioning multiple times by using the same parallelism to read MT
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HUDI-5534] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table#7642

[HUDI-5534] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table#7642
alexeykudinkin merged 28 commits intoapache:masterfrom
onehouseinc:ak/blm-idx-dag-opt

Commits on Jan 26, 2023

Rebasing MT Bloom Index partition listing to do bulk-request to MT listing all partition

Revisit Bloom Index to avoid superfluous conversions

Adding missing utils to `HoodieData` hierarchy

Streamline Bloom Index to pass around `HoodiePairData` to avoid unnecessary (re-)mapping

Fixing `ClassCastException`

Fixed `BloomFilter` deserialization seq to avoid mutating source `ByteBuffer`

[WIP] Split Bloom Index lookup in MT and actual file probing to allow them to run w/ different parallelism levels;

Rebased Bloom Index to rely on full `HoodieFileGroupId` instead of just the `FileGroupId` (which ain't unique across partitions);

Added static FS view as broadcasted value (to avoid the need to fire HTTP request for every file probed, killing performance);

De-duplicated `HoodieBloomIndexCheckFunction` to be re-used for List/Spark helpers

Tidying up

Use method rewriting the record that skips the validation

Tidying up

Reverting inadvertent changes

Fixing compilation for Scala 2.11

Scaffolded `HoodieRDDUtils` to handle incompatible change in Spark 3.x API

Fixed non-serializable lambdas

Killing dead code

Fixing compilation

Introduced `Transient` to auto-reset after being serialized based on provided serializable initializer

Cleaned up `BaseTableMetadata` to avoid holding `SerializableConfiguration` explicitly;

Made `HoodieBackedTableMetadata` serializable even when reusing the underlying MT readers

Simplified and cleaned up `FlatteningIterator`;

Added workaround to avoid re-processing whole partition for every file-group looked up

Adding java-docs

[XXX] Avoid repartitioning multiple times by using the same parallelism to read MT

Tidying up

Tidying up