[HUDI-3457] Refactored Spark DataSource Relations to avoid code duplication#4877

Merged

xushiyan merged 33 commits intoapache:masterfrom

onehouseinc:ak/spkds-ref-2

Mar 19, 2022

Commits on Mar 16, 2022

Abstracted & unified `buildScan` functionality for COW/MOR Relations;
Alexey Kudinkin
committed
`BaseFileOnlyViewRelation` > `BaseFileRelation`
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Extracted common converter utils to `HoodieCommonUtils`;
Alexey Kudinkin
committed
Abstracted common functionality;
Alexey Kudinkin
committed
Extracted common functionality to lists latest base files into `HoodieBaseRelation`
Alexey Kudinkin
committed
Streamlined `MergeOnReadSnapshotRelation` to re-use common functionality, avoid duplication
Alexey Kudinkin
committed
Killing dead code;
Alexey Kudinkin
committed
Further simplified `MergeOnReadSnapshotRelation`
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Cleaned up & streamlined `MergeOnReadIncrementalRelation`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Extract most of the incremental-specific aspects into a trait that could be shared w/ COW impl
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Cleaning up unnecessary filtering
Alexey Kudinkin
committed
After rebase fixes
Alexey Kudinkin
committed
Scaffolded `HoodieInMemoryFileIndex` and replicated `HoodieHadoopFSUtils` to override default behavior of InMemoryFileIndex filtering out all files stated w/ "."
Alexey Kudinkin
committed
Fixed usages
Alexey Kudinkin
committed
Moved tests
Alexey Kudinkin
committed
Missing licenses
Alexey Kudinkin
committed
Disabling linter
Alexey Kudinkin
committed
Fixed compilation for Spark 2.x
Alexey Kudinkin
committed
Added missing scala-docs
Alexey Kudinkin
committed
Fixed incorrect casting
Alexey Kudinkin
committed
Fixed partition path handling for MOR Incremental Relation
Alexey Kudinkin
committed
Fixed `HoodieIncrementalRelationTrait` to extend `HoodieBaseRelation` to be able to appropriately invoke super
Alexey Kudinkin
committed
Handle the case when there are no commits to handle in Incremental Relation
Alexey Kudinkin
committed
Return empty RDD in case there's no file-splits to handle
Alexey Kudinkin
committed
Cleaned up `listLatestBaseFiles`
Alexey Kudinkin
committed
Added TODO
Alexey Kudinkin
committed
Fixing file handle leak
Alexey Kudinkin
committed
Disabled vectorized reader to make sure MOR Incremental Relation works properly
Alexey Kudinkin
committed
Fixed Parquet column-projection tests
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[HUDI-3457] Refactored Spark DataSource Relations to avoid code duplication#4877

[HUDI-3457] Refactored Spark DataSource Relations to avoid code duplication#4877
xushiyan merged 33 commits intoapache:masterfrom
onehouseinc:ak/spkds-ref-2

Commits on Mar 16, 2022

Abstracted & unified `buildScan` functionality for COW/MOR Relations;

`BaseFileOnlyViewRelation` > `BaseFileRelation`

Fixing compilation

Extracted common converter utils to `HoodieCommonUtils`;

Abstracted common functionality;

Extracted common functionality to lists latest base files into `HoodieBaseRelation`

Streamlined `MergeOnReadSnapshotRelation` to re-use common functionality, avoid duplication

Killing dead code;

Further simplified `MergeOnReadSnapshotRelation`

`lint`

Cleaned up & streamlined `MergeOnReadIncrementalRelation`

Tidying up

Extract most of the incremental-specific aspects into a trait that could be shared w/ COW impl

Fixing compilation

Cleaning up unnecessary filtering

After rebase fixes

Scaffolded `HoodieInMemoryFileIndex` and replicated `HoodieHadoopFSUtils` to override default behavior of InMemoryFileIndex filtering out all files stated w/ "."

Fixed usages

Moved tests

Missing licenses

Disabling linter

Fixed compilation for Spark 2.x

Added missing scala-docs

Fixed incorrect casting

Fixed partition path handling for MOR Incremental Relation

Fixed `HoodieIncrementalRelationTrait` to extend `HoodieBaseRelation` to be able to appropriately invoke super

Handle the case when there are no commits to handle in Incremental Relation

Return empty RDD in case there's no file-splits to handle

Cleaned up `listLatestBaseFiles`

Added TODO

Fixing file handle leak

Disabled vectorized reader to make sure MOR Incremental Relation works properly

Fixed Parquet column-projection tests