[HUDI-4588][HUDI-4472] Addressing schema handling issues in the write path#6358

Merged

alexeykudinkin merged 134 commits intoapache:masterfrom

onehouseinc:ak/prq-prj-fix

Nov 24, 2022

Commits on Nov 23, 2022

Revert "[MINOR] retain avro's namespace (#6783)"
Alexey Kudinkin
committed
Fixing `HoodieParquetReader` to properly specify projected schema when reading Parquet file
Alexey Kudinkin
committed
Converted `DataTypeUtils` to Scala;
Alexey Kudinkin
committed
Fixed passing of the schema into Spark 3 `DefaultSource` impl to rely on JSON, rather than DDL,
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Cleaned up test for basic Schema Evolution, added cases being tested
Alexey Kudinkin
committed
Added `AvroSchemaUtils.isNullable` check to validate whether Avro schema is nullable
Alexey Kudinkin
committed
Fixed `isSchemaCompatible` sequence to treat "dropped column" as legitimate op
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixed basic Schema Evolution test
Alexey Kudinkin
committed
Run basic schema-evoluation test for all operations
Alexey Kudinkin
committed
Fixed `HoodieSparkSqlWriter` to fix schema handling for row-writing path, unifying it b/w
Alexey Kudinkin
committed
Throw exception in case table and incoming schemas could not be reconciled
Alexey Kudinkin
committed
Extracted `TestBasicSchemaEvolution` as standalone test
Alexey Kudinkin
committed
Fixing handling of incompatible schemas
Alexey Kudinkin
committed
Extended tests to MOR
Alexey Kudinkin
committed
Fixing compilation for Spark 2.x
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing handling of partition-column being dropped
Alexey Kudinkin
committed
Fixed `testSchemaEvolutionForTableType` and expanded
Alexey Kudinkin
committed
Fixed more tests (inserting w/ incompatible schema)
Alexey Kudinkin
committed
Fixed latest table's schema derivation seq to
Alexey Kudinkin
committed
Added `RecordType.name` to be able to encode Avro Record's names for lossless
Alexey Kudinkin
committed
Fixed error message
Alexey Kudinkin
committed
Cleaned up schema bespoke compatibility validation seq, to instead rely on Avro's `SchemaCompatibility`
Alexey Kudinkin
committed
Fixing invalid test
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing `InternalSchema` <> Avro conversion for Avro 1.8.2
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Rebased `InternalSchema` ctors to accept `RecordType` in lieu of list of `Field`s
Alexey Kudinkin
committed
Make sure `RecordType`s name is appropriately propagated during modifications
Alexey Kudinkin
committed
Use full-name when converting back to Avro schema
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing tests (after rebase)
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Make sure all Avro schemas are properly registered w/ Kryo
Alexey Kudinkin
committed
`WRITE_SCHEMA` > `WRITE_SCHEMA_OVERRIDE`;
Alexey Kudinkin
committed
Make schema compatibility validation configurable w/in `HoodieSparkSqlWriter`
Alexey Kudinkin
committed
Disambiguated writing schema to be just a single filed w/in `WriteHandle` impls;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Make sure original source schema is used to parse original record's Avro payload
Alexey Kudinkin
committed
Cleaned up `RECONCILE_SCHEMA` config description;
Alexey Kudinkin
committed
Disable schema canonicalization, reconciliation and validation for `MERGE INTO` statements
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Added test w/ data-type changing to basic SE test-case
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed schema evolution to be handled appropriately for the first commit;
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixing typo
Alexey Kudinkin
committed
Fixed invalid API usage
Alexey Kudinkin
committed
Avoid file-reader re-initialization, abstracting common vars;
Alexey Kudinkin
committed
Fixed `shouldRewriteInWriterSchema` flag handling to properly handle bootstrap case
Alexey Kudinkin
committed
Extracted (advanced) schema evolution handling into a standalone method
Alexey Kudinkin
committed
Dismabiguate writer/reader schemas in `HoodieMergeHelper`
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Rebased Schema Evolution seq to produce opaque record transformer;
Alexey Kudinkin
committed
Rebased record transformation to bypass Avro encoding/decoding (using object traversals instead)
Alexey Kudinkin
committed
Pass in record-iterator to bootstrap handler
Alexey Kudinkin
committed
Added ability to `HoodieMergeHelper` to rewrite records in the writer schema in cases when writer-schema is not strictly a projection of the reader-schema
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Rebase onto proper record re-writing utility
Alexey Kudinkin
committed
Cleaned up unnecessary generics
Alexey Kudinkin
committed
Cleaned up `MergeHelper` impls
Alexey Kudinkin
committed
Fixed Parquet-to-Avro converster to avoid treating "list" element wrappers as records (instead checking on whether these are actual lists)
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Cleaning up erroneous assertion
Alexey Kudinkin
committed
Removed superfluous impls of `MergeHelper`s for Flink/Java
Alexey Kudinkin
committed
Enable table schema validation by default
Alexey Kudinkin
committed
Make sure that schema of the incoming batch is properly canonicalized whenever schemas are reconciliated or not
Alexey Kudinkin
committed
Make sure Avro's record-name/-namespace are appropriately preserved
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Relocating test methods
Alexey Kudinkin
committed
Fixed `AvroInternalSchemaConverter` to generate compatible schemas
Alexey Kudinkin
committed
Disable schema validation in test that is relying on malformed commit metadata
Alexey Kudinkin
committed
Cleaned up `AvroInternalSchemaConverter` to be aligned w/ Catalyst's `SchemaConverters` producing compatible schemas
Alexey Kudinkin
committed
Cleaned up MOR `DataBlock`s impls to avoid spilling of Schema Evolution logic into them unnecessarily
Alexey Kudinkin
committed
Revisited Schema Evolution handling in AbstractHoodieLogRecordReader (to abstract it as a simple transformer atop of the record iterator)
Alexey Kudinkin
committed
Missing license
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fix `Fixed` schema handling when rewriting the record into the new Schema
Alexey Kudinkin
committed
Fixing NPEs
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Read in projected schema in case when writer's schema is simply a projection of the reader's one
Alexey Kudinkin
committed
Fixed `InternalSchema` conversion to generate schemas compatible w/ previously generated Avro schemas
Alexey Kudinkin
committed
Fixing typo
Alexey Kudinkin
committed
Reverting inadvertent change
Alexey Kudinkin
committed
Make use of Avro's schema full-name when converting from `InternalSchema`
Alexey Kudinkin
committed
Extracted writer schema derivation seq into a standalone method;
Alexey Kudinkin
committed
Revisited `DeltaSync` to leverage `HoodieSparkSqlWriter.deduceWriterSchema` to reconcile incoming schema w/ the table's one after applied transformation;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Fixed converted schema to bear table's fully-qualified Avro record-name
Alexey Kudinkin
committed
Localized `AvroSchemaCompatibility` from Avro 1.10;
Alexey Kudinkin
committed
Added appropriate current location tracking
Alexey Kudinkin
committed
Fixed Avro schema name-checking to only occur in 2 cases:
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Fixing tests
Alexey Kudinkin
committed
Abstracted reflection-based handling of the default value for Avro schemas inside `AvroDefaultValueAccessor`
Alexey Kudinkin
committed
Fixed `SparkAvroPostProcessor` to preserve Avro schema's qualified name
Alexey Kudinkin
committed
Fixed fixtures for post-processed schema test
Alexey Kudinkin
committed
Fixed NPE in default value extraction
Alexey Kudinkin
committed
Updating tests fixtures
Alexey Kudinkin
committed
Relocated default config overriding to `HoodieParquetReader`;
Alexey Kudinkin
committed
Fixed Flink's `ParquetSchemaConverter` to properly carryover nullability during schema conversion
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
`lint`
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing new tests
Alexey Kudinkin
committed
Tidying up docs
Alexey Kudinkin
committed
Fixed schema evolution handling in `HoodieMergeHelper`
Alexey Kudinkin
committed
Fixing more tests
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Make CANONICALIZE_SCHEMA config package-private
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed `isProjection` to properly account for nested projections;
Alexey Kudinkin
committed
Fixing invalid test
Alexey Kudinkin
committed
(WIP) Disable flaky `TestCleaner` test
Alexey Kudinkin
committed
Moved `isSchemaCompatible` from `TableSchemaResolver` to `AvroSchemaUtils`
Alexey Kudinkin
committed
Fixed `AvroSchemaUtils.isProjection` for Array, Map and Union types
Alexey Kudinkin
committed
Revisited schema reconciliation sequence to implement legacy semantic
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixed schema projection sequence to properly match atomic-types
Alexey Kudinkin
committed
Added tests
Alexey Kudinkin
committed
Updated tests
Alexey Kudinkin
committed
Updated RECONCILE_SCHEMA config description
Alexey Kudinkin
committed
Fixed invalid schema fetching in bootstrap tests
Alexey Kudinkin
committed
Fixed `HiveAvroSerializer` to properly resolve nullable array element-types
Alexey Kudinkin
committed
Fixed invalid compaction test
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fixing `isCompatibleProjection` check to ignore schema names when matching atomic-types
Alexey Kudinkin
committed