[HUDI-2471] Add support ignoring case in merge into #3700

dongkelun · 2021-09-22T02:04:18Z

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request

Add support ignoring case when column name matches in merge into

Brief change log

(for example:)

Add support ignoring case when column name matches in merge into

Verify this pull request

(example:)

Added unit test in TestMergeIntoTable2

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

nsivabalan · 2021-10-07T00:34:40Z

@pengzhiwei2018 @xushiyan : can either of you take care of reviewing this please.

xushiyan

LGTM. cc @YannByron

YannByron · 2021-10-25T02:35:11Z

...datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala

If the table field is defined in uppercase letters, does that work?

@YannByron yes,it can work in column name matching.Do I need to add a test case for upper case column name definition?
However, ignoring case matching has not been implemented in condition and action. I think we should support it. Do I submit another PR or support it in this PR?

It would be nice if you can solve all case matching including source/target schema, condition and action in this PR.

OK, I'll try to solve it

@YannByron Hello, I have modified the code to add support for ignoring case matching including condition and action.Can you please take a look?

...ark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala

YannByron · 2021-10-25T07:41:53Z

@dongkelun
I reproduce this using the table in UT in my local env, ClassCastException is raised. The detail trace stack:
Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer at org.apache.hudi.sql.payload.ExpressionPayloadEvaluator_228eb15b_f549_4127_a201_a71c459c1f61.eval(Unknown Source) at org.apache.spark.sql.hudi.command.payload.ExpressionPayload.evaluate(ExpressionPayload.scala:258)

is it same with yours?

dongkelun · 2021-10-25T08:14:52Z

@dongkelun I reproduce this using the table in UT in my local env, ClassCastException is raised. The detail trace stack: Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer at org.apache.hudi.sql.payload.ExpressionPayloadEvaluator_228eb15b_f549_4127_a201_a71c459c1f61.eval(Unknown Source) at org.apache.spark.sql.hudi.command.payload.ExpressionPayload.evaluate(ExpressionPayload.scala:258)

is it same with yours?

Sorry, I don't know what UT is or how to use it. Is UTC time set or something else?

YannByron · 2021-10-25T08:49:57Z

@dongkelun I reproduce this using the table in UT in my local env, ClassCastException is raised. The detail trace stack: Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer at org.apache.hudi.sql.payload.ExpressionPayloadEvaluator_228eb15b_f549_4127_a201_a71c459c1f61.eval(Unknown Source) at org.apache.spark.sql.hudi.command.payload.ExpressionPayload.evaluate(ExpressionPayload.scala:258)
is it same with yours?

Sorry, I don't know what UT is or how to use it. Is UTC time set or something else?

I mean UT is unit test. sql is run as followed :

create table h0 ( id int, name string, price double, ts long, dt string) using hudi options (primaryKey ='id', preCombineField = 'ts');

merge into h0 as t0 using (select 1 as ID, 'a1' as NAME, 1111 as TS, '2021-05-05' as DT, 111 as PRICE) as s0 on t0.id = s0.id when matched then update set * when not matched then insert *;

dongkelun · 2021-10-26T00:52:15Z

@dongkelun I reproduce this using the table in UT in my local env, ClassCastException is raised. The detail trace stack: Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer at org.apache.hudi.sql.payload.ExpressionPayloadEvaluator_228eb15b_f549_4127_a201_a71c459c1f61.eval(Unknown Source) at org.apache.spark.sql.hudi.command.payload.ExpressionPayload.evaluate(ExpressionPayload.scala:258)
is it same with yours?

Sorry, I don't know what UT is or how to use it. Is UTC time set or something else?

I mean UT is unit test. sql is run as followed :

create table h0 ( id int, name string, price double, ts long, dt string) using hudi options (primaryKey ='id', preCombineField = 'ts');

merge into h0 as t0 using (select 1 as ID, 'a1' as NAME, 1111 as TS, '2021-05-05' as DT, 111 as PRICE) as s0 on t0.id = s0.id when matched then update set * when not matched then insert *;

I haven't encountered such an exception. I can run unit test cases in the local Windows environment. I tried your SQL and there was no problem. It can run normally

dongkelun · 2021-10-26T11:32:14Z

@dongkelun I reproduce this using the table in UT in my local env, ClassCastException is raised. The detail trace stack: Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer at org.apache.hudi.sql.payload.ExpressionPayloadEvaluator_228eb15b_f549_4127_a201_a71c459c1f61.eval(Unknown Source) at org.apache.spark.sql.hudi.command.payload.ExpressionPayload.evaluate(ExpressionPayload.scala:258)
is it same with yours?

Sorry, I don't know what UT is or how to use it. Is UTC time set or something else?

I mean UT is unit test. sql is run as followed :

create table h0 ( id int, name string, price double, ts long, dt string) using hudi options (primaryKey ='id', preCombineField = 'ts');

merge into h0 as t0 using (select 1 as ID, 'a1' as NAME, 1111 as TS, '2021-05-05' as DT, 111 as PRICE) as s0 on t0.id = s0.id when matched then update set * when not matched then insert *;

@YannByron hello,I encountered a similar problem when I added test cases for condition and action to ignore case matching

Caused by: java.lang.RuntimeException: Error in execute expression: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Double.
Expressions is: [boundreference() AS `ID`  boundreference() AS `NAME`  (CAST(boundreference() AS DOUBLE) + boundreference()) AS `PRICE`  CAST(boundreference() AS `TS` AS BIGINT)  boundreference() AS `DT`]

YannByron · 2021-10-26T14:11:35Z

@dongkelun Sorry, I personally think the previous solution to lowercase fields may not be the best and most correct one. We should dig into the root cause.

dongkelun · 2021-10-26T14:35:40Z

@dongkelun Sorry, I personally think the previous solution to lowercase fields may not be the best and most correct one. We should dig into the root cause.

In column name matching, I can't think of any better method except to lowercase fields. Can you provide some ideas?

YannByron · 2021-10-26T16:00:27Z

@dongkelun Sorry, I personally think the previous solution to lowercase fields may not be the best and most correct one. We should dig into the root cause.

In column name matching, I can't think of any better method except to lowercase fields. Can you provide some ideas?

For

@dongkelun I reproduce this using the table in UT in my local env, ClassCastException is raised. The detail trace stack: Caused by: java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Integer at org.apache.hudi.sql.payload.ExpressionPayloadEvaluator_228eb15b_f549_4127_a201_a71c459c1f61.eval(Unknown Source) at org.apache.spark.sql.hudi.command.payload.ExpressionPayload.evaluate(ExpressionPayload.scala:258)
is it same with yours?

Sorry, I don't know what UT is or how to use it. Is UTC time set or something else?

I mean UT is unit test. sql is run as followed :
create table h0 ( id int, name string, price double, ts long, dt string) using hudi options (primaryKey ='id', preCombineField = 'ts');
merge into h0 as t0 using (select 1 as ID, 'a1' as NAME, 1111 as TS, '2021-05-05' as DT, 111 as PRICE) as s0 on t0.id = s0.id when matched then update set * when not matched then insert *;

@YannByron hello,I encountered a similar problem when I added test cases for condition and action to ignore case matching
Caused by: java.lang.RuntimeException: Error in execute expression: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Double.
Expressions is: [boundreference() AS `ID`  boundreference() AS `NAME`  (CAST(boundreference() AS DOUBLE) + boundreference()) AS `PRICE`  CAST(boundreference() AS `TS` AS BIGINT)  boundreference() AS `DT`]

Is this solved?

dongkelun · 2021-10-27T02:06:23Z

@hudi-bot run azure

dongkelun · 2021-10-27T03:42:03Z

@hudi-bot run azure

dongkelun · 2021-10-27T14:48:46Z

@YannByron Hello, I have replaced toLowerCase with resolver, can you please take a look?

...datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala

...udi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala

YannByron · 2021-10-28T11:40:05Z

LGTM.

nsivabalan · 2021-11-02T03:43:49Z

@hudi-bot azure run

dongkelun · 2021-11-03T08:37:14Z

@hudi-bot run azure

xushiyan

@dongkelun thank you for the change. just have a small question on the test case.

...ark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala

hudi-bot · 2021-11-05T10:26:30Z

CI report:

344a059 UNKNOWN
e51adfc32fb28fcdaedc63e33a911b3df9f6bb8c UNKNOWN
54e8c6f Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

vinothchandar assigned vinothchandar and pengzhiwei2018 Sep 23, 2021

nsivabalan assigned xushiyan Oct 7, 2021

xushiyan approved these changes Oct 23, 2021

View reviewed changes

YannByron reviewed Oct 25, 2021

View reviewed changes

...ark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala Outdated Show resolved Hide resolved

dongkelun force-pushed the HUDI-2471 branch from 98de9c0 to 421892e Compare October 26, 2021 12:45

dongkelun changed the title ~~[HUDI-2471] Add support ignoring case when column name matches in merge into~~ [HUDI-2471] Add support ignoring case in merge into Oct 26, 2021

dongkelun force-pushed the HUDI-2471 branch from 421892e to 07e7efe Compare October 27, 2021 14:25

YannByron reviewed Oct 28, 2021

View reviewed changes

...datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala Outdated Show resolved Hide resolved

YannByron reviewed Oct 28, 2021

View reviewed changes

...udi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala Outdated Show resolved Hide resolved

dongkelun force-pushed the HUDI-2471 branch from c1c41f3 to 7224358 Compare October 28, 2021 06:44

dongkelun mentioned this pull request Oct 29, 2021

[HUDI-2642] Add support ignoring case in update sql operation #3882

Merged

5 tasks

dongkelun added 2 commits November 1, 2021 23:43

[HUDI-2471] Add support ignoring case in merge into

5b53ce5

[HUDI-2471] Add support ignoring case in merge into

96d390c

nsivabalan force-pushed the HUDI-2471 branch from 7224358 to f4c30a0 Compare November 2, 2021 03:43

xushiyan approved these changes Nov 5, 2021

View reviewed changes

...ark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala Outdated Show resolved Hide resolved

xushiyan added the pr:author-action label Nov 5, 2021

[HUDI-2471] Add support ignoring case in merge into

344a059

dongkelun force-pushed the HUDI-2471 branch from 9898eb0 to 344a059 Compare November 5, 2021 08:40

[HUDI-2471] Add support ignoring case in merge into

54e8c6f

dongkelun force-pushed the HUDI-2471 branch from e51adfc to 54e8c6f Compare November 5, 2021 09:10

xushiyan removed the pr:author-action label Nov 5, 2021

xushiyan merged commit 844346c into apache:master Nov 5, 2021

[HUDI-2471] Add support ignoring case in merge into #3700

[HUDI-2471] Add support ignoring case in merge into #3700

Uh oh!

Conversation

dongkelun commented Sep 22, 2021

Tips

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

nsivabalan commented Oct 7, 2021

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

YannByron Oct 25, 2021

Choose a reason for hiding this comment

Uh oh!

dongkelun Oct 25, 2021

Choose a reason for hiding this comment

Uh oh!

YannByron Oct 25, 2021

Choose a reason for hiding this comment

Uh oh!

dongkelun Oct 25, 2021

Choose a reason for hiding this comment

Uh oh!

dongkelun Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

YannByron commented Oct 25, 2021

Uh oh!

dongkelun commented Oct 25, 2021

Uh oh!

YannByron commented Oct 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongkelun commented Oct 26, 2021

Uh oh!

dongkelun commented Oct 26, 2021

Uh oh!

YannByron commented Oct 26, 2021

Uh oh!

dongkelun commented Oct 26, 2021

Uh oh!

YannByron commented Oct 26, 2021

Uh oh!

dongkelun commented Oct 27, 2021

Uh oh!

dongkelun commented Oct 27, 2021

Uh oh!

dongkelun commented Oct 27, 2021

Uh oh!

Uh oh!

Uh oh!

YannByron commented Oct 28, 2021

Uh oh!

nsivabalan commented Nov 2, 2021

Uh oh!

dongkelun commented Nov 3, 2021

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hudi-bot commented Nov 5, 2021

CI report:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

YannByron commented Oct 25, 2021 •

edited

Loading