[HUDI-5189] Make HiveAvroSerializer compatible with hive3 #7173

xicm · 2022-11-10T08:58:45Z

Change Logs

Compilation of HiveAvroSerializer fails with hive3.

Impact

none

Risk level (write none, low medium or high below)

low

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

xicm · 2022-11-10T09:06:42Z

Hi @xiarixiaoyao ， please review if you are available.
I tested with hive3 on my pc.

xiarixiaoyao · 2022-11-11T01:32:22Z

@hudi-bot run azure

xiarixiaoyao · 2022-11-11T07:11:27Z

@xicm
Thank you for your contribution
Looks good, let me test it for presto

cdmikechen · 2022-11-11T07:15:57Z

@xiarixiaoyao @xicm
There seems to be a PR at the moment that may overlap with this issue, should we deal with this issue in a unified way?
#3391

xiarixiaoyao · 2022-11-11T09:51:03Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java

        return new Utf8(string);
      case DATE:
-        return DateWritable.dateToDays(((DateObjectInspector)fieldOI).getPrimitiveJavaObject(structFieldData));
+        return new DateWritable((DateWritable)structFieldData).getDays();


Hive3 will return DateWritableV2 for date type, we cannot convert DateWritableV2 to DateWritable direclty.

ohh, I compile with hive 3.1.2 and run TestHiveAvroSerializer, the test is successful. Let me do more tests.

xiarixiaoyao · 2022-11-11T09:51:35Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java

-        Timestamp timestamp =
-            ((TimestampObjectInspector) fieldOI).getPrimitiveJavaObject(structFieldData);
-        return timestamp.getTime();
+        return new TimestampWritable((TimestampWritable) structFieldData).getTimestamp().getTime();


xiarixiaoyao · 2022-11-11T09:52:35Z

@xiarixiaoyao @xicm There seems to be a PR at the moment that may overlap with this issue, should we deal with this issue in a unified way? #3391

yes, it will be better to deal with hive2/hive3 will a unifed way

xicm · 2022-11-14T10:19:13Z

@xiarixiaoyao @cdmikechen
#3391 should be resolved, if we want to support hive3.
@cdmikechen Could you fix the conflict in #3391?

xicm · 2022-11-16T02:16:54Z

Hi @cdmikechen , do you mind if I copy the code #3391 to this pr?

xicm · 2022-11-16T02:33:11Z

Hi @xiarixiaoyao, I merged #3391 and tested with the case you described in #6741.
a tricky result,
select * from tx_null -> The result is right.
select col7 from tx_null -> returns 5 records but the values are NULL.
I have checked the code but find nothing, , do you have any advice to solve this.
I will push the code later.

cdmikechen · 2022-11-16T09:07:58Z

@xiarixiaoyao @xicm
I've had a lot of workday stuff going on lately and haven't rebase the code for a while.
But I wonder if we could find a time for us to have an online chat(like wechat, zoom or slack at weekend), in which case I think we should be able to work this out well.

xicm · 2022-11-16T09:52:12Z

@xiarixiaoyao @xicm I've had a lot of workday stuff going on lately and haven't rebase the code for a while. But I wonder if we could find a time for us to have an online chat(like wechat, zoom or slack at weekend), in which case I think we should be able to work this out well.

I have merged your code , but there is still a problem with Timestamp, I'm debugging.

xiarixiaoyao · 2022-11-16T10:08:35Z

@xiarixiaoyao @xicm I've had a lot of workday stuff going on lately and haven't rebase the code for a while. But I wonder if we could find a time for us to have an online chat(like wechat, zoom or slack at weekend), in which case I think we should be able to work this out well.

my wechat 1037817390

xiarixiaoyao · 2022-12-02T07:54:06Z

@cdmikechen could you pls help review this pr， thanks

xicm · 2022-12-16T06:13:48Z

@hudi-bot run azure

danny0405 · 2023-03-28T07:15:05Z

@xiarixiaoyao Can we push-forward this PR again, seems s useful fix.

xiarixiaoyao · 2023-03-30T07:40:36Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java

   * @param realReader Parquet Reader
   */
  public RealtimeUnmergedRecordReader(RealtimeSplit split, JobConf job,
-      RecordReader<NullWritable, ArrayWritable> realReader) {


Do not change the original format， thanks

xiarixiaoyao · 2023-03-30T07:40:56Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/RealtimeUnmergedRecordReader.java

-          .withLogFilePaths(split.getDeltaLogPaths())
-          .withReaderSchema(getReaderSchema())
-          .withLatestInstantTime(split.getMaxCommitTime())
-          .withReadBlocksLazily(Boolean.parseBoolean(this.jobConf.get(HoodieRealtimeConfig.COMPACTION_LAZY_BLOCK_READ_ENABLED_PROP, HoodieRealtimeConfig.DEFAULT_COMPACTION_LAZY_BLOCK_READ_ENABLED)))


xiarixiaoyao · 2023-03-30T07:42:55Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java

-        return DateWritable.dateToDays(((DateObjectInspector)fieldOI).getPrimitiveJavaObject(structFieldData));
+        try {
+          Class<?> clazz = structFieldData.getClass();
+          return  clazz.getMethod("getDays").invoke(structFieldData);


This has a significant impact on performance，
Let's avoid using reflection every iteration。
Let's do the same thing as HiveUtils

xiarixiaoyao · 2023-04-19T11:14:27Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieHiveUtils.java

-  public static final String DATE_WRITEABLE_V2_CLASS = "org.apache.hadoop.hive.serde2.io.DateWritableV2";
-  public static final boolean SUPPORT_DATE_WRITEABLE_V2;
-  private static final Constructor DATE_WRITEABLE_V2_CONSTRUCTOR;
+  public static String DATE_WRITEABLE_V2_CLASS = "org.apache.hadoop.hive.serde2.io.DateWritableV2";


xiarixiaoyao · 2023-04-19T11:14:48Z

LGTM

xiarixiaoyao · 2023-04-19T11:15:10Z

@XuQianJin-Stars could you pls help review again

danny0405 · 2023-04-27T10:15:04Z

@xicm Thanks for the contribution, can we squash the commits into one, it is hard for code reviewing because of the merge cmd, let's use the git rebase instead of git merge, the git merge would generate a new log on the git history.

xicm · 2023-04-28T23:51:09Z

@hudi-bot run azure

danny0405 · 2023-05-24T10:14:24Z

@xicm Do you have plan to push forward this PR again?

xicm · 2023-05-25T05:51:43Z

@xicm Do you have plan to push forward this PR again?

Year, I resolved the conflict, review again when you are available.

… timestamp

danny0405 · 2023-05-26T11:59:26Z

Thanks for the contribution, overall looks good now, I have reviewed and created a patch:
5189.patch.zip

hudi-bot · 2023-05-26T21:29:12Z

CI report:

2346f8e UNKNOWN
cbeb002 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan · 2023-06-02T07:06:13Z

Awesome job on landing this. I guess this has been one of the longest pending gaps we had w/ hudi. timestamp col not readable w/ hive3. thank you folks.

Zouxxyy · 2023-06-05T12:47:24Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java

+      if (Arrays.stream(constructors)
+          .anyMatch(c -> c.getParameterCount() > 0 && c.getParameterTypes()[0]
+              .getName().equals(ParquetInputFormat.class.getName()))) {
+        supportAvroRead = true;


when will supportAvroRead be false? I add a pr for it~

@Zouxxyy You can read the comments. This is to address compatibility issues with spark

@Zouxxyy You can read the comments. This is to address compatibility issues with spark

yeal, but it always be true now

@Zouxxyy
In my impression, hive2 and spark3 have the same processing method and constructor, but there is a difference in hive3.
You can try switching hive to version 3 for verification

We should revisit this method, the name ParquetInputFormat.class.getName() in hive2 is the same as hive3.

This is for compatibility with spark，do you mean hive on spark? @cdmikechen

xiarixiaoyao self-assigned this Nov 11, 2022

xiarixiaoyao reviewed Nov 11, 2022

View reviewed changes

xiarixiaoyao added the priority:high Significant impact; potential bugs label Nov 13, 2022

xicm changed the title ~~[HUDI-5189] Make HiveAvroSerializer compatible with hive3~~ [WIP][HUDI-5189] Make HiveAvroSerializer compatible with hive3 Nov 14, 2022

xicm changed the title ~~[WIP][HUDI-5189] Make HiveAvroSerializer compatible with hive3~~ [HUDI-5189] Make HiveAvroSerializer compatible with hive3 Nov 25, 2022

xicm requested a review from xiarixiaoyao December 2, 2022 07:47

nsivabalan added the release-0.12.2 Patches targetted for 0.12.2 label Dec 6, 2022

codope added engine:hive Hive integration and removed release-0.12.2 Patches targetted for 0.12.2 labels Dec 7, 2022

xicm mentioned this pull request Jan 3, 2023

[SUPPORT] Exception when reading timestamp column from Hive #7475

Closed

xicm mentioned this pull request Jan 26, 2023

[SUPPORT] failed to read timestamp from hive #7724

Closed

xicm mentioned this pull request Mar 21, 2023

[SUPPORT]Compile Error of hudi-0.13.0 "org.apache.hadoop.hive.common.type can not be converted to java.sql.Date" #8216

Closed

xiarixiaoyao reviewed Mar 30, 2023

View reviewed changes

xicm requested a review from xiarixiaoyao March 31, 2023 07:11

xiarixiaoyao reviewed Apr 19, 2023

View reviewed changes

danny0405 added issue:version-compatibility Version compatibility issues release-0.14.0 priority:blocker Production down; release blocker labels Apr 24, 2023

xicm force-pushed the HUDI-5189 branch from 650d191 to 33e116e Compare April 28, 2023 02:48

codope removed the priority:blocker Production down; release blocker label May 1, 2023

xicm force-pushed the HUDI-5189 branch from 33e116e to eb64db7 Compare May 25, 2023 05:43

xicm force-pushed the HUDI-5189 branch from eb64db7 to 2346f8e Compare May 26, 2023 07:26

[HUDI-5189] Make HiveAvroSerializer compatible with hive3 and support…

3ea610b

… timestamp

xicm force-pushed the HUDI-5189 branch from 2346f8e to 3ea610b Compare May 26, 2023 07:30

fix check style

1bb181d

danny0405 self-assigned this May 26, 2023

abstract hive shims

cbeb002

danny0405 approved these changes May 27, 2023

View reviewed changes

danny0405 merged commit 1e3cdb6 into apache:master May 27, 2023

Anandonzy mentioned this pull request May 31, 2023

Hudi user ARRAY<INT> type have a Exception,NOT a record:"int" #8844

Closed

danny0405 mentioned this pull request Jun 2, 2023

[HUDI-83] Fix Timestamp/Date type read by Hive3 #3391

Closed

4 tasks

Zouxxyy reviewed Jun 5, 2023

View reviewed changes

Zouxxyy mentioned this pull request Jun 5, 2023

[MINOR] Reformat HoodieAvroParquetReader #8888

Merged

4 tasks

Zouxxyy mentioned this pull request Jun 14, 2023

[HUDI-6367] Fix NPE in HoodieAvroParquetReader and support complex schema with timestamp #8955

Merged

4 tasks

[HUDI-5189] Make HiveAvroSerializer compatible with hive3 #7173

[HUDI-5189] Make HiveAvroSerializer compatible with hive3 #7173

Uh oh!

Conversation

xicm commented Nov 10, 2022

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

xicm commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiarixiaoyao commented Nov 11, 2022

Uh oh!

xiarixiaoyao commented Nov 11, 2022

Uh oh!

cdmikechen commented Nov 11, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xicm Nov 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiarixiaoyao commented Nov 11, 2022

Uh oh!

xicm commented Nov 14, 2022

Uh oh!

xicm commented Nov 16, 2022

Uh oh!

xicm commented Nov 16, 2022

Uh oh!

cdmikechen commented Nov 16, 2022

Uh oh!

xicm commented Nov 16, 2022

Uh oh!

xiarixiaoyao commented Nov 16, 2022

Uh oh!

xiarixiaoyao commented Dec 2, 2022

Uh oh!

xicm commented Dec 16, 2022

Uh oh!

danny0405 commented Mar 28, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiarixiaoyao commented Apr 19, 2023

Uh oh!

xiarixiaoyao commented Apr 19, 2023

Uh oh!

danny0405 commented Apr 27, 2023

Uh oh!

xicm commented Apr 28, 2023

Uh oh!

danny0405 commented May 24, 2023

Uh oh!

xicm commented May 25, 2023

Uh oh!

danny0405 commented May 26, 2023

Uh oh!

hudi-bot commented May 26, 2023

CI report:

Uh oh!

nsivabalan commented Jun 2, 2023

Uh oh!

Zouxxyy Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

xicm commented Nov 10, 2022 •

edited

Loading

xicm Nov 11, 2022 •

edited

Loading

Zouxxyy Jun 5, 2023 •

edited

Loading

cdmikechen Jun 6, 2023 •

edited

Loading