Skip to content

[HUDI-1869] Upgrading Spark3 To 3.1 #3844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xushiyan merged 5 commits into from
Nov 3, 2021
Merged

[HUDI-1869] Upgrading Spark3 To 3.1 #3844

xushiyan merged 5 commits into from
Nov 3, 2021

Conversation

@YannByron
Copy link
Contributor

@YannByron YannByron commented Oct 22, 2021 •
edited
Loading

  • Upgrading spark3 version from the spark3.0.x to spark 3.1.x

  • Support both spark 3.0.x and spark 3.1.x.

  • Build package for spark 3.0.x use:

    mvn clean install -DskipTests -Dspark3 -Dspark3.0.x

    For other spark3 version :

    mvn clean install -DskipTests -Dspark3

Brief change log

(for example:)

  • Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot
Copy link
Collaborator

hudi-bot commented Oct 22, 2021 •
edited
Loading

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run travis re-run the last Travis build
  • @hudi-bot run azure re-run the last Azure build

@YannByron
Copy link
Contributor Author

YannByron commented Oct 24, 2021

@hudi-bot run azure

@xushiyan xushiyan mentioned this pull request Oct 24, 2021
Closed
5 tasks
@xushiyan xushiyan added the priority:blocker Production down; release blocker label Oct 24, 2021
vinothchandar
vinothchandar reviewed Oct 28, 2021
View reviewed changes
hudi-spark-datasource/hudi-spark/pom.xml Outdated
Copy link
Member

@vinothchandar vinothchandar Oct 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we stick to log4j?

Copy link
Contributor Author

@YannByron YannByron Nov 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The slf4j-api dependency is used by Spark3.1 when run hudi-spark UTs. And it's in test scope.

vinothchandar
vinothchandar approved these changes Oct 28, 2021
View reviewed changes
Copy link
Member

@vinothchandar vinothchandar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. We can land once you chime in on the slf4j changes and rebase and push

...-spark-datasource/hudi-spark3/src/main/java/org/apache/hudi/spark3/internal/ReflectUtil.java Outdated
Copy link
Member

@vinothchandar vinothchandar Oct 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking out loud. this uses reflection, but only seems to operate with a plan, so should be fine, performance overhead wise

@YannByron
Copy link
Contributor Author

YannByron commented Oct 29, 2021

@hudi-bot run azure

1 similar comment
@YannByron
Copy link
Contributor Author

YannByron commented Oct 29, 2021

@hudi-bot run azure

@YannByron
Copy link
Contributor Author

YannByron commented Oct 29, 2021

@vinothchandar @xushiyan I have removed slf4j dependency and rebase and push.
But CI failed. Got three different results wIth three running. Any suggestion for this?

@YannByron
Copy link
Contributor Author

YannByron commented Nov 1, 2021

@hudi-bot run azure

xushiyan
xushiyan reviewed Nov 1, 2021
View reviewed changes
Copy link
Member

@xushiyan xushiyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YannByron thanks. left some suggestions.

...spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestHoodieSqlBase.scala Outdated
Comment on lines +100 to +108
Copy link
Member

@xushiyan xushiyan Nov 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we able to use assertThrows() here from junit 5? or leave it the scala way also fine.

@YannByron
Copy link
Contributor Author

YannByron commented Nov 2, 2021

@xushiyan Can you please review this again ~

@YannByron
Copy link
Contributor Author

YannByron commented Nov 2, 2021

@hudi-bot run azure

@xushiyan xushiyan merged commit 1f17467 into apache:master Nov 3, 2021
@rubenssoto rubenssoto mentioned this pull request Nov 17, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vinothchandar vinothchandar vinothchandar approved these changes

@xushiyan xushiyan xushiyan approved these changes

Assignees

@vinothchandar vinothchandar

@xushiyan xushiyan

Labels

priority:blocker Production down; release blocker

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@YannByron @hudi-bot @vinothchandar @xushiyan