Skip to content

[SPARK-32332][SQL][3.0] Support columnar exchanges #29310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
andygrove wants to merge 2 commits into from
Closed

[SPARK-32332][SQL][3.0] Support columnar exchanges #29310

andygrove wants to merge 2 commits into from

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Jul 30, 2020

What changes were proposed in this pull request?

Backports SPARK-32332 to 3.0 branch.

Why are the changes needed?

Plugins cannot replace exchanges with columnar versions when AQE is enabled without this patch.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tests included.

@andygrove
Copy link
Member Author

andygrove commented Jul 30, 2020

@cloud-fan @tgravescs

@tgravescs
Copy link
Contributor

tgravescs commented Jul 30, 2020

ok to test

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jul 30, 2020

Jenkins seems to be out of order. For branch-3.0, we need Jenkins run.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jul 30, 2020

@andygrove . FYI, we usually use [3.0] tag on the PR title if the PR is made for branch-3.0 specifically. In addition, [2.4] is used for branch-2.4.

@andygrove andygrove changed the title [SPARK-32332][SQL] Support columnar exchanges in 3.0 branch [SPARK-32332][SQL][3.0] Support columnar exchanges Jul 30, 2020
@andygrove
Copy link
Member Author

andygrove commented Jul 30, 2020

Thanks @dongjoon-hyun I have updated the title.

@tgravescs
Copy link
Contributor

tgravescs commented Jul 30, 2020

ok to test

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 31, 2020

ok to test

@SparkQA
Copy link

SparkQA commented Jul 31, 2020

Test build #126844 has finished for PR 29310 at commit 7aaa0cf.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 31, 2020

retest this please

@SparkQA
Copy link

SparkQA commented Jul 31, 2020

Test build #126877 has finished for PR 29310 at commit 7aaa0cf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tgravescs
Copy link
Contributor

tgravescs commented Jul 31, 2020

tests pass so I think we are good. I'm going to commit this shortly.

asfgit pushed a commit that referenced this pull request Jul 31, 2020
@andygrove @tgravescs
[SPARK-32332][SQL][3.0] Support columnar exchanges
7c91b15 
### What changes were proposed in this pull request?
Backports SPARK-32332 to 3.0 branch.

### Why are the changes needed?
Plugins cannot replace exchanges with columnar versions when AQE is enabled without this patch.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Tests included.

Closes #29310 from andygrove/backport-SPARK-32332.

Authored-by: Andy Grove <[email protected]>
Signed-off-by: Thomas Graves <[email protected]>
@tgravescs
Copy link
Contributor

tgravescs commented Jul 31, 2020

merged to branch-3.0

@tgravescs
Copy link
Contributor

tgravescs commented Jul 31, 2020

@dongjoon-hyun sorry I don't remember, we need to manually close the 3.0 ones correct?

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jul 31, 2020 •
edited
Loading

Yes~ For non-master branch, we should close it manually. Let me close this, @tgravescs .

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Jul 31, 2020

Thank you for the patch, @andygrove and @tgravescs .

@tgravescs
Copy link
Contributor

tgravescs commented Jul 31, 2020

thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tgravescs tgravescs tgravescs approved these changes

Assignees

No one assigned

Labels

SQL STRUCTURED STREAMING

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@andygrove @tgravescs @dongjoon-hyun @HyukjinKwon @SparkQA