Skip to content

[SPARK-32056][SQL][Follow-up] Coalesce partitions for repartiotion hint and sql when AQE is enabled#28952

Closed
viirya wants to merge 3 commits into
apache:masterfrom
viirya:SPARK-32056-sql
Closed

[SPARK-32056][SQL][Follow-up] Coalesce partitions for repartiotion hint and sql when AQE is enabled#28952
viirya wants to merge 3 commits into
apache:masterfrom
viirya:SPARK-32056-sql

Conversation

@viirya
Copy link
Copy Markdown
Member

@viirya viirya commented Jun 30, 2020

What changes were proposed in this pull request?

As the followup of #28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled.

Why are the changes needed?

When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled.

Does this PR introduce any user-facing change?

Yes. After this change, if users don't specify the number of partitions when repartitioning using REPARTITION/REPARTITION_BY_RANGE hint or DISTRIBUTE BY/CLUSTER BY, AQE will coalesce partitions.

How was this patch tested?

Unit tests.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jun 30, 2020

I also think this might be worth creating a new jira ticket, but as initially we discussed it as follow-up. So I put it as a follow-up first.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jun 30, 2020

ok to test

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jun 30, 2020

Does jenkins not work?

@maropu
Copy link
Copy Markdown
Member

maropu commented Jun 30, 2020

Yea, it seems jenkins got sick last night...

cloud-fan
cloud-fan reviewed Jun 30, 2020
View reviewed changes
Comment thread sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala Outdated
case s: ShuffleExchangeExec => s
}
assert(shuffle.size == 1)
assert(shuffle(0).outputPartitioning.numPartitions == 10)
Copy link
Copy Markdown
Contributor

@cloud-fan cloud-fan Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's put 10 as a parameter, to make this method a bit more general.

Copy link
Copy Markdown
Member Author

@viirya viirya Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok.

@HyukjinKwon
Copy link
Copy Markdown
Member

HyukjinKwon commented Jun 30, 2020

+1 looks good to me too

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jun 30, 2020

Test build #124649 has finished for PR 28952 at commit e8b572f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jun 30, 2020

retest this please

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jul 1, 2020

Test build #124684 has finished for PR 28952 at commit e8b572f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jul 1, 2020

Test build #124712 has finished for PR 28952 at commit 9778824.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jul 1, 2020

retest this please

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jul 1, 2020

Test build #124726 has started for PR 28952 at commit 9778824.

@cloud-fan
Copy link
Copy Markdown
Contributor

cloud-fan commented Jul 1, 2020

retest this please

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jul 1, 2020

Test build #124774 has finished for PR 28952 at commit 9778824.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jul 1, 2020

retest this please...

@SparkQA
Copy link
Copy Markdown

SparkQA commented Jul 1, 2020

Test build #124804 has finished for PR 28952 at commit 9778824.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Jul 1, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @viirya and @cloud-fan .
Merged to master for Apache Spark 3.1.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented Jul 2, 2020

Thanks all!

@MGHawes MGHawes mentioned this pull request May 16, 2021
Closed
MGHawes pushed a commit to palantir/spark that referenced this pull request May 16, 2021
@viirya
[SPARK-32056][SQL][FOLLOW-UP] Coalesce partitions for repartiotion hi…
c098917 
…nt and sql when AQE is enabled

As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled.

When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled.

Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions.

Unit tests.

Closes apache#28952 from viirya/SPARK-32056-sql.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
LorenzoMartini pushed a commit to palantir/spark that referenced this pull request May 18, 2021
@viirya
[SPARK-32056][SQL][FOLLOW-UP] Coalesce partitions for repartiotion hi…
7b1c006 
…nt and sql when AQE is enabled

As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled.

When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled.

Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions.

Unit tests.

Closes apache#28952 from viirya/SPARK-32056-sql.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@LorenzoMartini LorenzoMartini mentioned this pull request May 18, 2021
Merged
LorenzoMartini pushed a commit to palantir/spark that referenced this pull request May 19, 2021
@viirya @LorenzoMartini
[SPARK-32056][SQL][FOLLOW-UP] Coalesce partitions for repartiotion hi…
e71b355 
…nt and sql when AQE is enabled

As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled.

When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled.

Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions.

Unit tests.

Closes apache#28952 from viirya/SPARK-32056-sql.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
16pierre pushed a commit to 16pierre/spark that referenced this pull request May 24, 2021
@viirya
[SPARK-32056][SQL][FOLLOW-UP] Coalesce partitions for repartiotion hi…
dd66b31 
…nt and sql when AQE is enabled

As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled.

When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled.

Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions.

Unit tests.

Closes apache#28952 from viirya/SPARK-32056-sql.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@viirya viirya deleted the SPARK-32056-sql branch December 27, 2023 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@viirya @maropu @HyukjinKwon @SparkQA @cloud-fan @dongjoon-hyun