Skip to content

[SPARK-34702][SQL] Avoid unnecessary code generation in JoinCodegenSupport.genBuildSideVars #31802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
c21 wants to merge 2 commits into from
Closed

[SPARK-34702][SQL] Avoid unnecessary code generation in JoinCodegenSupport.genBuildSideVars #31802

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
65f8a32
Avoid unnecessary code generation in JoinCodegenSupport.genBuildSideVars
c21 Mar 10, 2021
a0cb676
revert 3.1-compatible change pre comment
c21 Mar 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ case class BroadcastHashJoinExec(
left: SparkPlan,
right: SparkPlan,
isNullAwareAntiJoin: Boolean = false)
extends HashJoin with CodegenSupport {
extends HashJoin {
Copy link
Contributor Author

@c21 c21 Mar 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a minor fix packed in this PR. HashJoin already extends JoinCodegenSupport. So we don't need CodegenSupport here.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't mind, could you make a separate PR? You can use [MINOR] tag instead of creating a new JIRA.

Copy link
Contributor Author

@c21 c21 Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dongjoon-hyun - sure, will do it.


if (isNullAwareAntiJoin) {
require(leftKeys.length == 1, "leftKeys length should be 1")
Expand Down
40 changes: 22 additions & 18 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ package org.apache.spark.sql.execution.joins
import org.apache.spark.sql.catalyst.expressions.{BindReferences, BoundReference}
import org.apache.spark.sql.catalyst.expressions.codegen._
import org.apache.spark.sql.catalyst.expressions.codegen.Block._
import org.apache.spark.sql.catalyst.plans.InnerLike
import org.apache.spark.sql.catalyst.plans.{ExistenceJoin, InnerLike, LeftAnti, LeftOuter, LeftSemi, RightOuter}
import org.apache.spark.sql.execution.{CodegenSupport, SparkPlan}

/**
Expand Down Expand Up @@ -73,23 +73,27 @@ trait JoinCodegenSupport extends CodegenSupport with BaseJoinExec {
ctx.INPUT_ROW = buildRow
buildPlan.output.zipWithIndex.map { case (a, i) =>
val ev = BoundReference(i, a.dataType, a.nullable).genCode(ctx)
if (joinType.isInstanceOf[InnerLike]) {
ev
} else {
// the variables are needed even there is no matched rows
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val javaType = CodeGenerator.javaType(a.dataType)
val code = code"""
|boolean $isNull = true;
|$javaType $value = ${CodeGenerator.defaultValue(a.dataType)};
|if ($buildRow != null) {
| ${ev.code}
| $isNull = ${ev.isNull};
| $value = ${ev.value};
|}
""".stripMargin
ExprCode(code, JavaCode.isNullVariable(isNull), JavaCode.variable(value, a.dataType))
joinType match {
case _: InnerLike | LeftSemi | LeftAnti | _: ExistenceJoin =>
ev
case LeftOuter | RightOuter =>
// the variables are needed even there is no matched rows
val isNull = ctx.freshName("isNull")
val value = ctx.freshName("value")
val javaType = CodeGenerator.javaType(a.dataType)
val code = code"""
|boolean $isNull = true;
|$javaType $value = ${CodeGenerator.defaultValue(a.dataType)};
|if ($buildRow != null) {
| ${ev.code}
| $isNull = ${ev.isNull};
| $value = ${ev.value};
|}
""".stripMargin
ExprCode(code, JavaCode.isNullVariable(isNull), JavaCode.variable(value, a.dataType))
case _ =>
throw new IllegalArgumentException(
s"JoinCodegenSupport.genBuildSideVars should not take $joinType as the JoinType")
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/joins/ExistenceJoinSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ class ExistenceJoinSuite extends SparkPlanTest with SharedSparkSession {
ProjectExec(output, FilterExec(condition, join))
}

test(s"$testName using ShuffledHashJoin") {
testWithWholeStageCodegenOnAndOff(s"$testName using ShuffledHashJoin") { _ =>
Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test enforcement looks orthogonal to this PR (Avoiding unncessary code generation). Can we do this as a test PR independently? We can merge it separately to the other branches.

Copy link
Contributor Author

@c21 c21 Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dongjoon-hyun - if the purpose of separating PRs is to merge change to other branches as well, I can submit another PR for all test-related change, i.e. the change in InnerJoinSuite.scala, OuterJoinSuite.scala and ExistenceJoinSuite.scala.

Do we want to do that? These test changes are valid to be merged into 3.1 branch as well.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I want to merge the test improvement into branch-3.1.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPARK-34702 is for Apache Spark 3.2.0.

Copy link
Contributor Author

@c21 c21 Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dongjoon-hyun - sure, updated.

extractJoinParts().foreach { case (_, leftKeys, rightKeys, boundCondition, _, _, _) =>
withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
checkAnswer2(leftRows, rightRows, (left: SparkPlan, right: SparkPlan) =>
Expand Down
8 changes: 4 additions & 4 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/joins/InnerJoinSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ class InnerJoinSuite extends SparkPlanTest with SharedSparkSession {
}
}

test(s"$testName using ShuffledHashJoin (build=left)") {
testWithWholeStageCodegenOnAndOff(s"$testName using ShuffledHashJoin (build=left)") { _ =>
extractJoinParts().foreach { case (_, leftKeys, rightKeys, boundCondition, _, _, _) =>
withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
checkAnswer2(leftRows, rightRows, (leftPlan: SparkPlan, rightPlan: SparkPlan) =>
Expand All @@ -165,7 +165,7 @@ class InnerJoinSuite extends SparkPlanTest with SharedSparkSession {
}
}

test(s"$testName using ShuffledHashJoin (build=right)") {
testWithWholeStageCodegenOnAndOff(s"$testName using ShuffledHashJoin (build=right)") { _ =>
extractJoinParts().foreach { case (_, leftKeys, rightKeys, boundCondition, _, _, _) =>
withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
checkAnswer2(leftRows, rightRows, (leftPlan: SparkPlan, rightPlan: SparkPlan) =>
Expand Down Expand Up @@ -198,7 +198,7 @@ class InnerJoinSuite extends SparkPlanTest with SharedSparkSession {
}
}

test(s"$testName using BroadcastNestedLoopJoin build left") {
testWithWholeStageCodegenOnAndOff(s"$testName using BroadcastNestedLoopJoin build left") { _ =>
Copy link
Contributor Author

@c21 c21 Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BroadcastNestedLoopJoin codegen is merged in 3.2 (current master), so keep test change here.

withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
checkAnswer2(leftRows, rightRows, (left: SparkPlan, right: SparkPlan) =>
BroadcastNestedLoopJoinExec(left, right, BuildLeft, Inner, Some(condition())),
Expand All @@ -207,7 +207,7 @@ class InnerJoinSuite extends SparkPlanTest with SharedSparkSession {
}
}

test(s"$testName using BroadcastNestedLoopJoin build right") {
testWithWholeStageCodegenOnAndOff(s"$testName using BroadcastNestedLoopJoin build right") { _ =>
Copy link
Contributor Author

@c21 c21 Mar 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same reason as change above.

withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
checkAnswer2(leftRows, rightRows, (left: SparkPlan, right: SparkPlan) =>
BroadcastNestedLoopJoinExec(left, right, BuildRight, Inner, Some(condition())),
Expand Down
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/joins/OuterJoinSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ class OuterJoinSuite extends SparkPlanTest with SharedSparkSession {
ExtractEquiJoinKeys.unapply(join)
}

test(s"$testName using ShuffledHashJoin") {
testWithWholeStageCodegenOnAndOff(s"$testName using ShuffledHashJoin") { _ =>
extractJoinParts().foreach { case (_, leftKeys, rightKeys, boundCondition, _, _, _) =>
withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
val buildSide = if (joinType == LeftOuter) BuildRight else BuildLeft
Expand Down