Skip to content

[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition #31857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 8 commits into from
Closed

[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition #31857

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9ad1833
Push down the foldable predicate to both sides of Join
wangyum Mar 17, 2021
7997eee
fix
wangyum Mar 17, 2021
5988d28
Fix
wangyum Mar 18, 2021
ee5b4eb
Fix
wangyum Mar 18, 2021
9e913e9
pushDownJoinConditions
wangyum Mar 18, 2021
25689da
fix
wangyum Mar 19, 2021
debb094
fix
wangyum Mar 19, 2021
3202590
Merge remote-tracking branch 'upstream/master' into SPARK-28220
wangyum Mar 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1457,8 +1457,9 @@ object PushPredicateThroughJoin extends Rule[LogicalPlan] with PredicateHelper {
val (pushDownCandidates, nonDeterministic) = condition.partition(_.deterministic)
val (leftEvaluateCondition, rest) =
pushDownCandidates.partition(_.references.subsetOf(left.outputSet))
val (rightEvaluateCondition, commonCondition) =
rest.partition(expr => expr.references.subsetOf(right.outputSet))
val (rightEvaluateCondition, _) =
pushDownCandidates.partition(_.references.subsetOf(right.outputSet))
val commonCondition = rest.filterNot(e => rightEvaluateCondition.exists(_.semanticEquals(e)))

(leftEvaluateCondition, rightEvaluateCondition, commonCondition ++ nonDeterministic)
}
Expand Down
10 changes: 10 additions & 0 deletions ...catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1384,4 +1384,14 @@ class FilterPushdownSuite extends PlanTest {
condition = Some("x.a".attr === "z.a".attr)).analyze
comparePlans(optimized, correctAnswer)
}

test("SPARK-28220: Push down the foldable predicate to both sides of Join") {
val x = testRelation.subquery('x)
val y = testRelation.subquery('y)
val originalQuery = x.join(y, condition = Some(false))

val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer = x.where(false).join(y.where(false), condition = None).analyze
comparePlans(optimized, correctAnswer.analyze)
}
}