Skip to content

Commit 0820484

Browse files
SongYadonggatorsmile
authored andcommitted
[SPARK-25716][SQL][MINOR] remove unnecessary collection operation in valid constraints generation
## What changes were proposed in this pull request? Project logical operator generates valid constraints using two opposite operations. It substracts child constraints from all constraints, than union child constraints again. I think it may be not necessary. Aggregate operator has the same problem with Project. This PR try to remove these two opposite collection operations. ## How was this patch tested? Related unit tests: ProjectEstimationSuite CollapseProjectSuite PushProjectThroughUnionSuite UnsafeProjectionBenchmark GeneratedProjectionSuite CodeGeneratorWithInterpretedFallbackSuite TakeOrderedAndProjectSuite GenerateUnsafeProjectionSuite BucketedRandomProjectionLSHSuite RemoveRedundantAliasAndProjectSuite AggregateBenchmark AggregateOptimizeSuite AggregateEstimationSuite DecimalAggregatesSuite DateFrameAggregateSuite ObjectHashAggregateSuite TwoLevelAggregateHashMapSuite ObjectHashAggregateExecBenchmark SingleLevelAggregateHaspMapSuite TypedImperativeAggregateSuite RewriteDistinctAggregatesSuite HashAggregationQuerySuite HashAggregationQueryWithControlledFallbackSuite TypedImperativeAggregateSuite TwoLevelAggregateHashMapWithVectorizedMapSuite Closes #22706 from SongYadong/generate_constraints. Authored-by: SongYadong <song.yadong1@zte.com.cn> Signed-off-by: gatorsmile <gatorsmile@gmail.com>
1 parent 56247c1 commit 0820484

2 files changed

Lines changed: 6 additions & 6 deletions

File tree

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -152,10 +152,10 @@ abstract class UnaryNode extends LogicalPlan {
152152
override final def children: Seq[LogicalPlan] = child :: Nil
153153

154154
/**
155-
* Generates an additional set of aliased constraints by replacing the original constraint
156-
* expressions with the corresponding alias
155+
* Generates all valid constraints including an set of aliased constraints by replacing the
156+
* original constraint expressions with the corresponding alias
157157
*/
158-
protected def getAliasedConstraints(projectList: Seq[NamedExpression]): Set[Expression] = {
158+
protected def getAllValidConstraints(projectList: Seq[NamedExpression]): Set[Expression] = {
159159
var allConstraints = child.constraints.asInstanceOf[Set[Expression]]
160160
projectList.foreach {
161161
case a @ Alias(l: Literal, _) =>
@@ -170,7 +170,7 @@ abstract class UnaryNode extends LogicalPlan {
170170
case _ => // Don't change.
171171
}
172172

173-
allConstraints -- child.constraints
173+
allConstraints
174174
}
175175

176176
override protected def validConstraints: Set[Expression] = child.constraints

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ case class Project(projectList: Seq[NamedExpression], child: LogicalPlan)
6464
}
6565

6666
override def validConstraints: Set[Expression] =
67-
child.constraints.union(getAliasedConstraints(projectList))
67+
getAllValidConstraints(projectList)
6868
}
6969

7070
/**
@@ -595,7 +595,7 @@ case class Aggregate(
595595

596596
override def validConstraints: Set[Expression] = {
597597
val nonAgg = aggregateExpressions.filter(_.find(_.isInstanceOf[AggregateExpression]).isEmpty)
598-
child.constraints.union(getAliasedConstraints(nonAgg))
598+
getAllValidConstraints(nonAgg)
599599
}
600600
}
601601

0 commit comments

Comments
 (0)