Skip to content

[SPARK-32268][SQL][TESTS][FOLLOW-UP] Use function registry in the SparkSession #36576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 2 commits into from
Closed

[SPARK-32268][SQL][TESTS][FOLLOW-UP] Use function registry in the SparkSession #36576

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c01a877
Use function registry in the SparkSession
HyukjinKwon May 17, 2022
fe1d1ea
Remove if-else checkpoint
HyukjinKwon May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions sql/core/src/test/scala/org/apache/spark/sql/BloomFilterAggregateQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
package org.apache.spark.sql

import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate.BloomFilterAggregate
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec
Expand All @@ -35,23 +34,26 @@ class BloomFilterAggregateQuerySuite extends QueryTest with SharedSparkSession {
val funcId_bloom_filter_agg = new FunctionIdentifier("bloom_filter_agg")
val funcId_might_contain = new FunctionIdentifier("might_contain")

// Register 'bloom_filter_agg' to builtin.
FunctionRegistry.builtin.registerFunction(funcId_bloom_filter_agg,
Copy link
Member Author

@HyukjinKwon HyukjinKwon May 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Scala version or something else caused the test failure by the different class reference order (vs. CI) during running tests:

  1. BloomFilterAggregateQuerySuite gets referred first somehow/somewhere
  2. These functions get registered first (because these previous codes will be executed when BloomFilterAggregateQuerySuite is referred).
  3. Other tests run before BloomFilterAggregateQuerySuite actually runs (and deregisters these functions at afterAll.
  4. Other tests fail because of two functions added by BloomFilterAggregateQuerySuite.

new ExpressionInfo(classOf[BloomFilterAggregate].getName, "bloom_filter_agg"),
(children: Seq[Expression]) => children.size match {
case 1 => new BloomFilterAggregate(children.head)
case 2 => new BloomFilterAggregate(children.head, children(1))
case 3 => new BloomFilterAggregate(children.head, children(1), children(2))
})

// Register 'might_contain' to builtin.
FunctionRegistry.builtin.registerFunction(funcId_might_contain,
new ExpressionInfo(classOf[BloomFilterMightContain].getName, "might_contain"),
(children: Seq[Expression]) => BloomFilterMightContain(children.head, children(1)))
override def beforeAll(): Unit = {
super.beforeAll()
// Register 'bloom_filter_agg' to builtin.
spark.sessionState.functionRegistry.registerFunction(funcId_bloom_filter_agg,
new ExpressionInfo(classOf[BloomFilterAggregate].getName, "bloom_filter_agg"),
(children: Seq[Expression]) => children.size match {
case 1 => new BloomFilterAggregate(children.head)
case 2 => new BloomFilterAggregate(children.head, children(1))
case 3 => new BloomFilterAggregate(children.head, children(1), children(2))
})

// Register 'might_contain' to builtin.
spark.sessionState.functionRegistry.registerFunction(funcId_might_contain,
new ExpressionInfo(classOf[BloomFilterMightContain].getName, "might_contain"),
(children: Seq[Expression]) => BloomFilterMightContain(children.head, children(1)))
}

override def afterAll(): Unit = {
FunctionRegistry.builtin.dropFunction(funcId_bloom_filter_agg)
FunctionRegistry.builtin.dropFunction(funcId_might_contain)
spark.sessionState.functionRegistry.dropFunction(funcId_bloom_filter_agg)
spark.sessionState.functionRegistry.dropFunction(funcId_might_contain)
super.afterAll()
}

Expand Down
3 changes: 3 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,9 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
test("SPARK-14415: All functions should have own descriptions") {
for (f <- spark.sessionState.functionRegistry.listFunction()) {
if (!Seq("cube", "grouping", "grouping_id", "rollup").contains(f.unquotedString)) {
if (f.unquotedString == "bloom_filter_agg") {
Copy link
Contributor

@cloud-fan cloud-fan May 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this? it looks weird to handle bloom_filter_agg here.

Copy link
Member Author

@HyukjinKwon HyukjinKwon May 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops mistake.

Copy link
Member Author

@HyukjinKwon HyukjinKwon May 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was debugging with checkpointing here with IDE 😂

null
}
checkKeywordsNotExist(sql(s"describe function $f"), "N/A.")
}
}
Expand Down