Skip to content

Commit 73e3c36

Browse files
Emil EjbyfeldtHyukjinKwon
authored andcommitted
[SPARK-40385][SQL] Fix interpreted path for companion object constructor
### What changes were proposed in this pull request? Fixes encoding of classes that uses companion object constructors in the interpreted path. Without this change the that is added in this change would fail with ``` ... Cause: java.lang.RuntimeException: Error while decoding: java.lang.RuntimeException: Couldn't find a valid constructor on interface org.apache.spark.sql.catalyst.ScroogeLikeExample newInstance(interface org.apache.spark.sql.catalyst.ScroogeLikeExample) at org.apache.spark.sql.errors.QueryExecutionErrors$.expressionDecodingError(QueryExecutionErrors.scala:1199) ... ``` As far as I can tell this bug has existed since the initial implementation in SPARK-8288 #23062 The existing spec that tested this part of the code incorrectly provided an outerPointer which hid the bug from that test. ### Why are the changes needed? Fixes a bug, the new spec in the ExpressionsEncoderSuite shows that this is in fact a bug. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug. ### How was this patch tested? New and existing specs in ExpressionEncoderSuite and ObjectExpressionsSuite. Closes #37837 from eejbyfeldt/spark-40385. Authored-by: Emil Ejbyfeldt <eejbyfeldt@liveintent.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent 969d2d4 commit 73e3c36

3 files changed

Lines changed: 7 additions & 4 deletions

File tree

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -850,8 +850,8 @@ object ScalaReflection extends ScalaReflection {
850850
applyMethods.find { method =>
851851
val params = method.typeSignature.paramLists.head
852852
// Check that the needed params are the same length and of matching types
853-
params.size == paramTypes.tail.size &&
854-
params.zip(paramTypes.tail).forall { case(ps, pc) =>
853+
params.size == paramTypes.size &&
854+
params.zip(paramTypes).forall { case(ps, pc) =>
855855
ps.typeSignature.typeSymbol == mirror.classSymbol(pc)
856856
}
857857
}.map { applyMethodSymbol =>

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ import scala.reflect.runtime.universe.TypeTag
2626

2727
import org.apache.spark.SparkArithmeticException
2828
import org.apache.spark.sql.{Encoder, Encoders}
29-
import org.apache.spark.sql.catalyst.{FooClassWithEnum, FooEnum, OptionalData, PrimitiveData}
29+
import org.apache.spark.sql.catalyst.{FooClassWithEnum, FooEnum, OptionalData, PrimitiveData, ScroogeLikeExample}
3030
import org.apache.spark.sql.catalyst.analysis.AnalysisTest
3131
import org.apache.spark.sql.catalyst.dsl.plans._
3232
import org.apache.spark.sql.catalyst.expressions.AttributeReference
@@ -477,6 +477,9 @@ class ExpressionEncoderSuite extends CodegenInterpretedPlanTest with AnalysisTes
477477
encodeDecodeTest(Option("abc"), "option of string")
478478
encodeDecodeTest(Option.empty[String], "empty option of string")
479479

480+
encodeDecodeTest(ScroogeLikeExample(1),
481+
"SPARK-40385 class with only a companion object constructor")
482+
480483
productTest(("UDT", new ExamplePoint(0.1, 0.2)))
481484

482485
test("AnyVal class with Any fields") {

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -423,7 +423,7 @@ class ObjectExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
423423
inputTypes = Nil,
424424
propagateNull = false,
425425
dataType = ObjectType(classOf[ScroogeLikeExample]),
426-
outerPointer = Some(() => outerObj))
426+
outerPointer = None)
427427
checkObjectExprEvaluation(newInst3, ScroogeLikeExample(1))
428428
}
429429

0 commit comments

Comments
 (0)