Skip to content

Commit 32789ba

Browse files
JoshRosendongjoon-hyun
authored andcommitted
[SPARK-48081] Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type
### What changes were proposed in this pull request? While migrating the `NTile` expression's type check failures to the new error class framework, PR #38457 removed a pair of not-unnecessary `return` statements and thus caused certain branches' values to be discarded rather than returned. As a result, invalid usages like ``` select ntile(99.9) OVER (order by id) from range(10) ``` trigger internal errors like errors like ``` java.lang.ClassCastException: class org.apache.spark.sql.types.Decimal cannot be cast to class java.lang.Integer (org.apache.spark.sql.types.Decimal is in unnamed module of loader 'app'; java.lang.Integer is in module java.base of loader 'bootstrap') at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:99) at org.apache.spark.sql.catalyst.expressions.NTile.checkInputDataTypes(windowExpressions.scala:877) ``` instead of clear error framework errors like ``` org.apache.spark.sql.catalyst.ExtendedAnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "ntile(99.9)" due to data type mismatch: The first parameter requires the "INT" type, however "99.9" has the type "DECIMAL(3,1)". SQLSTATE: 42K09; line 1 pos 7; 'Project [unresolvedalias(ntile(99.9) windowspecdefinition(id#0L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())))] +- Range (0, 10, step=1, splits=None) at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:73) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7(CheckAnalysis.scala:315) ``` ### Why are the changes needed? Improve error messages. ### Does this PR introduce _any_ user-facing change? Yes, it improves an error message. ### How was this patch tested? Added a new test case to AnalysisErrorSuite. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46333 from JoshRosen/SPARK-48081. Authored-by: Josh Rosen <joshrosen@databricks.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit b99a64b) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 70ce67c commit 32789ba

2 files changed

Lines changed: 36 additions & 2 deletions

File tree

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -848,7 +848,7 @@ case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindow
848848
// for each partition.
849849
override def checkInputDataTypes(): TypeCheckResult = {
850850
if (!buckets.foldable) {
851-
DataTypeMismatch(
851+
return DataTypeMismatch(
852852
errorSubClass = "NON_FOLDABLE_INPUT",
853853
messageParameters = Map(
854854
"inputName" -> "buckets",
@@ -859,7 +859,7 @@ case class NTile(buckets: Expression) extends RowNumberLike with SizeBasedWindow
859859
}
860860

861861
if (buckets.dataType != IntegerType) {
862-
DataTypeMismatch(
862+
return DataTypeMismatch(
863863
errorSubClass = "UNEXPECTED_INPUT_TYPE",
864864
messageParameters = Map(
865865
"paramIndex" -> "1",

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,40 @@ class AnalysisErrorSuite extends AnalysisTest {
316316
listRelation.select(Explode($"list").as("a"), Explode($"list").as("b")),
317317
"only one generator" :: "explode" :: Nil)
318318

319+
errorClassTest(
320+
"the buckets of ntile window function is not foldable",
321+
testRelation2.select(
322+
WindowExpression(
323+
NTile(Literal(99.9f)),
324+
WindowSpecDefinition(
325+
UnresolvedAttribute("a") :: Nil,
326+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
327+
UnspecifiedFrame)).as("window")),
328+
errorClass = "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
329+
messageParameters = Map(
330+
"sqlExpr" -> "\"ntile(99.9)\"",
331+
"paramIndex" -> "first",
332+
"inputSql" -> "\"99.9\"",
333+
"inputType" -> "\"FLOAT\"",
334+
"requiredType" -> "\"INT\""))
335+
336+
337+
errorClassTest(
338+
"the buckets of ntile window function is not int literal",
339+
testRelation2.select(
340+
WindowExpression(
341+
NTile(AttributeReference("b", IntegerType)()),
342+
WindowSpecDefinition(
343+
UnresolvedAttribute("a") :: Nil,
344+
SortOrder(UnresolvedAttribute("b"), Ascending) :: Nil,
345+
UnspecifiedFrame)).as("window")),
346+
errorClass = "DATATYPE_MISMATCH.NON_FOLDABLE_INPUT",
347+
messageParameters = Map(
348+
"sqlExpr" -> "\"ntile(b)\"",
349+
"inputName" -> "`buckets`",
350+
"inputExpr" -> "\"b\"",
351+
"inputType" -> "\"INT\""))
352+
319353
errorClassTest(
320354
"unresolved attributes",
321355
testRelation.select($"abcd"),

0 commit comments

Comments
 (0)