Skip to content

Commit cffd4c7

Browse files
bersprocketsgengliangwang
authored andcommitted
[SPARK-42384][SQL] Check for null input in generated code for mask function
### What changes were proposed in this pull request? When generating code for the mask function, call `ctx.nullSafeExec` to produce null safe code. This change assumes that the mask function returns null only when the input is null (which appears to be the case, from reading the code of `Mask.transformInput`). ### Why are the changes needed? The following query fails with a `NullPointerException`: ``` create or replace temp view v1 as select * from values (null), ('AbCD123-$#') as data(col1); cache table v1; select mask(col1) from v1; 23/02/07 16:36:06 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3) java.lang.NullPointerException at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760) ``` The generated code calls `UnsafeWriter.write(0, value_0)` regardless of whether `Mask.transformInput` returns null or not. The `UnsafeWriter.write` method for `UTF8String` does not expect a null pointer. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit tests. Closes #39945 from bersprockets/mask_npe_issue. Authored-by: Bruce Robbins <bersprockets@gmail.com> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit 7ff8ba2) Signed-off-by: Gengliang Wang <gengliang@apache.org>
1 parent e724f1e commit cffd4c7

3 files changed

Lines changed: 37 additions & 3 deletions

File tree

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -223,16 +223,32 @@ case class Mask(
223223
val fifthGen = children(4).genCode(ctx)
224224
val resultCode =
225225
f(firstGen.value, secondGen.value, thirdGen.value, fourthGen.value, fifthGen.value)
226-
ev.copy(
227-
code = code"""
226+
if (nullable) {
227+
// this function is somewhat like a `UnaryExpression`, in that only the first child
228+
// determines whether the result is null
229+
val nullSafeEval = ctx.nullSafeExec(children(0).nullable, firstGen.isNull)(resultCode)
230+
ev.copy(code = code"""
231+
${firstGen.code}
232+
${secondGen.code}
233+
${thirdGen.code}
234+
${fourthGen.code}
235+
${fifthGen.code}
236+
boolean ${ev.isNull} = ${firstGen.isNull};
237+
${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
238+
$nullSafeEval
239+
""")
240+
} else {
241+
ev.copy(
242+
code = code"""
228243
${firstGen.code}
229244
${secondGen.code}
230245
${thirdGen.code}
231246
${fourthGen.code}
232247
${fifthGen.code}
233248
${CodeGenerator.javaType(dataType)} ${ev.value} = ${CodeGenerator.defaultValue(dataType)};
234249
$resultCode""",
235-
isNull = FalseLiteral)
250+
isNull = FalseLiteral)
251+
}
236252
}
237253

238254
/**

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -440,6 +440,12 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
440440
}
441441
}
442442

443+
test("SPARK-42384: Mask with null input") {
444+
val NULL_LITERAL = Literal(null, StringType)
445+
checkEvaluation(
446+
new Mask(NULL_LITERAL, Literal('Q'), Literal('q'), Literal('d')), null)
447+
}
448+
443449
test("string for ascii") {
444450
val a = $"a".long.at(0)
445451
checkEvaluation(Chr(Literal(48L)), "0", create_row("abdef"))

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -702,4 +702,16 @@ class StringFunctionsSuite extends QueryTest with SharedSparkSession {
702702
)
703703
)
704704
}
705+
706+
test("SPARK-42384: mask with null input") {
707+
val df = Seq(
708+
("AbCD123-@$#"),
709+
(null)
710+
).toDF("a")
711+
712+
checkAnswer(
713+
df.selectExpr("mask(a,'Q','q','d','o')"),
714+
Row("QqQQdddoooo") :: Row(null) :: Nil
715+
)
716+
}
705717
}

0 commit comments

Comments
 (0)