Commit c274c5a
[SPARK-49476][SQL] Fix nullability of base64 function
### What changes were proposed in this pull request?
Fix the nullability of the `Base64` expression to be based on the child's nullability, and not always be nullable.
### Why are the changes needed?
apache#47303 had a side effect of changing the nullability by the switch to using `StaticInvoke`. This was also backported to Spark 3.5.2 and caused schema mismatch errors for stateful streams when we upgraded. This restores the previous behavior which is supported by StaticInvoke through the `returnNullable` argument. If the child is non-nullable, we know the result will be non-nullable.
### Does this PR introduce _any_ user-facing change?
Restores the nullability of the `Base64` expression to what is was in Spark 3.5.1 and earlier.
### How was this patch tested?
New UT
### Was this patch authored or co-authored using generative AI tooling?
No
Closes apache#47941 from Kimahriman/base64-nullability.
Lead-authored-by: Adam Binford <[email protected]>
Co-authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>1 parent cef3c86 commit c274c5a
File tree
2 files changed
+9
-1
lines changed- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/expressions
- test/scala/org/apache/spark/sql/catalyst/expressions
2 files changed
+9
-1
lines changedLines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2714 | 2714 | | |
2715 | 2715 | | |
2716 | 2716 | | |
2717 | | - | |
| 2717 | + | |
| 2718 | + | |
2718 | 2719 | | |
2719 | 2720 | | |
2720 | 2721 | | |
| |||
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
467 | 467 | | |
468 | 468 | | |
469 | 469 | | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
470 | 477 | | |
471 | 478 | | |
472 | 479 | | |
| |||
0 commit comments