Skip to content

Commit a67acba

Browse files
anchovYucloud-fan
authored andcommitted
[SPARK-37575][SQL][FOLLOWUP] Update the migration guide for added legacy flag for the breaking change of write null value in csv to unquoted empty string
### What changes were proposed in this pull request? This is a follow-up of updating the migration guide for #36110 which adds a legacy flag to restore the pre-change behavior. It also fixes a typo in the previous flag description. ### Why are the changes needed? The flag needs to be documented. ### Does this PR introduce _any_ user-facing change? It changes the migration doc for users. ### How was this patch tested? No tests Closes #36268 from anchovYu/flags-null-to-csv-migration-guide. Authored-by: Xinyi Yu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent 7144e1a commit a67acba

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

docs/sql-migration-guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ license: |
5454

5555
- Since Spark 3.3, the `strfmt` in `format_string(strfmt, obj, ...)` and `printf(strfmt, obj, ...)` will no longer support to use "0$" to specify the first argument, the first argument should always reference by "1$" when use argument index to indicating the position of the argument in the argument list.
5656

57-
- Since Spark 3.3, nulls are written as empty strings in CSV data source by default. In Spark 3.2 or earlier, nulls were written as empty strings as quoted empty strings, `""`. To restore the previous behavior, set `nullValue` to `""`.
57+
- Since Spark 3.3, nulls are written as empty strings in CSV data source by default. In Spark 3.2 or earlier, nulls were written as empty strings as quoted empty strings, `""`. To restore the previous behavior, set `nullValue` to `""`, or set the configuration `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` to `true`.
5858

5959
- Since Spark 3.3, DESCRIBE FUNCTION fails if the function does not exist. In Spark 3.2 or earlier, DESCRIBE FUNCTION can still run and print "Function: func_name not found".
6060

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3758,7 +3758,7 @@ object SQLConf {
37583758
buildConf("spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv")
37593759
.internal()
37603760
.doc("When set to false, nulls are written as unquoted empty strings in CSV data source. " +
3761-
"If set to false, it restores the legacy behavior that nulls were written as quoted " +
3761+
"If set to true, it restores the legacy behavior that nulls were written as quoted " +
37623762
"empty strings, `\"\"`.")
37633763
.version("3.3.0")
37643764
.booleanConf

0 commit comments

Comments
 (0)