Skip to content

Commit 86c3a9d

Browse files
committed
Clarify migration guide comment
1 parent 88dc6b6 commit 86c3a9d

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

docs/sql-migration-guide-upgrade.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,7 @@ license: |
130130

131131
- Since Spark 3.0, Spark will cast `String` to `Date/TimeStamp` in binary comparisons with dates/timestamps. The previous behaviour of casting `Date/Timestamp` to `String` can be restored by setting `spark.sql.legacy.typeCoercion.datetimeToString` to `true`.
132132

133-
- Since Spark 3.0, if files or subdirectories disappear during recursive directory listing (i.e. they appear in an intermediate listing but then cannot be read or listed further, due to either concurrent deletions or object store consistency issues) then the listing will fail with a `FileNotFoundException` unless `spark.sql.files.ignoreMissingFiles` is `true` (default `false`). In previous versions, these `FileNotFoundException`s would be logged as warnings and the missing files or subdirectories would be ignored.
133+
- Since Spark 3.0, if files or subdirectories disappear during recursive directory listing (i.e. they appear in an intermediate listing but then cannot be read or listed during later phases of the recursive directory listing, due to either concurrent file deletions or object store consistency issues) then the listing will fail with an exception unless `spark.sql.files.ignoreMissingFiles` is `true` (default `false`). In previous versions, these missing files or subdirectories would be ignored. Note that this change of behavior only applies during initial table file listing (or during `REFRESH TABLE`), not during query execution: the net change is that `spark.sql.files.ignoreMissingFiles` is now obeyed during table file listing / query planning, not only at query execution time.
134134

135135
## Upgrading from Spark SQL 2.4 to 2.4.1
136136

0 commit comments

Comments
 (0)