Commit bbda3f9
[SPARK-46396][SQL] Timestamp inference should not throw exception (apache#363)
### What changes were proposed in this pull request?
When setting `spark.sql.legacy.timeParserPolicy=LEGACY`, Spark will use the LegacyFastTimestampFormatter to infer potential timestamp columns. The inference shouldn't throw exception.
However, when the input is 23012150952, there is exception:
```
For input string: "23012150952"
java.lang.NumberFormatException: For input string: "23012150952"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.parseInt(Integer.java:786)
at org.apache.commons.lang3.time.FastDateParser$NumberStrategy.parse(FastDateParser.java:304)
at org.apache.commons.lang3.time.FastDateParser.parse(FastDateParser.java:1045)
at org.apache.commons.lang3.time.FastDateFormat.parse(FastDateFormat.java:651)
at org.apache.spark.sql.catalyst.util.LegacyFastTimestampFormatter.parseOptional(TimestampFormatter.scala:418)
```
This PR is to fix the issue.
### Why are the changes needed?
Bug fix, Timestamp inference should not throw exception
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
New test case + existing tests
### Was this patch authored or co-authored using generative AI tooling?
No
Closes apache#44338 from gengliangwang/fixParseOptional.
Authored-by: Gengliang Wang <[email protected]>
(cherry picked from commit 4a79ae9)
Signed-off-by: Gengliang Wang <[email protected]>
Co-authored-by: Gengliang Wang <[email protected]>1 parent 94f10e1 commit bbda3f9
2 files changed
Lines changed: 10 additions & 5 deletions
File tree
- sql
- api/src/main/scala/org/apache/spark/sql/catalyst/util
- catalyst/src/test/scala/org/apache/spark/sql/catalyst/util
Lines changed: 8 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
412 | 412 | | |
413 | 413 | | |
414 | 414 | | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
419 | 423 | | |
420 | 424 | | |
421 | 425 | | |
| |||
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
502 | 502 | | |
503 | 503 | | |
504 | 504 | | |
| 505 | + | |
505 | 506 | | |
506 | 507 | | |
507 | 508 | | |
508 | | - | |
| 509 | + | |
509 | 510 | | |
510 | 511 | | |
0 commit comments