Commit ef521d3
[SPARK-39354][SQL] Ensure show
### What changes were proposed in this pull request?
After SPARK-38118, `dataTypeMismatchError` related to `Filter` will be checked and throw in `RemoveTempResolvedColumn`, this will cause compatibility issue with exception message presentation.
For example, the following case:
```
spark.sql("create table t1(user_id int, auct_end_dt date) using parquet;")
spark.sql("select * from t1 join t2 on t1.user_id = t2.user_id where t1.auct_end_dt >= Date_sub('2020-12-27', 90)").show
```
The expected message is
```
Table or view not found: t2
```
But the actual message is
```
org.apache.spark.sql.AnalysisException: cannot resolve 'date_sub('2020-12-27', 90)' due to data type mismatch: argument 1 requires date type, however, ''2020-12-27'' is of string type.; line 1 pos 76
```
For forward compatibility, this pr change to only records `DATA_TYPE_MISMATCH_ERROR_MESSAGE` in the `RemoveTempResolvedColumn` check process , and move `failAnalysis` to `CheckAnalysis#checkAnalysis`
### Why are the changes needed?
Fix analysis exception message compatibility.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass Github Actions and add a new test case
Closes #36746 from LuciferYang/SPARK-39354.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
(cherry picked from commit 89fdb8a)
Signed-off-by: Max Gekk <max.gekk@gmail.com>Table or view not found even if there are dataTypeMismatchError related to Filter at the same time1 parent fef5695 commit ef521d3
3 files changed
Lines changed: 32 additions & 8 deletions
File tree
- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/analysis
- test/scala/org/apache/spark/sql/catalyst/analysis
Lines changed: 2 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
4328 | 4328 | | |
4329 | 4329 | | |
4330 | 4330 | | |
4331 | | - | |
4332 | | - | |
4333 | | - | |
4334 | | - | |
| 4331 | + | |
4335 | 4332 | | |
4336 | 4333 | | |
4337 | 4334 | | |
| |||
Lines changed: 16 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
| 54 | + | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
| |||
174 | 176 | | |
175 | 177 | | |
176 | 178 | | |
177 | | - | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
178 | 193 | | |
179 | 194 | | |
180 | 195 | | |
| |||
Lines changed: 14 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1170 | 1170 | | |
1171 | 1171 | | |
1172 | 1172 | | |
1173 | | - | |
| 1173 | + | |
1174 | 1174 | | |
1175 | | - | |
| 1175 | + | |
1176 | 1176 | | |
1177 | 1177 | | |
1178 | 1178 | | |
1179 | 1179 | | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
1180 | 1192 | | |
1181 | 1193 | | |
1182 | 1194 | | |
| |||
0 commit comments