Commit 94d313b
[SPARK-32721][SQL][FOLLOWUP] Simplify if clauses with null and boolean
### What changes were proposed in this pull request?
This is a follow-up on SPARK-32721 and PR #29567. In the previous PR we missed two more cases that can be optimized:
```
if(p, false, null) ==> and(not(p), null)
if(p, true, null) ==> or(p, null)
```
### Why are the changes needed?
By transforming if to boolean conjunctions or disjunctions, we can enable more filter pushdown to datasources.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests.
Closes #29603 from sunchao/SPARK-32721-2.
Authored-by: Chao Sun <sunchao@apache.org>
Signed-off-by: DB Tsai <d_tsai@apple.com>1 parent a701bc7 commit 94d313b
2 files changed
Lines changed: 11 additions & 1 deletion
File tree
- sql/catalyst/src
- main/scala/org/apache/spark/sql/catalyst/optimizer
- test/scala/org/apache/spark/sql/catalyst/optimizer
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
465 | 465 | | |
466 | 466 | | |
467 | 467 | | |
| 468 | + | |
| 469 | + | |
468 | 470 | | |
469 | 471 | | |
470 | 472 | | |
| |||
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
Lines changed: 9 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
166 | 166 | | |
167 | 167 | | |
168 | 168 | | |
169 | | - | |
| 169 | + | |
170 | 170 | | |
171 | 171 | | |
172 | 172 | | |
173 | 173 | | |
| 174 | + | |
| 175 | + | |
174 | 176 | | |
175 | 177 | | |
176 | 178 | | |
177 | 179 | | |
178 | 180 | | |
| 181 | + | |
| 182 | + | |
179 | 183 | | |
180 | 184 | | |
181 | 185 | | |
182 | 186 | | |
183 | 187 | | |
184 | 188 | | |
| 189 | + | |
| 190 | + | |
185 | 191 | | |
186 | 192 | | |
187 | 193 | | |
188 | 194 | | |
189 | 195 | | |
190 | 196 | | |
191 | 197 | | |
| 198 | + | |
| 199 | + | |
192 | 200 | | |
193 | 201 | | |
194 | 202 | | |
0 commit comments