Commit 41a7de6
[SPARK-25084][SQL] "distribute by" on multiple columns (wrap in brackets) may lead to codegen issue
## What changes were proposed in this pull request?
"distribute by" on multiple columns (wrap in brackets) may lead to codegen issue.
Simple way to reproduce:
```scala
val df = spark.range(1000)
val columns = (0 until 400).map{ i => s"id as id$i" }
val distributeExprs = (0 until 100).map(c => s"id$c").mkString(",")
df.selectExpr(columns : _*).createTempView("test")
spark.sql(s"select * from test distribute by ($distributeExprs)").count()
```
## How was this patch tested?
Add UT.
Closes #22066 from yucai/SPARK-25084.
Authored-by: yucai <yyu1@ebay.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>1 parent b73eb0e commit 41a7de6
2 files changed
Lines changed: 29 additions & 6 deletions
File tree
- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions
- core/src/test/scala/org/apache/spark/sql
Lines changed: 17 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
404 | 404 | | |
405 | 405 | | |
406 | 406 | | |
| 407 | + | |
407 | 408 | | |
408 | | - | |
| 409 | + | |
409 | 410 | | |
410 | 411 | | |
411 | | - | |
| 412 | + | |
412 | 413 | | |
413 | 414 | | |
414 | | - | |
| 415 | + | |
415 | 416 | | |
416 | 417 | | |
417 | 418 | | |
418 | 419 | | |
419 | 420 | | |
420 | 421 | | |
421 | 422 | | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
422 | 427 | | |
423 | 428 | | |
424 | 429 | | |
| |||
778 | 783 | | |
779 | 784 | | |
780 | 785 | | |
| 786 | + | |
781 | 787 | | |
782 | 788 | | |
783 | 789 | | |
784 | | - | |
| 790 | + | |
785 | 791 | | |
786 | 792 | | |
787 | 793 | | |
788 | 794 | | |
789 | 795 | | |
790 | 796 | | |
791 | 797 | | |
792 | | - | |
| 798 | + | |
793 | 799 | | |
794 | 800 | | |
795 | | - | |
| 801 | + | |
796 | 802 | | |
797 | 803 | | |
798 | 804 | | |
| |||
801 | 807 | | |
802 | 808 | | |
803 | 809 | | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
804 | 815 | | |
805 | 816 | | |
806 | 817 | | |
| |||
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2840 | 2840 | | |
2841 | 2841 | | |
2842 | 2842 | | |
| 2843 | + | |
| 2844 | + | |
| 2845 | + | |
| 2846 | + | |
| 2847 | + | |
| 2848 | + | |
| 2849 | + | |
| 2850 | + | |
| 2851 | + | |
| 2852 | + | |
| 2853 | + | |
| 2854 | + | |
2843 | 2855 | | |
0 commit comments