Optimize planning for projected nested union #18713

logan-keede · 2025-11-14T16:28:58Z

Which issue does this PR close?

part of Planning time for queries with many columns with union and order by is very slow #17261.

Rationale for this change

as explained in #17261 (comment) and discussion in the above mentioned issue

What changes are included in this PR?

Projected Nested Union are now merged with their parent union.

Are these changes tested?

Tested it by Github CI. Works for the scenario discussed in the above mentioned issue.

> explain analyze (SELECT c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c0)
  UNION ALL 
(SELECT 0 as c0, c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c1)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c2)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c3)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c4)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, c5, 6 as c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c5)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, c6, 7 as c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c6)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, c7, 8 as c8, 9 as c9 FROM t1 ORDER BY c7)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, c8, 9 as c9 FROM t1 ORDER BY c8)
  UNION ALL 
(SELECT 0 as c0, 1 as c1, 2 as c2, 3 as c3, 4 as c4, 5 as c5, 6 as c6, 7 as c7, 8 as c8, c9 FROM t1 ORDER BY c9)
ORDER BY c0, c1, c2, c3, c4, c5, c6, c7, c8, c9;
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | SortPreservingMergeExec: [c0@0 ASC NULLS LAST, c1@1 ASC NULLS LAST, c2@2 ASC NULLS LAST, c3@3 ASC NULLS LAST, c4@4 ASC NULLS LAST, c5@5 ASC NULLS LAST, c6@6 ASC NULLS LAST, c7@7 ASC NULLS LAST, c8@8 ASC NULLS LAST, c9@9 ASC NULLS LAST], metrics=[output_rows=0, elapsed_compute=3.817µs, output_bytes=0.0 B, output_batches=0]                                                                                                                                                        |
|                   |   UnionExec, metrics=[output_rows=0, elapsed_compute=345.775µs, output_bytes=0.0 B, output_batches=0]                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c0@0 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(c0@0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c1@1 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(c1@0 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c2@2 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(c2@0 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c3@3 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(c3@0 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c4@4 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(c4@0 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c5@5 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(c5@0 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c6@6 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(c6@0 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c7@7 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(c7@0 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c8@8 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(c8@0 AS Decimal128(20, 0)) as c8, CAST(9 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |     SortExec: expr=[c9@9 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0, spill_count=0, spilled_bytes=0.0 B, spilled_rows=0, batches_split=0]                                                                                                                                                                                                                                                           |
|                   |       ProjectionExec: expr=[CAST(0 AS Decimal128(20, 0)) as c0, CAST(1 AS Decimal128(20, 0)) as c1, CAST(2 AS Decimal128(20, 0)) as c2, CAST(3 AS Decimal128(20, 0)) as c3, CAST(4 AS Decimal128(20, 0)) as c4, CAST(5 AS Decimal128(20, 0)) as c5, CAST(6 AS Decimal128(20, 0)) as c6, CAST(7 AS Decimal128(20, 0)) as c7, CAST(8 AS Decimal128(20, 0)) as c8, CAST(c9@0 AS Decimal128(20, 0)) as c9], metrics=[output_rows=0, elapsed_compute=1ns, output_bytes=0.0 B, output_batches=0] |
|                   |         DataSourceExec: partitions=1, partition_sizes=[0], metrics=[]                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Are there any user-facing changes?

Yes? logical plans for some queries should be different(and better) now. No syntax changes.

logan-keede · 2025-11-14T17:05:55Z

cc @alamb @Omega359

alamb · 2025-11-14T17:10:15Z

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing optimize_projected_union (f2e5d0a) to 1dddf03 diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=optimize_projected_union
Results will be posted here when complete

Omega359 · 2025-11-14T18:00:31Z

I'll run the sql_planner_extended benchmark against this branch as that is the one that to me most reflects whether #17261 should be closed. I'm hopeful :)

alamb · 2025-11-14T18:04:47Z

I'll run the sql_planner_extended benchmark against this branch as that is the one that to me most reflects whether #17261 should be closed. I'm hopeful :)

I have my vm in gc working on it too

alamb · 2025-11-14T18:14:33Z

🤖: Benchmark completed

Details

group                                                 main                                   optimize_projected_union
-----                                                 ----                                   ------------------------
logical_aggregate_with_join                           1.00    632.8±3.04µs        ? ?/sec    1.00    632.5±3.30µs        ? ?/sec
logical_select_all_from_1000                          1.00     10.7±0.04ms        ? ?/sec    1.00     10.7±0.05ms        ? ?/sec
logical_select_one_from_700                           1.00    420.9±1.60µs        ? ?/sec    1.00    419.2±3.94µs        ? ?/sec
logical_trivial_join_high_numbered_columns            1.00    378.4±3.25µs        ? ?/sec    1.00    376.7±2.37µs        ? ?/sec
logical_trivial_join_low_numbered_columns             1.00    363.5±1.39µs        ? ?/sec    1.00    362.1±3.11µs        ? ?/sec
physical_intersection                                 1.00    842.1±4.94µs        ? ?/sec    1.00    840.2±4.32µs        ? ?/sec
physical_join_consider_sort                           1.00  1400.4±14.40µs        ? ?/sec    1.00  1405.8±12.64µs        ? ?/sec
physical_join_distinct                                1.01    355.3±2.13µs        ? ?/sec    1.00    353.0±1.16µs        ? ?/sec
physical_many_self_joins                              1.00      9.7±0.04ms        ? ?/sec    1.00      9.6±0.03ms        ? ?/sec
physical_plan_clickbench_all                          1.00    178.9±2.09ms        ? ?/sec    1.01    181.0±1.55ms        ? ?/sec
physical_plan_clickbench_q1                           1.00      2.3±0.02ms        ? ?/sec    1.01      2.3±0.02ms        ? ?/sec
physical_plan_clickbench_q10                          1.00      3.2±0.03ms        ? ?/sec    1.01      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q11                          1.00      3.4±0.02ms        ? ?/sec    1.01      3.4±0.04ms        ? ?/sec
physical_plan_clickbench_q12                          1.00      3.5±0.03ms        ? ?/sec    1.01      3.6±0.04ms        ? ?/sec
physical_plan_clickbench_q13                          1.00      3.2±0.03ms        ? ?/sec    1.01      3.2±0.04ms        ? ?/sec
physical_plan_clickbench_q14                          1.00      3.4±0.02ms        ? ?/sec    1.00      3.4±0.04ms        ? ?/sec
physical_plan_clickbench_q15                          1.00      3.2±0.03ms        ? ?/sec    1.01      3.3±0.03ms        ? ?/sec
physical_plan_clickbench_q16                          1.00      3.1±0.07ms        ? ?/sec    1.00      3.1±0.04ms        ? ?/sec
physical_plan_clickbench_q17                          1.00      3.2±0.04ms        ? ?/sec    1.00      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q18                          1.00      2.7±0.02ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q19                          1.00      3.6±0.04ms        ? ?/sec    1.01      3.6±0.03ms        ? ?/sec
physical_plan_clickbench_q2                           1.00      2.7±0.03ms        ? ?/sec    1.01      2.8±0.02ms        ? ?/sec
physical_plan_clickbench_q20                          1.00      2.5±0.02ms        ? ?/sec    1.01      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q21                          1.00      2.8±0.06ms        ? ?/sec    1.00      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q22                          1.00      3.4±0.04ms        ? ?/sec    1.00      3.4±0.02ms        ? ?/sec
physical_plan_clickbench_q23                          1.00      3.6±0.03ms        ? ?/sec    1.01      3.7±0.05ms        ? ?/sec
physical_plan_clickbench_q24                          1.00      4.1±0.04ms        ? ?/sec    1.01      4.1±0.03ms        ? ?/sec
physical_plan_clickbench_q25                          1.00      2.9±0.02ms        ? ?/sec    1.00      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q26                          1.00      2.7±0.02ms        ? ?/sec    1.01      2.7±0.03ms        ? ?/sec
physical_plan_clickbench_q27                          1.00      3.0±0.02ms        ? ?/sec    1.03      3.0±0.16ms        ? ?/sec
physical_plan_clickbench_q28                          1.00      3.7±0.04ms        ? ?/sec    1.01      3.7±0.13ms        ? ?/sec
physical_plan_clickbench_q29                          1.00      4.0±0.04ms        ? ?/sec    1.01      4.0±0.05ms        ? ?/sec
physical_plan_clickbench_q3                           1.00      2.7±0.02ms        ? ?/sec    1.01      2.7±0.05ms        ? ?/sec
physical_plan_clickbench_q30                          1.00     12.7±0.13ms        ? ?/sec    1.01     12.8±0.13ms        ? ?/sec
physical_plan_clickbench_q31                          1.00      3.6±0.03ms        ? ?/sec    1.02      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q32                          1.00      3.7±0.03ms        ? ?/sec    1.01      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q33                          1.00      3.2±0.02ms        ? ?/sec    1.01      3.2±0.04ms        ? ?/sec
physical_plan_clickbench_q34                          1.00      2.9±0.02ms        ? ?/sec    1.01      2.9±0.05ms        ? ?/sec
physical_plan_clickbench_q35                          1.00      3.0±0.04ms        ? ?/sec    1.01      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q36                          1.00      3.7±0.03ms        ? ?/sec    1.00      3.7±0.03ms        ? ?/sec
physical_plan_clickbench_q37                          1.00      3.8±0.03ms        ? ?/sec    1.02      3.9±0.03ms        ? ?/sec
physical_plan_clickbench_q38                          1.00      3.8±0.05ms        ? ?/sec    1.01      3.9±0.04ms        ? ?/sec
physical_plan_clickbench_q39                          1.00      3.7±0.05ms        ? ?/sec    1.04      3.8±0.06ms        ? ?/sec
physical_plan_clickbench_q4                           1.00      2.4±0.02ms        ? ?/sec    1.02      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q40                          1.00      4.5±0.07ms        ? ?/sec    1.05      4.7±0.13ms        ? ?/sec
physical_plan_clickbench_q41                          1.00      3.9±0.05ms        ? ?/sec    1.05      4.1±0.10ms        ? ?/sec
physical_plan_clickbench_q42                          1.00      3.8±0.04ms        ? ?/sec    1.05      4.0±0.06ms        ? ?/sec
physical_plan_clickbench_q43                          1.00      4.2±0.05ms        ? ?/sec    1.05      4.4±0.05ms        ? ?/sec
physical_plan_clickbench_q44                          1.00      2.6±0.02ms        ? ?/sec    1.04      2.7±0.03ms        ? ?/sec
physical_plan_clickbench_q45                          1.00      2.5±0.02ms        ? ?/sec    1.03      2.6±0.03ms        ? ?/sec
physical_plan_clickbench_q46                          1.00      3.0±0.02ms        ? ?/sec    1.04      3.1±0.04ms        ? ?/sec
physical_plan_clickbench_q47                          1.00      3.5±0.04ms        ? ?/sec    1.03      3.7±0.05ms        ? ?/sec
physical_plan_clickbench_q48                          1.00      4.3±0.03ms        ? ?/sec    1.05      4.5±0.06ms        ? ?/sec
physical_plan_clickbench_q49                          1.00      4.6±0.05ms        ? ?/sec    1.04      4.8±0.08ms        ? ?/sec
physical_plan_clickbench_q5                           1.00      2.6±0.02ms        ? ?/sec    1.01      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q50                          1.00      4.1±0.04ms        ? ?/sec    1.02      4.2±0.04ms        ? ?/sec
physical_plan_clickbench_q51                          1.00      3.1±0.03ms        ? ?/sec    1.01      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q6                           1.00      2.6±0.02ms        ? ?/sec    1.01      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q7                           1.00      2.4±0.02ms        ? ?/sec    1.01      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q8                           1.00      3.2±0.04ms        ? ?/sec    1.01      3.2±0.02ms        ? ?/sec
physical_plan_clickbench_q9                           1.00      3.1±0.03ms        ? ?/sec    1.00      3.1±0.02ms        ? ?/sec
physical_plan_tpcds_all                               1.00   1034.8±4.22ms        ? ?/sec    1.00   1036.8±3.43ms        ? ?/sec
physical_plan_tpch_all                                1.00     64.2±0.51ms        ? ?/sec    1.01     64.6±0.34ms        ? ?/sec
physical_plan_tpch_q1                                 1.00      2.0±0.01ms        ? ?/sec    1.00      2.0±0.01ms        ? ?/sec
physical_plan_tpch_q10                                1.00      4.0±0.01ms        ? ?/sec    1.00      3.9±0.01ms        ? ?/sec
physical_plan_tpch_q11                                1.01      3.5±0.02ms        ? ?/sec    1.00      3.5±0.02ms        ? ?/sec
physical_plan_tpch_q12                                1.00   1803.2±8.13µs        ? ?/sec    1.00   1811.3±7.38µs        ? ?/sec
physical_plan_tpch_q13                                1.00  1452.0±11.22µs        ? ?/sec    1.00   1451.9±7.04µs        ? ?/sec
physical_plan_tpch_q14                                1.00  1961.0±13.52µs        ? ?/sec    1.01  1986.2±10.66µs        ? ?/sec
physical_plan_tpch_q16                                1.00      2.5±0.01ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
physical_plan_tpch_q17                                1.01      2.6±0.02ms        ? ?/sec    1.00      2.6±0.01ms        ? ?/sec
physical_plan_tpch_q18                                1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
physical_plan_tpch_q19                                1.00      3.2±0.02ms        ? ?/sec    1.01      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q2                                 1.00      5.9±0.03ms        ? ?/sec    1.00      5.9±0.10ms        ? ?/sec
physical_plan_tpch_q20                                1.01      3.2±0.03ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q21                                1.00      4.2±0.02ms        ? ?/sec    1.00      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q22                                1.00      2.8±0.01ms        ? ?/sec    1.01      2.9±0.02ms        ? ?/sec
physical_plan_tpch_q3                                 1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.05ms        ? ?/sec
physical_plan_tpch_q4                                 1.00  1489.3±10.46µs        ? ?/sec    1.00   1490.4±6.70µs        ? ?/sec
physical_plan_tpch_q5                                 1.00      3.2±0.01ms        ? ?/sec    1.00      3.2±0.01ms        ? ?/sec
physical_plan_tpch_q6                                 1.00    852.8±5.82µs        ? ?/sec    1.00    852.2±6.40µs        ? ?/sec
physical_plan_tpch_q7                                 1.00      4.2±0.01ms        ? ?/sec    1.00      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q8                                 1.00      5.5±0.02ms        ? ?/sec    1.00      5.4±0.02ms        ? ?/sec
physical_plan_tpch_q9                                 1.01      4.0±0.02ms        ? ?/sec    1.00      4.0±0.01ms        ? ?/sec
physical_select_aggregates_from_200                   1.01     16.9±0.06ms        ? ?/sec    1.00     16.8±0.08ms        ? ?/sec
physical_select_all_from_1000                         1.00     23.5±0.14ms        ? ?/sec    1.00     23.5±0.14ms        ? ?/sec
physical_select_one_from_700                          1.00   1085.3±4.93µs        ? ?/sec    1.00   1085.0±7.86µs        ? ?/sec
physical_sorted_union_order_by_10_int64               1.00      6.0±0.02ms        ? ?/sec    1.00      6.0±0.02ms        ? ?/sec
physical_sorted_union_order_by_10_uint64              1.00     13.0±0.15ms        ? ?/sec    1.23     15.9±0.09ms        ? ?/sec
physical_sorted_union_order_by_50_int64               1.00    162.8±1.15ms        ? ?/sec    1.00    162.3±1.02ms        ? ?/sec
physical_sorted_union_order_by_50_uint64              1.00    374.4±2.21ms        ? ?/sec    2.54    952.4±5.61ms        ? ?/sec
physical_theta_join_consider_sort                     1.00  1760.8±13.45µs        ? ?/sec    1.01   1770.9±7.47µs        ? ?/sec
physical_unnest_to_join                               1.00  1858.9±17.32µs        ? ?/sec    1.00   1859.8±8.00µs        ? ?/sec
physical_window_function_partition_by_12_on_values    1.00  1087.2±12.54µs        ? ?/sec    1.00   1092.2±8.59µs        ? ?/sec
physical_window_function_partition_by_30_on_values    1.00      2.2±0.01ms        ? ?/sec    1.00      2.2±0.01ms        ? ?/sec
physical_window_function_partition_by_4_on_values     1.01    661.5±3.93µs        ? ?/sec    1.00    656.2±8.07µs        ? ?/sec
physical_window_function_partition_by_7_on_values     1.00    808.1±6.34µs        ? ?/sec    1.01    812.9±6.26µs        ? ?/sec
physical_window_function_partition_by_8_on_values     1.00    865.7±4.93µs        ? ?/sec    1.00    866.8±3.68µs        ? ?/sec
with_param_values_many_columns                        1.00    646.8±7.46µs        ? ?/sec    1.00    647.3±6.61µs        ? ?/sec

alamb · 2025-11-14T18:24:29Z

physical_sorted_union_order_by_10_int64               1.00      6.0±0.02ms        ? ?/sec    1.00      6.0±0.02ms        ? ?/sec
physical_sorted_union_order_by_10_uint64              1.00     13.0±0.15ms        ? ?/sec    1.23     15.9±0.09ms        ? ?/sec
physical_sorted_union_order_by_50_int64               1.00    162.8±1.15ms        ? ?/sec    1.00    162.3±1.02ms        ? ?/sec
physical_sorted_union_order_by_50_uint64              1.00    374.4±2.21ms        ? ?/sec    2.54    952.4±5.61ms        ? ?/sec

1.23.- 2.5x better certainly seems much better 🎉 ! Though I am not sure if we have fixed the exponential planning time

Omega359 · 2025-11-14T19:27:30Z

Given the time estimate on my machine for the extended planner test I'm going to assume it's not impacting that benchmark as much as I'd like to see:

Running benches/sql_planner_extended.rs (target/release/deps/sql_planner_extended-4fd78f102f98e6a6)
Benchmarking logical_plan_optimize: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6063.2s, or reduce sample count to 10.
Benchmarking logical_plan_optimize: Collecting 100 samples in estimated 6063.2 s (100 iterations

logan-keede · 2025-11-14T21:42:22Z

I guess this won't be enough to close the issue but I will take any small wins for now.
I will try and see if I can find any other lead that can explain the exponential planning time.

Omega359 · 2025-11-14T23:16:08Z

ogical_plan_optimize   time:   [68.066 s 68.685 s 69.325 s]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild

+ critcmp main optimize_projected_union
group                    main                                   optimize_projected_union
-----                    ----                                   ------------------------
logical_plan_optimize    1.00      68.7±3.19s        ? ?/sec    1.00      68.7±8.30s        ? ?/sec

logan-keede · 2025-11-22T12:51:01Z

@alamb @Omega359 Do you want to merge these changes? If so, I will update this branch. Please feel free to close it otherwise.

alamb

Thanks @logan-keede -- sorry I haven't reviewed this earlier

Can you please add tests for this PR showing what behavior has changed? I didn't see anything different in the tests, so I worry the behavior might regress in the future and we wouldn't be able to tell

alamb · 2025-11-22T13:22:49Z

datafusion/optimizer/src/eliminate_nested_union.rs

        _config: &dyn OptimizerConfig,
    ) -> Result<Transformed<LogicalPlan>> {
-        match plan {
+        match plan.clone() {


why did you add this clone?

leftover from something I was trying earlier, I thought clippy catches this stuff ...

… optimize_projected_union

alamb

I think this is an improvement in my mind. Thank you @logan-keede and @Omega359

I don't think it closes the original issue, but it is a nice improvement

alamb · 2025-11-23T13:46:24Z

datafusion/optimizer/src/optimize_unions.rs

            .into_iter()
            .map(Arc::unwrap_or_clone)
            .collect::<Vec<_>>(),
+        LogicalPlan::Projection(Projection {


Am I right in understanding this change that it transforms

Projection Union

to

Union Projection

(aka it pushes the projection to each input of the union?)

Maybe some comments could help future readers

alamb · 2025-11-23T13:50:04Z

datafusion/optimizer/src/optimize_unions.rs

    }

+    #[test]
+    fn eliminate_nested_union_in_projection() -> Result<()> {


Without the change in this PR the test fails like

────────────┬─────────────────────────────────────────────────────────────────── 1 1 │ Union 2 2 │ Projection: id AS table_id, key, value 3 │- TableScan: table 4 │- Projection: id AS table_id, key, value 5 │- TableScan: table 3 │+ Union 4 │+ TableScan: table 5 │+ TableScan: table 6 6 │ TableScan: table ────────────┴───────────────────────────────────────────────────────────────────

In other words, the plan is left as

Union Projection: id AS table_id, key, value Union TableScan: table TableScan: table TableScan: table

alamb · 2025-11-25T14:09:32Z

Thanks again @logan-keede and @Omega359

logan-keede added 3 commits November 14, 2025 03:24

It works ig

ea59584

Okay, this one maybe, wrapped projection nicely

466ca17

clippy

c516035

github-actions bot added the optimizer Optimizer rules label Nov 14, 2025

Merge branch 'main' into optimize_projected_union

f2e5d0a

alamb mentioned this pull request Nov 14, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-17 #18711

Closed

46 tasks

alamb reviewed Nov 22, 2025

View reviewed changes

github-actions bot added ffi Changes to the ffi crate physical-plan Changes to the physical-plan crate spark labels Nov 23, 2025

Merge branch 'main' of https://github.com/logan-keede/datafusion into…

91b6c9a

… optimize_projected_union

logan-keede force-pushed the optimize_projected_union branch from 84049a8 to 91b6c9a Compare November 23, 2025 00:09

remove clone

d9acde5

alamb mentioned this pull request Nov 23, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-24 #18888

Closed

37 tasks

alamb approved these changes Nov 23, 2025

View reviewed changes

add inline comment for explanation

681a173

alamb added this pull request to the merge queue Nov 25, 2025

Merged via the queue into apache:main with commit 48cc4c8 Nov 25, 2025
31 checks passed

Optimize planning for projected nested union #18713

Optimize planning for projected nested union #18713

Uh oh!

Conversation

logan-keede commented Nov 14, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

logan-keede commented Nov 14, 2025

Uh oh!

alamb commented Nov 14, 2025

Uh oh!

Omega359 commented Nov 14, 2025

Uh oh!

alamb commented Nov 14, 2025

Uh oh!

alamb commented Nov 14, 2025

Uh oh!

alamb commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Omega359 commented Nov 14, 2025

Uh oh!

logan-keede commented Nov 14, 2025

Uh oh!

Omega359 commented Nov 14, 2025

Uh oh!

logan-keede commented Nov 22, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

logan-keede Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

logan-keede commented Nov 14, 2025 •

edited by alamb

Loading

alamb commented Nov 14, 2025 •

edited

Loading

alamb Nov 23, 2025 •

edited

Loading