Skip to content

Conversation

@dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Oct 20, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

refactor(query): add experiment aggregate final to support recursive spill

introduce enable_experiment_aggregate_final setting, default is false

  • Temporarily enabled (true) for CI testing convenience; will be turned off before merge

This PR aims to address the OOM issue occurring during the final aggregate stage.

Key changes

  • Introduces recursive repartitioning of large partition payloads until they are sufficiently small.
  • Use stream partition pattern to control the spill and restore payload size

Planned follow-ups

  • Update benchmarks to enable this setting by default.
  • Introduce a setting option to control the maximum recursive spill depth.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test (TODO)
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Oct 20, 2025
@dqhl76 dqhl76 changed the title refactor(query): add experiment aggregate final to support recursive spill refactor(query): experimental aggregate final operator with recursive spill support Oct 20, 2025
@dqhl76 dqhl76 changed the title refactor(query): experimental aggregate final operator with recursive spill support refactor(query): experimental aggregate final with recursive spill support Oct 20, 2025
@dqhl76 dqhl76 force-pushed the group-by-restore-refactor branch from 2a8545c to 9e1c7c3 Compare October 25, 2025 12:44
@dqhl76 dqhl76 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Oct 26, 2025
@github-actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-18866-3f17e07-1761463276

note: this image tag is only available for internal use.

Revert "chore: add random_spill_percentage used for testing"

This reverts commit 0c45ea6
@dqhl76 dqhl76 marked this pull request as ready for review October 27, 2025 00:57
@dqhl76 dqhl76 requested a review from zhang2014 October 27, 2025 00:59
@zhang2014 zhang2014 merged commit cd9cb4e into databendlabs:main Oct 27, 2025
179 of 182 checks passed
@dqhl76 dqhl76 deleted the group-by-restore-refactor branch October 27, 2025 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants