Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -884,12 +884,16 @@ config_namespace! {
/// into the file scan phase.
pub enable_join_dynamic_filter_pushdown: bool, default = true

/// When set to true attempts to push down dynamic filters generated by operators (topk & join) into the file scan phase.
/// When set to true, the optimizer will attempt to push down Aggregate dynamic filters
/// into the file scan phase.
pub enable_aggregate_dynamic_filter_pushdown: bool, default = true

/// When set to true attempts to push down dynamic filters generated by operators (TopK, Join & Aggregate) into the file scan phase.
/// For example, for a query such as `SELECT * FROM t ORDER BY timestamp DESC LIMIT 10`, the optimizer
/// will attempt to push down the current top 10 timestamps that the TopK operator references into the file scans.
/// This means that if we already have 10 timestamps in the year 2025
/// any files that only have timestamps in the year 2024 can be skipped / pruned at various stages in the scan.
/// The config will suppress `enable_join_dynamic_filter_pushdown` & `enable_topk_dynamic_filter_pushdown`
/// The config will suppress `enable_join_dynamic_filter_pushdown`, `enable_topk_dynamic_filter_pushdown` & `enable_aggregate_dynamic_filter_pushdown`
/// So if you disable `enable_topk_dynamic_filter_pushdown`, then enable `enable_dynamic_filter_pushdown`, the `enable_topk_dynamic_filter_pushdown` will be overridden.
pub enable_dynamic_filter_pushdown: bool, default = true

Expand Down Expand Up @@ -1201,6 +1205,7 @@ impl ConfigOptions {
self.optimizer.enable_dynamic_filter_pushdown = bool_value;
self.optimizer.enable_topk_dynamic_filter_pushdown = bool_value;
self.optimizer.enable_join_dynamic_filter_pushdown = bool_value;
self.optimizer.enable_aggregate_dynamic_filter_pushdown = bool_value;
}
return Ok(());
}
Expand Down
Loading