Skip to content

Commit c37b851

Browse files
authored
Minor: Add more links to cooperative / scheduling docs (#16484)
1 parent 9556bcd commit c37b851

File tree

2 files changed

+13
-2
lines changed

2 files changed

+13
-2
lines changed

datafusion/core/src/lib.rs

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -498,10 +498,21 @@
498498
//! While preparing for execution, DataFusion tries to create this many distinct
499499
//! `async` [`Stream`]s for each [`ExecutionPlan`].
500500
//! The [`Stream`]s for certain [`ExecutionPlan`]s, such as [`RepartitionExec`]
501-
//! and [`CoalescePartitionsExec`], spawn [Tokio] [`task`]s, that are run by
501+
//! and [`CoalescePartitionsExec`], spawn [Tokio] [`task`]s, that run on
502502
//! threads managed by the [`Runtime`].
503503
//! Many DataFusion [`Stream`]s perform CPU intensive processing.
504504
//!
505+
//! ### Cooperative Scheduling
506+
//!
507+
//! DataFusion uses cooperative scheduling, which means that each [`Stream`]
508+
//! is responsible for yielding control back to the [`Runtime`] after
509+
//! some amount of work is done. Please see the [`coop`] module documentation
510+
//! for more details.
511+
//!
512+
//! [`coop`]: datafusion_physical_plan::coop
513+
//!
514+
//! ### Network I/O and CPU intensive tasks
515+
//!
505516
//! Using `async` for CPU intensive tasks makes it easy for [`TableProvider`]s
506517
//! to perform network I/O using standard Rust `async` during execution.
507518
//! However, this design also makes it very easy to mix CPU intensive and latency

datafusion/physical-plan/src/coop.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
//!
2020
//! # Cooperative scheduling
2121
//!
22-
//! A single call to `poll_next` on a top-level `Stream` may potentially perform a lot of work
22+
//! A single call to `poll_next` on a top-level [`Stream`] may potentially perform a lot of work
2323
//! before it returns a `Poll::Pending`. Think for instance of calculating an aggregation over a
2424
//! large dataset.
2525
//! If a `Stream` runs for a long period of time without yielding back to the Tokio executor,

0 commit comments

Comments
 (0)