File tree Expand file tree Collapse file tree 2 files changed +13
-2
lines changed
Expand file tree Collapse file tree 2 files changed +13
-2
lines changed Original file line number Diff line number Diff line change 498498//! While preparing for execution, DataFusion tries to create this many distinct
499499//! `async` [`Stream`]s for each [`ExecutionPlan`].
500500//! The [`Stream`]s for certain [`ExecutionPlan`]s, such as [`RepartitionExec`]
501- //! and [`CoalescePartitionsExec`], spawn [Tokio] [`task`]s, that are run by
501+ //! and [`CoalescePartitionsExec`], spawn [Tokio] [`task`]s, that run on
502502//! threads managed by the [`Runtime`].
503503//! Many DataFusion [`Stream`]s perform CPU intensive processing.
504504//!
505+ //! ### Cooperative Scheduling
506+ //!
507+ //! DataFusion uses cooperative scheduling, which means that each [`Stream`]
508+ //! is responsible for yielding control back to the [`Runtime`] after
509+ //! some amount of work is done. Please see the [`coop`] module documentation
510+ //! for more details.
511+ //!
512+ //! [`coop`]: datafusion_physical_plan::coop
513+ //!
514+ //! ### Network I/O and CPU intensive tasks
515+ //!
505516//! Using `async` for CPU intensive tasks makes it easy for [`TableProvider`]s
506517//! to perform network I/O using standard Rust `async` during execution.
507518//! However, this design also makes it very easy to mix CPU intensive and latency
Original file line number Diff line number Diff line change 1919//!
2020//! # Cooperative scheduling
2121//!
22- //! A single call to `poll_next` on a top-level `Stream` may potentially perform a lot of work
22+ //! A single call to `poll_next` on a top-level [ `Stream`] may potentially perform a lot of work
2323//! before it returns a `Poll::Pending`. Think for instance of calculating an aggregation over a
2424//! large dataset.
2525//! If a `Stream` runs for a long period of time without yielding back to the Tokio executor,
You can’t perform that action at this time.
0 commit comments