feat(uv): per-node managed Python venvs at build + runtime (rescue of #1515)

[heyong4725]

Rescues @Harsh-Sahu43's #1515 design (per-node uv-managed Python envs at build + runtime). Reimplemented against current main after 7+ weeks of build/daemon drift made a direct merge impossible (5 conflicts, including the post-#1716 CI rebalance).

Why this matters

Today's --uv is a half-measure: it just swaps pip → uv pip in build commands. It buys uv's speed but none of uv's hermetic-env semantics. Python nodes still run against ambient site-packages, which means:

1. Build/runtime drift — install at build, system Python changes, run uses different Python
2. Cross-dataflow contamination — last dora build wins, no per-dataflow isolation
3. Cross-node contamination within a dataflow — tensorflow node and pytorch node fight over shared site-packages
4. No reproducibility — dora build today, dora run 3 weeks later, transitive deps drift silently

Industry baseline: Python isolation is table stakes for any production-grade Python framework (ROS2 underlay/overlay, Ray runtime_env, Bazel hermetic toolchains, Modal per-function envs).

What this PR does

For Python nodes (.py custom nodes with a build: block, and runtime nodes with Python operators) when --uv is set:

• dora build --uv creates <working-dir>/.dora/python-envs/<node-id>/ per node via uv venv --clear
• Installs the Dora Python API into it (preferring an in-tree workspace package over PyPI, idempotent via import dora probe)
• Runs the node's build commands inside that venv with VIRTUAL_ENV set and the env's bin/ (or Scripts/ on Windows) prepended to PATH
• dora start / dora run automatically reuse the same interpreter at spawn time
• Failures are fail-closed: missing interpreter errors out rather than silently falling through to ambient Python

Custom Python nodes without a build: block keep using ambient uv run python — script-only nodes don't need isolation. Restart path re-derives the env dir deterministically so node restarts pick up the same venv.

Plus a dora doctor uv availability check so users without uv get a clear hint before they hit a cryptic build error.

Pieces

File	Change
libraries/core/src/build/mod.rs	New managed_python_env_dir, managed_python_bin_dir, managed_python_interpreter helpers; Builder::build_node_inner prepares the env when uv=true; new prepare_python_env orchestrator
libraries/core/src/build/build_command.rs	New prepare_managed_python_env, ensure_managed_python_runtime, apply_managed_python_env, managed-PATH composition; run_build_command gains python_env_dir arg
binaries/cli/src/command/build/local.rs + binaries/daemon/src/lib.rs	Capture python_env_dirs per node alongside node_working_dirs
binaries/daemon/src/spawn/spawner.rs + command.rs	Spawn-side uses managed interpreter when one was prepared; falls back to uv run python for unbuilt nodes
binaries/cli/src/command/doctor.rs	New check_uv() step in dora doctor
libraries/core/Cargo.toml	Adds tokio io-util and macros features for build-command output piping
docs/cli.md + guide/src/operations/cli.md	Documents the new --uv semantics

Per-node env path (race fix from pre-landing review)

Initially used <working-dir>/.dora/python-env (matching the original PR design). Pre-landing review caught that Builder.base_working_dir is shared across all nodes — under parallel builds (--mode parallel), two Python nodes sharing a working_dir would race on uv venv --clear against the same directory and silently clobber each other. Fixed to <working-dir>/.dora/python-envs/<node-id>/. Regression test added (env_dir_includes_node_id_so_parallel_builds_do_not_race).

BuildInfo backward compat

#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct BuildInfo {
    pub node_working_dirs: BTreeMap<NodeId, PathBuf>,
    #[serde(default)]
    pub python_env_dirs: BTreeMap<NodeId, PathBuf>,
}

#[serde(default)] means existing build-info YAMLs still parse — verified by build_info_deserializes_without_python_env_dirs.

Tests added

In libraries/core/src/build/mod.rs::tests:

• assigns_managed_env_dir_to_python_custom_nodes — Python .py node gets <working-dir>/.dora/python-envs/<id>
• skips_managed_env_dir_for_non_python_custom_nodes — Rust/C++ nodes get None
• env_dir_includes_node_id_so_parallel_builds_do_not_race — regression test for the race fix
• build_info_deserializes_without_python_env_dirs — old build-info YAML still parses
• managed_python_helpers_use_platform_layout — bin/python on Unix, Scripts/python.exe on Windows

Tests use YAML deserialization (not struct literals) to avoid coupling to the exact ResolvedNode field list.

Local validation

• cargo fmt --all -- --check — clean
• cargo clippy -p dora-core --features build -p dora-cli -p dora-daemon -- -D warnings — clean
• cargo test -p dora-core --features build — 157 passing (5 new)
• cargo test -p dora-cli — 145 passing
• cargo test -p dora-daemon — 32 passing
• dora doctor smoke-run on the host — confirmed PASS uv: uv 0.10.8 (...)
• Authoritative integration test: needs uv on PATH + a real Python dataflow. Reviewer with uv installed can verify:

  dora build --uv examples/python-dataflow/dataflow.yml --local
  ls examples/python-dataflow/.dora/python-envs/   # expect one dir per Python node
  dora run examples/python-dataflow/dataflow.yml --uv --stop-after 5s

  Expected: dora spawns the managed bin/python interpreter rather than uv run python.

Follow-ups (not in this PR)

• Build output truncation — forward_build_output's tokio::select! pattern stops draining one stream when the other closes first, dropping output. The Prototype Dora-managed Python isolation for build and runtime #1515 author had already fixed this with tokio_stream::StreamExt::merge, but porting requires adding tokio-stream to dora-core's Cargo.toml and is tangential to Python env management. Filed as fix(build): build output silently truncated when one of stdout/stderr closes first #1821.
• Author's .github/workflows/ci.yml change — targeted the pre-ci: rebalance PR CI -> fast Linux-only gate, move breadth to nightly #1716 CI layout. Not portable; new nightly CI already exercises Python examples. Explicit --uv env-isolation CI coverage can be a follow-up.

Closes

Closes #1515.

Co-Authored-By: Harsh-Sahu43

🤖 Generated with Claude Code

[trunk-io]

😎 Merged successfully - details.

[heyong4725]

Both findings addressed in 54f126d. Reviewer was right on both — silent degradation in the missing-env case + the runtime not actually being hermetic.

Finding 1 — silent fallback to ambient uv

binaries/daemon/src/lib.rs (the spawn dispatch around the old line 2413) now computes the expected managed env dir for the node and bails with a clear error when:

• uv=true, AND
• python_env_dirs.get(&node_id) is None, AND
• managed_python_env_dir(&node, &node_working_dir) returns Some (i.e., the node would need a managed env)

Error message points the user at the right next step:

node <node_id> is a Python node that needs a managed env under --uv, but no managed env was recorded for it during build. Re-run dora build --uv <dataflow> against the current build session, or omit --uv to run against the ambient Python.

Non-Python nodes are unaffected (the predicate function returns None for them, so the bail doesn't fire).

Finding 2 — runtime not hermetic

New apply_managed_python_runtime_env helper in binaries/daemon/src/spawn/spawner.rs:

• Sets VIRTUAL_ENV = python_env_dir
• Prepends <python_env_dir>/bin (or Scripts on Windows) to PATH
• PATH composition order: managed bin first, then node.env's PATH if set, then daemon's ambient PATH

Called from both spawn paths after the existing env injection but before stdio setup:

• Custom Python: when self.uv && python_env_dir.is_some() && n.build.is_some() — mirrors the condition in path_spawn_command that selects the managed interpreter
• Runtime Python: when self.uv && python_env_dir.is_some() — Python runtime nodes always use the managed interpreter when one was prepared

Why both VIRTUAL_ENV and PATH (not just one): VIRTUAL_ENV is conventional metadata that some tools check; the actual resolution work is done by PATH. Setting both matches what uv itself sets when you uv run a script.

Local validation:

cargo fmt --all -- --check                                            clean
cargo clippy -p dora-core --features build -p dora-cli -p dora-daemon clean
cargo test -p dora-core --features build                              157 passing
cargo test -p dora-cli                                                145 passing
cargo test -p dora-daemon                                             32 passing

Caveat I'd flag separately: the runtime hermeticity fix is exercised end-to-end only by actually running dora start --uv with a Python node that does subprocess.run(["pip", ...]) and seeing the managed pip get invoked. No unit test for that — the env-application is straightforward (read PATH, prepend, write back), but the integration assertion would require spawning a real Python process. Worth a follow-up integration test once qa-examples gains a Python-isolation case.

[heyong4725]

Right, the fail-closed guard was over-fired. Fix in 40f1a7d.

Root cause — three call sites had drifted to different predicates:

Call site	Predicate for "node needs managed env"
Builder::build_node_inner (Custom branch)	is_python_custom_node(n) + outer `.filter(
path_spawn_command (spawn-side fallback)	node.build.is_some() (implicit .py)
managed_python_env_dir	just is_python_custom_node(n)
New guard at lib.rs:2419	managed_python_env_dir(...).is_some()

So managed_python_env_dir answered "is this a .py custom node?" but the actual predicate used by build+spawn was "is this a .py custom node with build:". The guard, which I built on top of managed_python_env_dir, inherited the looser predicate and over-fired.

Fix — pull build.is_some() into node_requires_managed_python_env itself:

fn node_requires_managed_python_env(node: &ResolvedNode) -> bool {
    match &node.kind {
        CoreNodeKind::Custom(custom) => {
            is_python_custom_node(custom) && custom.build.is_some()
        }
        CoreNodeKind::Runtime(runtime) => runtime
            .operators
            .iter()
            .any(|op| matches!(op.config.source, OperatorSource::Python(_))),
    }
}

managed_python_env_dir now answers the right question. The redundant .filter(|_| n.build.is_some()) at Builder::build_node_inner (mod.rs:87) was removed since it's now load-bearing on the function rather than the caller. The spawn-side filter at command.rs:84 and the new guard now both naturally match.

Behavior matrix after the fix:

Node	Managed env?
Custom .py with build:	Yes
Custom .py without build: (script-only)	No — uses ambient uv run python
Custom non-.py	No
Runtime with Python operator	Yes
Runtime non-Python	No

Tests added/updated:

• assigns_managed_env_dir_to_build_backed_python_custom_nodes (renamed from assigns_managed_env_dir_to_python_custom_nodes; test node yaml now includes a build: line)
• skips_managed_env_dir_for_script_only_python_custom_nodes — new regression test; would have caught this exact issue
• env_dir_includes_node_id_so_parallel_builds_do_not_race updated to include build: (otherwise both test nodes would now return None and the test would have been a no-op)

Local validation:

cargo fmt --all -- --check                                            clean
cargo clippy -p dora-core --features build -p dora-cli -p dora-daemon clean
cargo test -p dora-core --features build                              158 passing (was 157)
cargo test -p dora-cli                                                145 passing
cargo test -p dora-daemon                                             32 passing

[heyong4725]

Good catch — the conda_env interaction was a real env-manager-collision bug. Fix in c0986b3.

Root cause: node_requires_managed_python_env (libraries/core/src/build/mod.rs) was any(|op| matches!(op.config.source, OperatorSource::Python(_))) for Runtime nodes — it didn't look at the operator's conda_env field. So a Python operator with conda_env: my-env still got a managed env dir, which meant:

1. Build wastefully ran uv venv --clear .dora/python-envs/<node-id>/ and uv pip install dora-rs even though the runtime spawn was going to ignore that env entirely.
2. Spawn-side, path_spawn_command correctly built conda run -n my-env python -uc "import dora..." for the operator, but my new apply_managed_python_runtime_env then injected VIRTUAL_ENV=.dora/python-envs/... and prepended .dora/python-envs/.../bin to PATH over the top. So the conda interpreter ran, but subprocess.run(["pip", ...]), console scripts, and python -m pip inside the operator would resolve from the uv venv — not the conda env the user asked for. Two env managers fighting.

Fix: require the matched Python operator's conda_env to be None for the Runtime branch of the predicate:

CoreNodeKind::Runtime(runtime) => runtime.operators.iter().any(|operator| {
    matches!(
        &operator.config.source,
        OperatorSource::Python(py) if py.conda_env.is_none()
    )
}),

Applying the fix at the predicate level fixes every downstream site in one shot:

• Build (Builder::build_node_inner): managed_python_env_dir returns None, no venv created, no wasted disk
• Spawn injection (apply_managed_python_runtime_env): its condition is python_env_dir.is_some(), which is now false → no VIRTUAL_ENV/PATH overlay on the conda command
• Fail-closed guard (lib.rs): expected.is_some() is now false → no bail, conda path runs cleanly
• Restart path: re-derives None deterministically → spawn falls through to the conda branch

Regression tests added in libraries/core/src/build/mod.rs::tests:

• skips_managed_env_dir_for_runtime_python_operators_with_conda_env — the actual regression
• assigns_managed_env_dir_for_runtime_python_operators_without_conda_env — confirms we didn't accidentally exclude plain Python operators too

Local validation:

cargo fmt --all -- --check                                            clean
cargo clippy -p dora-core --features build -p dora-cli -p dora-daemon clean
cargo test -p dora-core --features build                              160 passing (2 new)
cargo test -p dora-cli                                                145 passing
cargo test -p dora-daemon                                             32 passing

The pattern emerging across the last few reviews — these all came from managed_python_env_dir's predicate being a different shape than what the build/spawn flows actually expected. The predicate is now load-bearing in three places (build, spawn injection, fail-closed guard, restart) and explicitly enumerates the four real cases (custom .py + build, custom .py script-only, runtime Python + conda_env, runtime Python plain). The tests now cover all four shapes plus the parallel-build race regression.

[heyong4725]

/trunk merge

[phil-opp]

I reviewed the diff rather than the PR description. I see one compatibility risk and one testing gap worth considering.

Potential regression for Python operators without build:

The new managed-env predicate appears to give every runtime Python operator without conda_env: a managed venv, even when the operator has no build: command. That differs from custom .py nodes, where script-only nodes explicitly keep using the ambient uv env.

This could break existing flows like examples/python-operator-dataflow/dataflow_llm.yml, where operators use python: ... without build: and likely rely on dependencies installed in the caller's environment. After this change, dora run --uv would spawn those operators inside a fresh per-node env containing only the Dora runtime package, so imports for packages installed via requirements_llm.txt may fail.

A safer direction would be either:

• only create managed envs for runtime Python operators that have build:, matching the custom-node behavior
• treat this as an intentional breaking change and update docs/examples accordingly

Missing end-to-end coverage for the new contract

The added tests mostly cover path/predicate logic. Given this touches CLI build metadata, daemon spawn, runtime env injection, and Python behavior, I would expect at least one focused smoke/integration test that proves:

• dora build --uv installs into a per-node env
• dora run --uv reuses that interpreter
• subprocess python/pip resolution comes from the managed env

Smaller note

dora doctor now warns about missing uv for everyone, including pure Rust/C++ users. Since it is only a warning, this is not a blocker, but the wording should stay clearly scoped to Python --uv flows to avoid noise.

(generated by codex)