feat: add `conda/outputs` backend protocol by baszalmstra · Pull Request #4056 · prefix-dev/pixi

baszalmstra · 2025-06-30T11:09:06Z

This PR adds the conda/outputs backend protocol method. Instead of letting the build backends solve the host and build environments, with this PR pixi itself takes care of that. We do this to allow pixi to act as the build orchestrator and to allow host and build environments of packages to also contain source dependencies.

The protocol method is not finalized since we already identified some weaknesses. Im putting this PR up now so we can already start doing initial reviews.

Some notable changes:

I tried to remove references to (Nameless)MatchSpec as much as possible in favour of PixiSpec/BinarySpec. This provides a more homogeneous API on the pixi side.

crates/pixi_build_type_conversions/src/project_model.rs

baszalmstra · 2025-06-30T11:14:26Z

crates/pixi_build_types/src/procedures/conda_outputs.rs

@@ -0,0 +1,211 @@
+//! Describes the `conda/outputs` request and its parameters.


Please take a very close look at this file. This describes the conda/outputs protocol. We want to make sure we dont have to change this.

baszalmstra · 2025-06-30T11:18:21Z

crates/pixi_build_types/src/procedures/conda_outputs.rs

+// #[serde_as]
+// #[derive(Debug, Serialize, Deserialize, Clone, Hash, Eq, PartialEq)]
+// #[serde(rename_all = "camelCase")]
+// pub struct CondaCacheMetadata {


I explicitly left this in

baszalmstra · 2025-06-30T11:20:31Z

crates/pixi_command_dispatcher/src/instantiate_tool_env/mod.rs


        // Determine the cache key for the environment.
        let cache_key = self.cache_key();
-        let name = self.requirement.0.as_source().to_string();


I changed the code below because I always want to solve the environment to ensure we get the latest versions.

crates/pixi_build_types/src/procedures/conda_outputs.rs

Hofer-Julian

Another part of the review :)

crates/pixi_build_types/src/project_model.rs

crates/pixi_build_type_conversions/src/project_model.rs

crates/pixi_build_types/src/procedures/conda_outputs.rs

Hofer-Julian · 2025-07-01T07:42:52Z

crates/pixi_build_types/src/procedures/conda_outputs.rs

+    /// subdir) or for the current architecture if the target is `noarch`.
+    ///
+    /// For C++ packages these would be libraries to link against.
+    pub host_dependencies: Option<CondaOutputDependencies>,


Really like the way you write docs :)

crates/pixi_build_types/src/procedures/conda_outputs.rs

Co-authored-by: Hofer-Julian <[email protected]>

Hofer-Julian

I had a look at all code changes now, but didn't try it out locally yet

crates/pixi_build_types/src/procedures/conda_outputs.rs

crates/pixi_command_dispatcher/src/build/work_dir_key.rs

crates/pixi_command_dispatcher/src/command_dispatcher_processor/build_backend_metadata.rs

crates/pixi_command_dispatcher/src/solve_pixi/source_metadata_collector.rs

…r/build_backend_metadata.rs Co-authored-by: Hofer-Julian <[email protected]>

nichmor · 2025-07-02T19:31:22Z

First observations while testing:

context of testing:
pixi-build-backends: I'm using this branch https://github.com/prefix-dev/pixi-build-backends/pull/220/files

On pixi side, I'm running ``PIXI_BUILD_BACKEND_OVERRIDE_ALL=1 cargo run -- install`
for examples/pixi-build/cpp-sdl.

The first time it builds fine. Then if I modify:

pixi.toml and add boltons in host-dependenices, it doesnt rebuild
main.cc it doesnt rebuild

nichmor · 2025-07-02T19:34:21Z

crates/pixi_command_dispatcher/src/build/build_environment.rs


+impl BuildEnvironment {
+    /// Constructs a new build environment where the host environment is the same as the build environment.
+    pub fn to_build_from_build(&self) -> Self {


maybe rename from_host_env?

nichmor · 2025-07-02T19:36:16Z

crates/pixi_command_dispatcher/src/instantiate_tool_env/mod.rs


        // Determine the cache key for the environment.
        let cache_key = self.cache_key();
-        let name = self.requirement.0.as_source().to_string();


nichmor · 2025-07-02T19:39:18Z

crates/pixi_build_types/src/procedures/conda_outputs.rs

+#[serde(rename_all = "camelCase")]
+pub struct CondaOutputMetadata {
+    /// The identifier of the output.
+    pub identifier: CondaOutputIdentifier,


maybe we can rename it to metadata ? I would expect identifier to be an id or something, but here it is more like a subset of some metadata kind

yeah better!

nichmor · 2025-07-02T19:41:59Z

crates/pixi_build_types/src/procedures/conda_metadata.rs

@@ -1,3 +1,5 @@
+//! This API was introduced in Pixi Build API version 0.


this will be deprecated in a way that we will not use conda-getmetadata but conda-outputs?

nichmor · 2025-07-02T19:42:55Z

crates/pixi_command_dispatcher/src/build/source_metadata_cache.rs


        // Try to acquire a lock on the cache file.
-        let cache_file_path = cache_dir.join(input.hash_key()).with_extension("json");
+        let cache_file_path = cache_dir.join("metadata.json");


why it is metadata.json? I can't find a place where this file is created/written

It will be created in something like .pixi/build/meta-v0/<hash>/metadata.json

baszalmstra · 2025-07-03T08:50:09Z

The first time it builds fine. Then if I modify:

This PR only modifies locking so installing with the new backends does not work yet. Thats also likely why you dont see a change. Changes to the host section should cause a relock though. So if you add boltons to the host-dependencies then the lock-file should be out of date but it doesnt necesarlly cause a rebuild.

baszalmstra · 2025-07-10T14:35:01Z

Continued in #4118

baszalmstra added 14 commits June 24, 2025 09:24

feat: add type for pixi build api

ba69812

wip

55ba704

wip: split fetching backend metadata from source metadata

cad9fed

feat: extract run dependencies from outputs

901295a

wip

e0813e1

feat: finish

e0882c3

simplify run export conversion

c0a5190

wip

596d31f

fix: small fixes

fac9102

fix: source anchor

c45d91e

feat: read capatbilities from backend

4a11e8e

remove cache for now

23477da

more small fixes

053f020

fix: revert platform removals

4076790

baszalmstra commented Jun 30, 2025

View reviewed changes

baszalmstra added 4 commits June 30, 2025 15:03

Merge remote-tracking branch 'upstream/main' into feat/split-protocol

1b7d612

fix build error

72dd129

fmt

c6da81f

update snapshots

24d09ff

Hofer-Julian reviewed Jun 30, 2025

View reviewed changes

crates/pixi_build_types/src/procedures/conda_outputs.rs Show resolved Hide resolved

crates/pixi_build_types/src/procedures/conda_outputs.rs Outdated Show resolved Hide resolved

small fixes

3c1300d

Hofer-Julian reviewed Jul 1, 2025

View reviewed changes

baszalmstra and others added 2 commits July 1, 2025 10:05

Apply suggestions from code review

7cdfb7a

Co-authored-by: Hofer-Julian <[email protected]>

Adapt comments

f7f48ab

Hofer-Julian reviewed Jul 1, 2025

View reviewed changes

baszalmstra and others added 3 commits July 1, 2025 15:59

Update crates/pixi_command_dispatcher/src/command_dispatcher_processo…

cb38383

…r/build_backend_metadata.rs Co-authored-by: Hofer-Julian <[email protected]>

Merge remote-tracking branch 'upstream/main' into feat/split-protocol

151ff42

Merge remote-tracking branch 'upstream/main' into feat/split-protocol

7f3d7ab

nichmor reviewed Jul 2, 2025

View reviewed changes

baszalmstra added 2 commits July 3, 2025 10:59

fix: rename identifier to metadata

380990b

Merge remote-tracking branch 'upstream/main' into feat/split-protocol

4d2405b

This was referenced Jul 3, 2025

Add test for recursive build deps prefix-dev/pixi-build-testsuite#9

Merged

Support pixi private channel prefix-dev/pixi-build-backends#216

Closed

fmt

b941579

baszalmstra mentioned this pull request Jul 10, 2025

feat: add conda/outputs and conda/build_v2 backend protocol #4118

Merged

baszalmstra closed this Jul 10, 2025

		@@ -0,0 +1,211 @@
		//! Describes the `conda/outputs` request and its parameters.

		@@ -1,3 +1,5 @@
		//! This API was introduced in Pixi Build API version 0.

Conversation

baszalmstra commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Hofer-Julian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hofer-Julian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nichmor commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

baszalmstra commented Jul 3, 2025

Uh oh!

baszalmstra commented Jul 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

baszalmstra commented Jun 30, 2025 •

edited

Loading