feat: pypi-options.dependency-overrides

[HernandoR]

add dependency-overrides for overriding.

only updated manifest for now

related #3917 #3890

[HernandoR]

I'm a little bit concerned about the union strategy for pypi-options.dependency-overrides.

For now, I followed a strategy like pypi-dependency: the requirements feature will override the default.
i.e. if a feature asks for numpy=">=2.0.0" while default numpy=">=2.1.0", it will union to numpy=">=2.0.0".

The order of union logic (b overwrite a) is referred to

pixi/crates/pixi_manifest/src/feature.rs

Lines 305 to 326 in bb47111

	pub fn pypi_dependencies(
	&self,
	platform: Option<Platform>,
	) -> Option<Cow<'_, IndexMap<PypiPackageName, PixiPypiSpec>>> {
	self.targets
	.resolve(platform)
	// Get the targets in reverse order, from least specific to most specific.
	// This is required because the extend function will overwrite existing keys.
	.rev()
	.filter_map(|t| t.pypi_dependencies.as_ref())
	.filter(|deps| !deps.is_empty())
	.fold(None, |acc, deps| match acc {
	None => Some(Cow::Borrowed(deps)),
	Some(mut acc) => {
	acc.to_mut().extend(
	deps.into_iter()
	.map(|(name, spec)| (name.clone(), spec.clone())),
	);
	Some(acc)
	}
	})
	}

where, default comes last.

But I'm not quite sure the order of features in this snippet

pixi/crates/pixi_manifest/src/features_ext.rs

Lines 198 to 218 in bb47111

	fn pypi_options(&self) -> PypiOptions {
	// Collect all the pypi-options from the features in one set,
	// deduplicate them and sort them on feature index, default feature comes last.
	let pypi_options: Vec<_> = self
	.features()
	.filter_map(|feature| {
	if feature.pypi_options().is_none() {
	self.workspace_manifest().workspace.pypi_options.as_ref()
	} else {
	feature.pypi_options()
	}
	})
	.collect();
	// Merge all the pypi options into one.
	pypi_options
	.into_iter()
	.fold(PypiOptions::default(), |acc, opts| {
	acc.union(opts)
	.expect("merging of pypi-options should already have been checked")
	})

[HernandoR]

We need some tests on this feature. @tdejager can you assign someone?

[tdejager]

@nichmor could you have a look seeing that I'm on holiday? Thanks! 🙏

[tdejager]

Note you also need to add the specification to the full.toml file. Not sure about the name, on my phone currently. But contains all the available manifest options.

[tdejager]

Yeah so the confusing thing for PyPI-options is that even when using no-default one can have workspace PyPI dependencies that are always added. And I think, but maybe @ruben-arts can verify is that we wanted the last defined feature to take precedence. However, I remember us explicitly going back and forth on this, and maybe deviating from how dependencies function. So I could very well be wrong. But I think how it is defined in code for PyPI-options is the correct way, so I'd suggest following that for now.

[HernandoR]

I rely heavily on this feature. Plz let me know when and how we can push this forward.

[nichmor]

I rely heavily on this feature. Plz let me know when and how we can push this forward.

assigning to me. I will review it tomorrow and let you know how we can continue with it.

[nichmor]

I think we also need to write an integration test here. You can take a look at tests/integration_python/test_pypi_options.py and add a similar one for overrides. You will run pixi install and then you can assert that the override version is indeed present in the lockfile.

let me know if you will have questions

[HernandoR]

@nichmor. I added an integration test. But I don't know how to run it without uninstalling my existing installed pixi

[tdejager]

There are pixi tasks that use the compiled version :) Check the pixi.toml for the list of them :)

[HernandoR]

@tdejager i updated the test case and passed test-integration-fast, but i don't know how to find the solved lock file.

[HernandoR]

@nichmor I have added integration test, and verified that the generated lock file holds valid version.

[nichmor]

@nichmor I have added integration test, and verified that the generated lock file holds valid version.

thanks for the update! I will take a look today.
meanwhile, could you please solve the merge conflicts? so we could run all the tests

[tdejager]

I merged main and added extra slow tests, these test a lot of PyPI integration stuff :)

[nichmor]

one small one and we are ready to go!

[nichmor]

Looks great! Thanks for the work!

I think if you fix the tests we can merge it

[HernandoR]

@nichmor I can't reproduce this CI issue locally. It may be a temporary network problem. Could you please re-trigger the CI?

[HernandoR]

I really got no clue of the relation between the failed CI test and changes in this PR.

[nichmor]

It seems that it is related with the timeout - I will take a look tomorrow and will take care of merging it

[HernandoR]

@nichmor Shall I expect this feature been available in next release?

[tdejager]

Okay I tested the entire suite locally, and it all seems okay, lets' wait for this final batch of CI to run and if it has the same flaky failure we should be able to merge it.

[tdejager]

@HernandoR merging now, thanks for the contribution, this feature really is quite important to a lot of people. So it's super nice of you to have taken the time to implement this. Thanks @nichmor for pushing this through the review rounds.