Update primary development workflow to use Pixi

[VeckoTheGecko]

Our current development installation uses a single conda environment file containing all our runtime, documentation, tutorial, testing, qaqc (typing and linting) dependencies. This environment is used for our unit tests, integration tests, and typechecking in CI. Although this workflow has served us well, and is a workflow adopted by other packages¹, migrating to environment management à la Pixi would have the following upsides.

Pixi upsides

Composable environments

Pixi docs - Multi Environment
Motivating Example#
There are multiple scenarios where multiple environments are useful.

Testing of multiple package versions, e.g. py39 and py310 or polars 0.12 and 0.13.
Smaller single tool environments, e.g. lint or docs.
Large developer environments, that combine all the smaller environments, e.g. dev.
Strict supersets of environments, e.g. prod and test-prod where test-prod is a strict superset of prod.
Multiple system requirements within one workspace, e.g. a cuda environment and a cpu environment.
And many more. (Feel free to edit this document in our GitHub and add your use case.)
This prepares pixi for use in large workspaces with multiple use-cases, multiple developers and different CI needs.

Currently we effectively have one environment (environment.yml). We don't have much control over the versions of dependencies installed in this environment, with it just defaulting to the latest versions (we don't test against minimum supported versions of other core packages (xarray, zarr, uxarray). This means that our lower pinning in our environment.yml file as well as our feedstock is likely out of date).

streamlines task running

• Want to see the docs? That's pixi run docs (which uses an environment with runtime and docs dependencies)
• Want to test the tutorials? That's pixi run test-tutorials (which uses an environment with runtime and tutorial dependencies)
• Want to run unit tests? That's pixi run test-unit (which uses an environment with runtime and test-unit dependencies)
• Want to run unit tests but with latest packages? That's pixi run test-unit-latest (which uses an environment with `
• Want to run all the tests? That's pixi run test-all (which runs the other testing tasks with their corresponding environments)
• Want to see all the available tasks? That's pixi run

We already have some support for this in Parcels via these task definitions. They're just all using the same environment at the moment.

simplified CI

This also:

• simplifies CI to be install pixi, pixi run ...
• allows us to easily test things from CI locally
• allows us to remove the .github/actions/install-parcels/action.yml action

This will probably also improve our CI performance since the environments installed will be done so faster (since pixi has fast installs), and leaner.

Sets the foundations for supporting optional dependencies

Making uxarray an optional dependency would be a "nice have". Not everyone wants to run on unstructured grids, so installing uxarray (and the whole dependency tree pulled in by it, which as evident in #2202 includes geopandas, geoviews, pyogrio...) creates a more bloated than necessary environment and might cause conflicts with users other dependencies.

Supporting uxarray as an optional dependency would require us to have different environments in testing, as well as some added robustness/complexity to our test suite². This issue helps with the first half of that equation.

I don't think making uxarray an optional dep is a high priority or something we should dive into. If we do add support for other grids though, this goes from a "nice have" to a "we should really have this".

Additional considerations

We won't get rid of the conda environment.yml file - so developers who want to continue using conda can still do so (for personal preference, or out of necessity due to e.g., unavailability on Lorenz/their institute HPC). We will update the contributing guidelines accordingly with pixi instructions.

---

Pixi doesn't have great support for dynamic versioning in source dependencies³ (prefix-dev/pixi#2923). We can either:

• Install git as a runtime dependency (I think let's just go for this)
• Remove dynamic versioning in Parcels (means during release we'll need to update our pyproject.toml with the new version number, and after release we will add something like 3.2.1dev0). An extra step to our release workflow, and also means that if people doing a development install of

---

Pixi build (what we will use for installing from source dependencies) is in "preview" which means that they might ship backwards incompatible API changes. I don't think this is a concern (we just pin the pixi version in CI - and they aren't really making these breaking changes anyway. See "Additional reading" in #2202 (comment) )

Footnotes

1. With caveats. Xarray, for example, mainly leverages Conda but has a whole folder dedicated to different environment files specifying different environments they want to test with. all-but-dask.yml bare-minimum.yml environment-benchmark.yml environment.yml
all-but-numba.yml doc.yml environment-windows-3.14.yml min-all-deps.yml bare-min-and-scipy.yml environment-3.14.yml environment-windows.yml. It seems like a bit of a maintenance headache. ↩

2. Something like the has_* combined with pytest.skipif pattern that xarray has in their test suite (code). ↩

3. Source dependencies = stuff that is installed from the source files (pip install .). In this case - Parcels ↩

[VeckoTheGecko]

As part of scoping for writing this issue, I made and uploaded the pixi-build-from-source branch (mainly exploring dynamic versioning). I am going to park it for now since this is comparatively a low priority (feel free anyone to build off this!). The main changes left here to complete this issue are:

• split the base environment into different "features" defined in the pixi config
• update the tasks to compose these features together
• update CI
• update contrib docs

[erikvansebille]

Thanks for this write-up @VeckoTheGecko; very useful!

[VeckoTheGecko]

e.g., unavailability on Lorenz/their institute HPC

Actually, this isn't an issue since pixi installs on a user level (no need for sudo/admin). https://pixi.sh/dev/installation/

[VeckoTheGecko]

The readthedocs environment can also use Pixi: https://docs.readthedocs.com/platform/stable/build-customization.html#install-dependencies-with-pixi

[VeckoTheGecko]

This is how we can update Binder to use the pixi environment (no native support, but I think this is good) jupyterhub/repo2docker#1339 (comment)