Ideas for a CI overhaul

[matrss]

I think the CI setup could be simplified in a lot of ways and I would like to get some input on what you think would be desirable.

1. Removing the docker containers from the CI

Instead of using this bespoke contraption of testing-{stable,develop} docker containers, we could instead switch to a more standard setup with e.g. the micromamba setup action: https://github.com/mamba-org/setup-micromamba.

My understanding is that we have the docker containers mainly because then we don't have to reinstall the mamba environment with every test run. The micromamba action can do the same by caching the environment in GitHub's caching mechanism, which could potentially be even faster than the docker approach (because it is directly a part of the GitHub Actions offering).

If we dropped container usage from the CI this would also open up a (hopefully easy) path to testing on the macos and windows runners (those do not support the container: options, AFAIK) as well, without much difference in the actions themselves.

This change would entail removing the localbuild directory (for which I don't see a point anyway, conda packaging matters should happen in mss-feedstock) and instead having a more standard environment.yml file.

2. Drop the testing-stable.yml and testing-develop.yml files

It should be possible to declare the given on: options in testing.yml directly and drop the reusable workflow usage that is there right now.

3. Move the scheduled tests into the mss-feedstock repository

The purpose of the scheduled tests is to check that the package as it gets installed with mamba at that moment still works. AFAICT the scheduled tests in this repository do not actually do this properly, e.g. they always use the test suite from develop instead of the released package.

I think this workflow should instead be part of the mss-feedstock repository and it should open issues there, since it is a packaging matter if the published package in conda-forge suddenly breaks.

We could still have a scheduled test in this repository though, but just to prime the environment cache described in 1. every time that it expires. This would, again, just be another on: clause in testing.yml.

4. Unify testing.yml and testing_gsoc.yml

Those two workflows have a lot in common and I think they don't have to be separate files.

---

@ReimarBauer I would like to get some input on this before I start to change anything in this regard.

[ReimarBauer]

4. have to be resolved before we have the next gsoc repository. The current solution can be seen as workaround

[ReimarBauer]

3. stable is based on the stable container e.g. https://github.com/Open-MSS/MSS/actions/runs/7375226750/job/20066530341#step:2:17

The package list gets reinstalled e.g.
https://github.com/Open-MSS/MSS/actions/runs/7375226750/job/20066530341#step:7:23

anyway moving the workflow to the conda-forge/mss-feedstock when we get it there persistant is a good improvement.
we maybe shoud do it in a way that we can get it also added e.g. on basemap, and some other of our important dependencies. Maybe a github action could be created and maintained.

I would prefer linked issues in both repositories. Because a fast fix in the feedstock can be a pinning. Then our test suite in open-mss still fails.

[matrss]

3. stable is based on the stable container e.g. https://github.com/Open-MSS/MSS/actions/runs/7375226750/job/20066530341#step:2:17

Yes, it uses the stable container, but it uses the code and test suite from develop. You can see this here: https://github.com/Open-MSS/MSS/actions/runs/7375226750/job/20066530341#step:4:55 or in the fact that the recent runs use pytest-qt even though that has not landed in stable yet: https://github.com/Open-MSS/MSS/actions/runs/7606946919/job/20713481948#step:9:23.

This is what I meant with "it does not properly check the package as released and installed with mamba".

[ReimarBauer]

1. the docker containers can be differently used when you want to switch between environments having always the same as on github used. We describe this also for developers and new contributors, e.g. https://github.com/Open-MSS/MSS/blob/develop/docs/development.rst#using-predefined-docker-images-instead-of-installing-all-requirements

this removes the hazzle to start by downloading and installing lots of libs. This is also a workaround currently for the new M machines.

When we create and update these containers and go a different path that gives more options.

The localbuild is a mapping to the conda-forge meta.yaml. I want to have something in the repo which enables an easy same configuration by creating the release. We can also have a requirements.txt which gives access to githubs security checks and have a script to create a meta.yaml file.

We can do only for a release a packaging in the conda-forge repo. Anything pushed there is a release when merged.

[ReimarBauer]

3. stable is based on the stable container e.g. https://github.com/Open-MSS/MSS/actions/runs/7375226750/job/20066530341#step:2:17

Yes, it uses the stable container, but it uses the code and test suite from develop. You can see this here: https://github.com/Open-MSS/MSS/actions/runs/7375226750/job/20066530341#step:4:55 or in the fact that the recent runs use pytest-qt even though that has not landed in stable yet: https://github.com/Open-MSS/MSS/actions/runs/7606946919/job/20713481948#step:9:23.

This is what I meant with "it does not properly check the package as released and installed with mamba".

hidden bug until now, after it was reimplemented.

[matrss]

1. the docker containers can be differently used when you want to switch between environments having always the same as on github used.

Using a proper lock file through conda-lock or preferably pixi would have the same effect, i.e. pixi install would always produce the same environment and the lock file would be tracked in the branch it belongs to. This would also bring the environment into this repo, making it a single source of truth, instead of depending on an external repository, i.e. docker hub.

this removes the hazzle to start by downloading and installing lots of libs.

It is the same with the docker image, its just all bundled into one opaque image there.

The localbuild is a mapping to the conda-forge meta.yaml. I want to have something in the repo which enables an easy same configuration by creating the release. We can also have a requirements.txt which gives access to githubs security checks and have a script to create a meta.yaml file.

Having a copy of what's in the mss-feedstock is error prone, since it could easily become out-of-sync if changes aren't carefully applied on both sides (this is potentially not just limited to pinning). Making this a pixi project would solve having a list of dependencies here and also give us a lock file for reliably using the same environment. That is everything that is required in the MSS repo itself, in my opinion.

Checking that the package can be build with conda-forge should be done with the actual conda-forge build recipe in my opinion, i.e. in the mss-feedstock repo. This is pretty easy to do, the only diff required to use the latest develop sources in mss-feedstock is this:

diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index 5520ba8..c1509d0 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -1,4 +1,4 @@
-{% set version = "8.3.2" %}
+{% set version = "develop" %}
 
 package:
   name: mss
@@ -6,7 +6,6 @@ package:
 
 source:
   url: https://github.com/Open-MSS/MSS/archive/{{ version }}.tar.gz
-  sha256: 0ebda4bb290b32faa7f9c79b75eafaaa75041e23202bbaf429259cf115e111c8
   patches:
     # start without environment by a application menu needs a PROJ_LIB env too
     - msui_set_proj_lib_data_dir.patch

This could be done in a long-running PR to mss-feedstock, where changed dependencies can be added or removed until the next release of MSS is cut, at which point this PR just needs to be changed to point at the new release and could be merged.

We can do only for a release a packaging in the conda-forge repo. Anything pushed there is a release when merged.

As described above, we wouldn't have to merge anything there until a release is made.

Ultimately I just think that development and packaging matters should be more separated and e.g. there should be a way to get a development environment without messing with installing a released mamba packages dependencies and then checking what differences with another list of dependencies exist. Ideally a pixi install could be everything that's needed, I we want to go this route. This would also enable more possibilities (e.g. windows, macos runners) in the CI since it would make the docker images completely redundant.

[ReimarBauer]

I think we currently also discussing a rolling release versus stable/develop. The current approach is also from the issue tracker simple. Everything broken in released stable is a bug and gets into a patch level new version. Small enhancenments can go into a minor release. Everthing new or API changes has to be done in develop.

We have to include semantic versioning in our considerations.

[matrss]

I don't think anything I've proposed conflicts with semantic versioning and backports of bug fixes, or the notion of stable and develop branches.

[ReimarBauer]

@matrss when from this discussion something is left over, make a new issue. That keeps work done and ideas matched.