Testing against oldest python dependencies

[seberg]

E.g. for NumPy 2 it makes sense at least for some projects (cudf, cuml, cucim) to test both NumPy 2 and NumPy 1.x, however, RAPIDS currently doesn't do this.

The following PR would allow each projects to add explicit oldest dependencies to their testing workflows: rapidsai/shared-workflows#228

• Shared workflow PR
• Include dependency file generator in wheel image

For each of the projects, we need then pin the interesting oldest dependencies:

• rmm:
  • numpy==1.23, numba=0.57
  • PR
• cuDF:
  • numpy==1.23, pandas==2.0, numba=0.57, (fsspec=0.6.0)
    pyarrow==16.1.0, cupy==12.0.0
  • PR
• raft:
  • numpy=1.23, numba==0.57, (joblib==0.11)
• cuML:
  • numpy=1.23, numba==0.57, scipy==1.8, joblib==0.11
• cuGraph:
  • cupy==12.0.0, numpy=1.23, numba==0.57, networkx==3.0, (fsspec=0.6.0)
• cuCIM:
  • cupy==12.0.0, numpy=1.23, scikit-image==0.19.0, (lazy_loader==0.1)
• cuSpatial:
  • cupy==12.0.0, numpy=1.23, geopandas=0.11.0
• cuxfilter:
  • cupy==12.0.0, numpy=1.23, numba>=0.57, geopandas=0.11.0, holoviews==1.16.0, bokeh==3.1, panel==1.0
• ucxx:
  • numpy==1.23, numba=0.57 (only inherited from RMM, maybe skip)
• KvikIO:
  • cupy==12.0.0, numpy=1.23, zarr==? (no range specified)
• wholegraph:
  • probably skip dependencies seem only test dependencies not runtime.
• cuVS:
  • cupy==12.0.0, numpy==1.23

Many of these depend on rmm so could e.g. have a numba=0.57, but maybe it is sufficient to concentrate on explicitly specified dependencies only. I bracketed a few that seemed not very relevant, although it may make sense to just put them anyway.

[jameslamb]

+1, I think it'll be great to have something like this. I've seen it be very effective in other projects.

And I specifically think we'll be happy to have it when pandas 3.0 comes out: pandas-dev/pandas#57064

[jakirkham]

@bdice do you have thoughts on this proposal? 🙂

[bdice]

Yes, I support this. This is what @seberg and I discussed when I drafted rapidsai/shared-workflows#228.

[jakirkham]

The first 2 steps are done above. Have checked those boxes

Next step is rolling this out to projects (as described by the remaining checklist items)

Though please let me know if I've missed anything

[vyasr]

Is this something @seberg can continue working on for the rest of RAPIDS?

[seberg]

Sorry, was out for a while, so didn't pick it up again after the cudf start. Yes, I think I can continue working on this again very soon.

[jakirkham]

Having learned from the cuDF experience, it is worth noting that we will need a collaborator from each of the libraries to help push these PRs to completion

Setting up the matrix is doable. Debugging the various RAPIDS project specific errors that come up is something best left in the hands of a project-focused developer

[vyasr]

I agree, that sounds like the right plan. I think having a PR coming from someone build-focused can help motivate the library devs to come up with the proper dependency constraints and fix any unexpected issues, so it's good to start here then hand off.