introduce rapids-generate-pip-constraints

[jameslamb]

Contributes to rapidsai/build-planning#81

rapidsai/shared-workflows#228 introduced a new environment variable for wheel-testing CI jobs, RAPIDS_DEPENDENCIES, which takes the following values:

• oldest = pin to oldest-supported versions
• latest = do not constraint versions

To take advantage of that in CI, PRs like rapidsai/cudf#16570 have been rolling out. They're all taking a similar approach, copying around the same snippet of code that generates a pip constraints.txt file using rapids-dependency-file-generator.

Based on the suggestion from rapidsai/cudf#16570 (comment), this proposes wrapping that behavior in a tool.

Notes for Reviewers

Benefits of this change

Simplifies and standardizes CI scripts.

Reduces the effort to roll out other types of testing with alternative dependency sets in the future.

How I tested this

With latest cudf (including the changes from rapidsai/cudf#16570) checked out, ran the following from the root of that repo:

PATH="${HOME}/repos/gha-tools/tools:${PATH}" \
RAPIDS_CUDA_VERSION="12.5.1" \
RAPIDS_DEPENDENCIES=latest \
RAPIDS_PY_VERSION="3.11" \
  ${GHA_TOOLS}/rapids-generate-pip-constraints \
      test_python \
      ./beep-boop.txt
      
cat ./beep-boop.txt
# (empty)

PATH="${HOME}/repos/gha-tools/tools:${PATH}" \
RAPIDS_CUDA_VERSION="12.5.1" \
RAPIDS_DEPENDENCIES=oldest \
RAPIDS_PY_VERSION="3.11" \
  ${GHA_TOOLS}/rapids-generate-pip-constraints \
      test_python \
      ./beep-boop.txt

# # This file is generated by `rapids-dependency-file-generator`.
# # To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`.
# cramjam
# cupy-cuda11x==12.0.0
# cupy-cuda12x==12.0.0
# cupy==12.0.0
# dask-cuda==24.10.*,>=0.0.0a0
# ...

PATH="${HOME}/repos/gha-tools/tools:${PATH}" \
RAPIDS_CUDA_VERSION="12.5.1" \
RAPIDS_DEPENDENCIES=oldset \
RAPIDS_PY_VERSION="3.11" \
  ${GHA_TOOLS}/rapids-generate-pip-constraints \
      test_python \
      ./beep-boop.txt
# [rapids-generate-pip-constraints] Got unrecognized value for RAPIDS_DEPENDENCIES: 'oldset'. Expected one of ('latest', 'oldest').

echo $?
# 1

[seberg]

LGTM, nice. Also nice touch to add an error to the if!

[vyasr]

@jameslamb could you put up a PR updating some repo (e.g. cudf) to use this tool? Cloning this branch in the CI of that repo and updating PATH should allow you to test the new tool.

[jameslamb]

@jameslamb could you put up a PR updating some repo (e.g. cudf) to use this tool? Cloning this branch in the CI of that repo and updating PATH should allow you to test the new tool.

I put up rapidsai/cudf#17131 doing that. It looks like it's doing what we want.

wheel-tests-cudf (..., oldest-deps)

# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`.
...
cupy-cuda11x==12.0.0
cupy-cuda12x==12.0.0
...
numpy==1.23.*
pandas==2.0.*
pyarrow==14.0.0
...
Successfully installed ... cupy-cuda12x-12.0.0 ... numpy-1.23.5 ... pandas-2.0.3 ... pyarrow-14.0.0 ...

(build link)

wheel-tests-cudf (..., latest-deps)

...
Successfully installed ... cupy-cuda11x-13.3.0 ... numpy-2.0.2 ... pandas-2.2.3 ... pyarrow-16.1.0 ...

(build link)

@vyasr @seberg is that convincing? Want to merge this and then use it for the rest of rapidsai/build-planning#81?

[vyasr]

Yup this LGTM!

[jameslamb]

🎉 alright great, thanks everyone! I'll merge this and go apply it to rmm and cudf (the only two repos that have this testing yet, I think, based on my read of rapidsai/build-planning#81).