Remove uses of 'pip --find-links' in CI

[jameslamb]

Description

Most RAPIDS projects have separate build jobs (which produce artifacts like wheels and conda packages) and test jobs (which install those artifacts and then run tests using the installed versions).

Some of those test jobs for wheels do something like this (pseudocode) to install those build-job artifacts:

download-from-s3 project.whl > ./dist/
pip install "project" --find-links ./dist

As we discovered in NVIDIA/raft#2348, that use of --find-links can result in some types of failures being missed in CI.

For example, if that wheel in ./dist has impossible-to-resolve dependencies, pip install will happily disregard it and backtrack to older wheels on https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/.

From "Finding Packages' in the pip docs (link)

pip looks for packages in a number of places: on PyPI ..., in the local filesystem, and in any additional repositories specified via --find-links or --index-url. There is no ordering in the locations that are searched. ...they are all checked, and the “best” match for the requirements ... is selected.

Wherever possible, use of --find-links should be avoided across RAPIDS CI jobs, in favor of asking pip to directly install the just-downloaded wheel(s) like this:

pip install ./dist/*.whl

Benefits of this work

• improves the chance of catching issues like "this wheel has unsatisfiable requirements" in CI, near the change that leads to that issue
• improves release confidence and velocity in changes to how wheels are built
  • e.g. rapids-build-backend (Update RAPIDS Python packages to use rapids-build-backend #31)
  • e.g. C++ wheels (Support dynamic linking between RAPIDS wheels #33)

Acceptance Criteria

• across RAPIDS projects' CI , there are as few --find-links or PIP_FIND_LINKS pointing at a local directory as possible

Approach

Use GitHub search to look for uses of the following across RAPIDS:

• pip install --find-links
• PIP_FIND_LINKS variable

Whenever possible, replace those by pointing pip install directly at the files to consider, e.g. like this:

# globbing is enough if not using [extras]
pip install ./dist/*

# echo the full name for use with [extras]
pip install "$(echo ./dist/*.whl)[test]"

Build jobs might require a different approach from test jobs

You'll find some uses like this one in rmm, where a wheel of some dependency is downloaded and then needed at build time.

CPP_WHEELHOUSE=$(
    RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" \
    rapids-download-wheels-from-s3 cpp /tmp/librmm_dist
)

pip wheel \
    -w dist \
    --no-deps \
    --find-links "${CPP_WHEELHOUSE}" \
    .

(rmm - ci/build_wheel.sh)

That use of --find-links is there because RAPIDS CI build jobs tend to use build isolation. Test out whether it's possible to additionally pass specific wheel files to pip wheel --constraint to force pip wheel to use them.

python -m pip wheel \
    -w dist \
    -vvv \
    --no-deps \
    --disable-pip-version-check \
    --find-links "${CPP_WHEELHOUSE}" \
    --constraint "${CPP_WHEELHOUSE}/*.whl" \
    .

Something like that might work to force pip to use that file, even in the isolated build environment.

But if not, it's fine to leave the build jobs alone, in exchange for the benefits we get from using build isolation.

Notes

Relevant discussions:

• [BUG] wheel tests do not fail when raft-dask wheel has unsatisfiable dependency requirements NVIDIA/raft#2348
• ensure raft-dask wheel tests install pylibraft wheel from the same CI run, fix wheel dependencies NVIDIA/raft#2349

### Tasks
- [x] cugraph (https://github.com/rapidsai/cugraph/pull/4509)
- [x] cugraph-gnn (https://github.com/rapidsai/cugraph-gnn/pull/7)
- [x] cugraph-pg (https://github.com/rapidsai/cugraph-pg/pull/22)
- [x] kvikio (https://github.com/rapidsai/kvikio/pull/397)
- [x] raft (https://github.com/rapidsai/raft/pull/2349)
- [x] rmm (builds: https://github.com/rapidsai/rmm/pull/1586 | tests: https://github.com/rapidsai/rmm/pull/1583)
- [x] ucxx (https://github.com/rapidsai/ucxx/pull/234)
- [x] ucx-wheels (https://github.com/rapidsai/ucx-wheels/pull/9)

[vyasr]

This is fundamentally the pip analog to #14. I moved us towards --find-links in some recent PRs because I thought it would better align our pip and conda solves, but you're right that it leads to pip installs having the same issues as the conda solve issues. It is quite tricky to come up with a good solution here though. Maybe you could do a [pip|conda] install --no-deps of the downloaded artifacts followed by an installation of the package list produced by dfg? The order here will matter, but I don't know which order will produce a cleaner result; in one case you might have to force upgrading, while in the other you'd have to prevent it.

As far as build isolation, it's worth noting that we might have to do away with that anyway in light of C++ wheels and the sccache issues that come with using this in ephemeral isolated build environments.

[jameslamb]

Thanks for that! I hadn't considered the connection between this idea and #14.

It is quite tricky to come up with a good solution here though

At a minimum, for wheel test jobs I think the pip install dist/*.whl pattern is unambiguously stricter and therefore preferable to the pip install --find-links ./dist pattern.

But yeah, the case for build jobs (at least as long as we want build isolation) is tough.

[vyasr]

Well like I said we might have to get rid of build isolation so we can figure that part out when we get done with C++ wheels.

[jameslamb]

Was experimenting with approaches for this today, and I think I found one that'll give us a strong guarantee of correctness for build dependencies even while using build isolation.

It appears that --constraint passed to pip wheel doesn't affect the creation of the build environment... but setting environment variable PIP_CONSTRAINT does.

So, given some wheel(s) downloaded like this...

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
CPP_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/librmm_dist)

... instead of this (the current pattern used in rmm, just as an example)...

python -m pip wheel \
    --no-deps \
    -w dist \
    --find-links "${CPP_WHEELHOUSE}" \
    .

... I think we could force pip to use exactly the wheel that was just downloaded like this:

echo "librmm @ file://$(echo ${CPP_WHEELHOUSE}/librmm_*.whl)" > ./constraints.txt

PIP_CONSTRAINT="${PWD}/constraints.txt" \
python -m pip wheel \
    --no-deps \
    -w dist \
    .

That worked for me in some local experiments using simple pure-Python projects. I think I could test it with rmm by publishing a librmm wheel (CI artifact only) with outright-broken runtime dependencies. If we see build of rmm wheels fail, we'll know that it tried to use exactly that broken librmm. If they succeed, then that'd probably be because pip wheel ignored the broken local wheel and fell back to a working one (the thing we're trying to avoid).

I'll try that some time soon.