Skip to content

CI Add a nightly job that uses scikit-learn's nightlies #7621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 9 commits into from
Jan 15, 2026
Merged

CI Add a nightly job that uses scikit-learn's nightlies #7621

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6712290
Add a nightly CI job that uses scikit-learn's nightlies
betatim Dec 17, 2025
187fd31
Merge remote-tracking branch 'upstream/main' into nightly-nightlies
betatim Dec 17, 2025
0894fe3
Address review comments
betatim Jan 6, 2026
ed93f54
Merge remote-tracking branch 'upstream/main' into nightly-nightlies
betatim Jan 6, 2026
6a9d619
Rename CI script
betatim Jan 7, 2026
5b745b3
Merge branch 'main' into nightly-nightlies
betatim Jan 13, 2026
9497623
Remove sccache token from CI job
betatim Jan 13, 2026
05bbbfd
Merge branch 'main' into nightly-nightlies
betatim Jan 14, 2026
71eddd1
Merge remote-tracking branch 'origin/main' into nightly-nightlies
csadorf Jan 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,17 @@ jobs:
sha: ${{ inputs.sha }}
script: ci/test_wheel.sh
sccache-dist-token-secret-name: GIST_REPO_READ_ORG_GITHUB_TOKEN
wheel-tests-cuml-nightly-dependencies-versions:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@main
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
node_type: "gpu-l4-latest-1"
script: ci/test_wheel_nightly_versions.sh
continue-on-error: true
wheel-tests-cuml-dask:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@main
Expand Down
50 changes: 50 additions & 0 deletions ci/test_wheel_nightly_dependencies_versions.sh
View file
Open in desktop
Comment thread
betatim marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
# SPDX-License-Identifier: Apache-2.0

# Test cuml with the nightly versions of its dependencies.

set -euo pipefail

source rapids-init-pip

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
CUML_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github python)
LIBCUML_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcuml_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp)
RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}
mkdir -p "${RAPIDS_TESTS_DIR}"

# Explicitly install the packages (and their dependencies) that we want to have from the nightly index.
rapids-pip-retry install \
Copy link
Copy Markdown
Member Author

@betatim betatim Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could include the --extra-index-url options in dependencies.yaml but we want to control what is installed from the scientific-python-nightly index. If we add the --extra-index-url to the install command that installs all other dependencies would mean that we get all dependencies that exist on the nightly index from there. As of today we don't want to install all the dependencies that are available from that index.

Copy link
Copy Markdown
Member

@jameslamb jameslamb Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is quite right. pip ignores pre-releases by default.

Using --isolated ignores any external configuration (to avoid "it works differently on my machine" issues). First, installing just scikit-learn (no constraint):

$ NIGHTLY_INDEX="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
$ pip install --isolated --dry-run --no-cache-dir --extra-index-url=${NIGHTLY_INDEX} 'scikit-learn'
Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
...
Would install joblib-1.5.3 numpy-2.3.5 scikit-learn-1.8.0 scipy-1.16.3 threadpoolctl-3.6.0

If you use a pre-release specifier in the requirement, then pip will consider pre-releases only for that package. For example, the following pulls in a nightly of scikit-learn but ignores nightlies for everything else:

$ pip install --isolated --dry-run --no-cache-dir --extra-index-url=${NIGHTLY_INDEX} 'scikit-learn>=1.8.0dev0'
Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
...
Would install joblib-1.5.3 numpy-2.3.5 scikit-learn-1.9.dev0 scipy-1.16.3 threadpoolctl-3.6.0

side note: that "pre-release specifiers tell pip nightlies are OK" behavior led to some recent work to remove those specifiers where they'd been put in places where we did NOT want nightlies:

The only way you'd get pre-releases of everything that has them on any of the considered indices is if you pass --pre or set the equivalent environment variable.

$ pip install --pre --isolated --dry-run --no-cache-dir --extra-index-url=${NIGHTLY_INDEX} 'scikit-learn'
Looking in indexes: https://pypi.org/simple, https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
..
Would install joblib-1.5.3 numpy-2.5.0.dev0 scikit-learn-1.9.dev0 scipy-1.18.0.dev0 threadpoolctl-3.6.0

I really think you'll want something like:

  • put pre-release specifiers like >=1.8.0dev0 on all the dependencies where you're willing to accept nightlies
  • consolidate to a single pip install so all requirements are considered together (separate calls increase the risk of a broken environment)

Given those constraints, totally up to you if you want the --extra-index-url in dependencies.yaml or not. I think it could be helpful for it to always get selected alongside constraints spelled like sciki-learn>=1.8.0dev0, but can also see how keeping it inline in this script prevents mistakes.

Copy link
Copy Markdown
Member Author

@betatim betatim Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should rephrase the comment, because having the dependencies of scikit-learn from the anaconda nightly channel is what I was expecting to happen (what --pre does).

What I want to avoid is that one day we suddenly get nightlies for other direct cuml dependencies. This could happen because the package decides to start uploading wheels to the nightlies from one day to the next.

I also want to avoid having to update the 1.n.0dev0 to 1.n+1.0dev0 every time a new version is released. In particular because the old command will keep running without an error but not do the right thing. Once the release of 1.8.0 has happened scikit-learn>=1.8.0dev0 will no longer install the latest nightly and there won't be an error either.

Copy link
Copy Markdown
Member

@jameslamb jameslamb Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, makes sense!

Once the release of 1.8.0 has happened scikit-learn>=1.8.0dev0 will no longer install the latest nightly and there won't be an error either.

As soon as there was a 1.8.1dev0 or something (the first nightly after the 1.8.0 release), pip should pick that up because it prefers the latest version.

And if there isn't a newer version on the nightly index then picking up 1.8.0 isn't a problem, because that is the latest version (just happens to also have been a released one).

That said, if you want to always guarantee this job installs the latest version on the nightly index, then the most reliable way I know to do that with pip is to download from the nightly index and then install locally-downloaded wheels.

Like this:

mkdir -p /tmp/nightly-wheels

pip download \
  --no-deps \
  --index-url=${NIGHTLY_INDEX} \
  --prefer-binary \
  -d /tmp/nightly-wheels \
     'numpy>=2.5.0.dev0' \
     'scikit-learn>=1.9.0dev0' \
     'scipy>=1.18.0.dev0'

rapids-pip-retry install \
   "${LIBCUML_WHEELHOUSE}"/libcuml*.whl \
  "$(echo "${CUML_WHEELHOUSE}"/cuml*.whl)[test]" \
  --requirement requirements.txt \
  --constraint "${PIP_CONSTRAINT}" \
  /tmp/nightly-wheels/*.whl

Passing /tmp/nightly-wheels/*.whl to pip install as a requirement tells pip "you MUST install these specific files", which will prevent it from preferring other packages for those libraries.

Copy link
Copy Markdown
Member Author

@betatim betatim Jan 13, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be ok with merging it as it is?

I'd like to see this running so we can fix hiccups in time for 26.02. This means we'd need to merge within the next days so that I can observe what happens and have time left to fix it before Friday. We can fine tune pip when we run into problems with it or see potential for optimisation.

Copy link
Copy Markdown
Member

@jameslamb jameslamb Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left 1 small comment about the CI config that should be resolved, but otherwise I won't block merging this.

If you're comfortable with the risks introduced by separating this into multiple pip install calls and happy generally that this is testing what you want to test, it's fine to merge.

Looks like you already have a ci-codeowners approval, so this would just need @dantegd to resolve his blocking review.

Copy link
Copy Markdown
Contributor

@csadorf csadorf Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should merge this as-is and then explore whether to adopt some of the proposed improvements to make this more robust in a follow-up.

--pre \
--extra-index-url=https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
scikit-learn

rapids-dependency-file-generator \
--output requirements \
--file-key test_python \
--matrix "dependencies=latest" | tee requirements.txt

# notes:
#
# * echo to expand wildcard before adding `[test,experimental]` requires for pip
# * need to provide --constraint="${PIP_CONSTRAINT}" because that environment variable is
# ignored if any other --constraint are passed via the CLI
#
rapids-pip-retry install \
"${LIBCUML_WHEELHOUSE}"/libcuml*.whl \
"$(echo "${CUML_WHEELHOUSE}"/cuml*.whl)[test]" \
--requirement requirements.txt \
--constraint "${PIP_CONSTRAINT}"

EXITCODE=0
trap "EXITCODE=1" ERR
set +e

rapids-logger "pytest cuml single GPU"
timeout 1h ./ci/run_cuml_singlegpu_pytests.sh \
--numprocesses=8 \
--dist=worksteal

rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}