Support CUDA 12.2 by jameslamb · Pull Request #672 · rapidsai/cucim

jameslamb · 2024-01-11T17:41:56Z

Description

switches to CUDA 12.2.2 for building conda packages and wheels
adds new tests running against CUDA 12.2.2
removes some unnecessary cuda-version={major}.{minor} stuff in dependencies.yaml that was missed in refactor CUDA versions in dependencies.yaml #671

Notes for Reviewers

This is part of ongoing work to build and test packages against CUDA 12.2.2 across all of RAPIDS.

For more details see:

Planning a second round of PRs to revert these references back to a proper branch-24.{nn} release branch of shared-workflows once rapidsai/shared-workflows#166 is merged.

(created with rapids-reviser)

…-cuda-12.2

jakirkham · 2024-01-23T22:54:09Z

The error seen here doesn't appear to be CUDA 12.2 specific

Reproduced here: #675 (comment)

Discussing offline on how to resolve

jakirkham · 2024-01-24T20:19:36Z

Updating branch to pull in recent CI fixes ( #680 )

Maybe that helps clear things up

jakirkham · 2024-01-24T21:54:30Z

The good news is CUDA 12.2 passes! 🎉

The bad news is it looks like the CUDA 11.8 Conda test is running into a bunch of test failures. Unfortunately the job dies around 12% of the way through the test suite. So we don't learn any more about what happened

Noticing that there are some CUDA 12 packages getting installed in the CUDA 11.8 build on CI. Looking at the PR, notice we are making some changes to the CUDA 11.8 environment. Maybe this is related?

Edit: Adding snippet of CTK packages below

Details

  + libcufile         1.8.1.2  0                                nvidia                  1MB
  + libnvjpeg       12.3.0.81  0                                nvidia                  3MB
  + libcublas        12.3.4.1  0                                nvidia                368MB
  + libcufft        11.0.12.1  0                                nvidia                 87MB
  + libcurand      10.3.4.107  0                                nvidia                 54MB
  + libcusolver    11.5.4.101  0                                nvidia                117MB
  + libcusparse    12.2.0.103  0                                nvidia                179MB

…-cuda-12.2

jakirkham · 2024-01-25T19:01:14Z

ci/test_python.sh

 rapids-mamba-retry install \
  --channel "${CPP_CHANNEL}" \
  --channel "${PYTHON_CHANNEL}" \
+  "cuda-version=${RAPIDS_CUDA_VERSION%.*}" \
  "libcucim=${RAPIDS_VERSION_NUMBER}" \
  "cucim=${RAPIDS_VERSION_NUMBER}"


After discussion offline, we determine the CUDA 11.8 build was failing as the packages were being upgraded in this step to CUDA 12.3, which was unexpected

To try and fix this, have pinned cuda-version while installing libcucim & cucim. It appears that resolves the upgrade issue and allows the tests to pass

That said, we didn't expect to need a cuda-version pinning here. That may deserve some additional investigation on its own (with possible follow up here and in other RAPIDS projects)

Can this be the root cause of what we see here? conda-forge/cupy-feedstock#247 (comment)

With cuda-version added to cupy in PR ( conda-forge/cupy-feedstock#249 ), think we can now try dropping cuda-version

Suggested change

rapids-mamba-retry install \

--channel "${CPP_CHANNEL}" \

--channel "${PYTHON_CHANNEL}" \

"cuda-version=${RAPIDS_CUDA_VERSION%.*}" \

"libcucim=${RAPIDS_VERSION_NUMBER}" \

"cucim=${RAPIDS_VERSION_NUMBER}"

rapids-mamba-retry install \

--channel "${CPP_CHANNEL}" \

--channel "${PYTHON_CHANNEL}" \

"libcucim=${RAPIDS_VERSION_NUMBER}" \

"cucim=${RAPIDS_VERSION_NUMBER}"

I agree! Thanks @jakirkham

For posterity, would note that when we saw the issue previously (before adding the cuda-version workaround above), we do see cuda-version=11.8 in the specs from the environment update on CI

Transaction Prefix: /opt/conda/envs/test Updating specs: - gputil[version='>=1.4.0'] - cuda-version=11.8 - imagecodecs[version='>=2021.6.8'] - matplotlib-base - openslide-python[version='>=1.3.0'] - pip - pooch[version='>=1.6.0'] - psutil[version='>=5.8.0'] - pytest-cov[version='>=2.12.1'] - pytest-lazy-fixture[version='>=0.6.3'] - pytest-xdist - pytest[version='>=6.2.4'] - python=3.10 - tifffile[version='>=2022.7.28']

IOW the solver recognizes we've explicitly requested cuda-version with a specific version constraint

Despite this the solver later ignores this constraint and updates cuda-version anyways later in the same CI log:

- cuda-version 11.8 h70ddcb2_2 conda-forge Cached + cuda-version 12.3 h32bc705_2 conda-forge 21kB

It looks like we still have this issue. However it is now with CUDA 12.0. Here is a relevant snippet below (also when cupy is installed with the PR build of cucim) taken from CI:

- cuda-version 12.0 hffde075_2 conda-forge Cached + cuda-version 12.3 h32bc705_2 conda-forge 21kB

The CUDA 12 problems should be resolved by the fixes discussed here: rapidsai/build-planning#8 (comment)

dependencies.yaml

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

ci/test_python.sh

…-cuda-12.2

bdice

CI logs look fine. I will file a follow-up PR to make libcufile dependencies included on only x86_64 (this was a pre-existing problem so I don't want to put it in-scope for this PR).

bdice · 2024-02-09T22:18:21Z

/merge

jakirkham · 2024-02-09T22:36:46Z

/merge

bdice · 2024-02-09T22:40:25Z

The promised follow-up PR is here: #699

Follow-up from #672. This fixes an issue where libcufile-dev could be included in aarch64 environments (this path was never called in CI so it wasn't a huge problem). I also fixed some duplication in dependencies.yaml. The CUDA compilers (for 11 and 12) are now included in the `build` dependency list, and all CUDA libraries are included in the `cuda` dependency list. As before, the CUDA version is constrained by the `cuda_version` dependency list. This is more aligned with how cudf's dependency list is structured. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Jake Awe (https://github.com/AyodeAwe) - https://github.com/jakirkham URL: #699

Follow-up to #672 For all GitHub Actions configs, replaces uses of the `test-cuda-12.2` branch on `shared-workflows` with `branch-24.04`, now that rapidsai/shared-workflows#166 has been merged. ### Notes for Reviewers This is part of ongoing work to build and test packages against CUDA 12.2 across all of RAPIDS. For more details see: * rapidsai/build-planning#7 *(created with `rapids-reviser`)* Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Ray Douglass (https://github.com/raydouglass) URL: #702

jameslamb added 2 commits January 11, 2024 11:41

use CUDA 12.2 for building and testing wheels and conda packages

d74e254

update dependencies.yaml

9ba0dea

jameslamb marked this pull request as ready for review January 11, 2024 17:45

jameslamb requested a review from a team as a code owner January 11, 2024 17:45

jameslamb mentioned this pull request Jan 11, 2024

Add support for CUDA 12.2 wheels rapidsai/build-planning#7

Closed

4 tasks

jameslamb changed the title ~~add CUDA 12.2 support for conda packages and wheels~~ WIP: add CUDA 12.2 support for conda packages and wheels Jan 11, 2024

jameslamb marked this pull request as draft January 11, 2024 19:21

remove 1.20 env file

44ea0f4

jameslamb changed the title ~~WIP: add CUDA 12.2 support for conda packages and wheels~~ WIP: use CUDA 12.2 for building and testing wheels Jan 11, 2024

jameslamb changed the title ~~WIP: use CUDA 12.2 for building and testing wheels~~ WIP: add CUDA 12.2 support for conda packages and wheels Jan 11, 2024

jameslamb changed the title ~~WIP: add CUDA 12.2 support for conda packages and wheels~~ WIP: (DO NOT MERGE) add CUDA 12.2 support for conda packages and wheels Jan 11, 2024

jameslamb changed the title ~~WIP: (DO NOT MERGE) add CUDA 12.2 support for conda packages and wheels~~ (DO NOT MERGE) add CUDA 12.2 support for conda packages and wheels Jan 11, 2024

jameslamb marked this pull request as ready for review January 11, 2024 22:51

jakirkham added feature request New feature or request non-breaking Introduces a non-breaking change do not merge labels Jan 13, 2024

jakirkham mentioned this pull request Jan 18, 2024

[NO MRG] Test CI #675

Closed

jameslamb changed the base branch from branch-24.02 to branch-24.04 January 22, 2024 15:38

jameslamb added 5 commits January 22, 2024 10:04

Merge branch 'branch-24.04' into test-cuda-12.2

b311a8b

Merge branch 'branch-24.04' into test-cuda-12.2

b03f694

Merge branch 'branch-24.04' into test-cuda-12.2

ffb0c47

Merge branch 'branch-24.04' into test-cuda-12.2

a856724

Merge branch 'test-cuda-12.2' of github.com:jameslamb/cucim into test…

7176701

…-cuda-12.2

Merge branch 'branch-24.04' into test-cuda-12.2

ccdcae9

jameslamb and others added 3 commits January 24, 2024 16:32

try adding cuda_version to py_build

d4cab19

Merge branch 'test-cuda-12.2' of github.com:jameslamb/cucim into test…

03e2253

…-cuda-12.2

Pin cuda-version during *cucim install

651adce

jakirkham reviewed Jan 25, 2024

View reviewed changes

jameslamb changed the title ~~(DO NOT MERGE) add CUDA 12.2 support for conda packages and wheels~~ Support CUDA 12.2 Jan 25, 2024

jameslamb mentioned this pull request Jan 26, 2024

Support CUDA 12.2 rapidsai/cuspatial#1320

Merged

leofang mentioned this pull request Jan 29, 2024

Install cupy without explicitly requesting cuda-version now defaults to CUDA 11.8 instead of CUDA 12 conda-forge/cupy-feedstock#247

Closed

1 task

jakirkham added 2 commits January 29, 2024 13:44

Merge rapidsai/branch-24.04 into jameslamb/test-cuda-12.2

38856c4

Drop cuda-version install workaround

2bed0b3

bdice reviewed Jan 29, 2024

View reviewed changes

dependencies.yaml Outdated Show resolved Hide resolved

jakirkham and others added 2 commits January 29, 2024 14:22

Remove leftover CUDA 11.8 workaround attempt

a378184

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

Run ci/release/update-version.sh 24.04.00.

e075700

jakirkham reviewed Jan 29, 2024

View reviewed changes

ci/test_python.sh Show resolved Hide resolved

jakirkham and others added 7 commits January 29, 2024 17:35

Merge rapidsai/branch-24.04 into jameslamb/test-cuda-12.2

3081630

Merge branch 'branch-24.04' into test-cuda-12.2

e703868

Ignore run-exports from CUDA 12 compiler.

b318986

Loosen run-exports from -dev libraries.

eca560e

Merge branch 'branch-24.04' into test-cuda-12.2

6e837c9

Add cuda-cudart-dev with run-exports ignored.

b56a9ac

Merge branch 'test-cuda-12.2' of github.com:jameslamb/cucim into test…

cd71cc9

…-cuda-12.2

AyodeAwe approved these changes Feb 9, 2024

View reviewed changes

bdice removed the do not merge label Feb 9, 2024

bdice approved these changes Feb 9, 2024

View reviewed changes

jakirkham approved these changes Feb 9, 2024

View reviewed changes

rapids-bot bot merged commit b0cdce1 into rapidsai:branch-24.04 Feb 9, 2024

bdice mentioned this pull request Feb 9, 2024

Refactor CUDA libraries in dependencies.yaml #699

Merged

jakirkham mentioned this pull request Feb 10, 2024

Enable libcufile on linux_aarch64 (w/CUDA 12.2+) #700

Open

jameslamb mentioned this pull request Feb 15, 2024

target branch-24.04 for GitHub Actions workflows #702

Merged

Conversation

jameslamb commented Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Notes for Reviewers

Uh oh!

jakirkham commented Jan 23, 2024

Uh oh!

jakirkham commented Jan 24, 2024

Uh oh!

jakirkham commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham Jan 25, 2024

Choose a reason for hiding this comment

Uh oh!

leofang Jan 26, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkham Jan 29, 2024

Choose a reason for hiding this comment

Uh oh!

jameslamb Jan 29, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkham Jan 29, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkham Jan 30, 2024

Choose a reason for hiding this comment

Uh oh!

bdice Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

bdice commented Feb 9, 2024

Uh oh!

jakirkham commented Feb 9, 2024

Uh oh!

bdice commented Feb 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jameslamb commented Jan 11, 2024 •

edited

Loading

jakirkham commented Jan 24, 2024 •

edited

Loading

bdice Jan 30, 2024 •

edited

Loading