Use conda to build python packages during GPU tests #897

jjacobelli · 2022-04-21T08:08:23Z

This PR convert the from sources build we are doing in GPU test job to a conda build. This is done for the following reasons:

This is required step to improve the Ops CI/CD setup to a more convenient pipeline
This is required to start using conda compilers and mamba to build RAPIDS packages
This prevent us from manually managing and installing the dependencies in GPU job
This ensure the packages can be installed
This ensure the tests are running and working against the package content and not the build results. Currently the Python packages are not tested.

This may increase the global pipeline time, but the usage of mamba should resolve this as mamba is faster than conda to build packages

pentschev · 2022-04-21T08:46:48Z

Is there some advantage to doing this, or is it just to test conda packaging as well?

jjacobelli · 2022-04-21T08:49:46Z

Is there some advantage to doing this, or is it just to test conda packaging as well?

That's one of the advantages. We are also doing this to remove the usage of the rapids-build-env package and to start using conda compilers with mamba to build packages. I will update the PR body with all of this before merging it

codecov-commenter · 2022-04-25T10:20:59Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.06@c4d542b). Click here to learn what that means.
The diff coverage is n/a.

@@              Coverage Diff               @@
##             branch-22.06    #897   +/-   ##
==============================================
  Coverage                ?   0.00%           
==============================================
  Files                   ?      22           
  Lines                   ?    3075           
  Branches                ?       0           
==============================================
  Hits                    ?       0           
  Misses                  ?    3075           
  Partials                ?       0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4d542b...0d5b7b6. Read the comment docs.

pentschev · 2022-04-25T20:36:35Z

rerun tests

pentschev · 2022-04-26T08:23:28Z

The timeout from dask_cuda.tests.test_spill.test_cudf_cluster_device_spill[params0] is being dealt with in #901 .

pentschev · 2022-04-26T08:23:37Z

rerun tests

pentschev · 2022-04-26T09:11:47Z

rerun tests

pentschev · 2022-04-26T13:11:58Z

This many timeouts doesn't look normal, even the timeout that was increased in #901 was hit here. @ethyling could you add conda list --show-channels after gpuci_mamba_retry install dask-cuda so I can compare the packages being picked here vs what was being picked before?

jjacobelli · 2022-04-27T08:34:07Z

This many timeouts doesn't look normal, even the timeout that was increased in #901 was hit here. @ethyling could you add conda list --show-channels after gpuci_mamba_retry install dask-cuda so I can compare the packages being picked here vs what was being picked before?

@pentschev PR updated with this

pentschev · 2022-05-04T13:00:24Z

I have been trying to reproduce timeouts on CUDA 11.0, driver 450.80.02, just like in CI, for days but have been unsuccessful. I can only imagine at this point that there is something related to the CI machines that make it just a tiny bit slower, in which case we may try to increase timeouts. I will run tests a few more times in CI here to see how it behaves and see what are the tests that may need increased timeouts.

pentschev · 2022-05-04T13:00:29Z

rerun tests

pentschev · 2022-05-04T13:59:45Z

Tests that failed in latest run are now resolved by #905, rerunning.

pentschev · 2022-05-04T13:59:49Z

rerun tests

pentschev · 2022-05-04T14:31:08Z

rerun tests

pentschev · 2022-05-04T15:01:42Z

rerun tests

pentschev · 2022-05-04T16:32:32Z

Three successful runs in a row. At this point I would assume we had a hiccup in CI when this was running before and would suggest we move forward as is.

@ethyling is there anything more that needs to be done here or should we consider reviewing/merging this PR?

jjacobelli · 2022-05-04T17:18:07Z

Three successful runs in a row. At this point I would assume we had a hiccup in CI when this was running before and would suggest we move forward as is.

Nice, thank you @pentschev!

@ethyling is there anything more that needs to be done here or should we consider reviewing/merging this PR?

I prefer to wait that all the PRs pass for all the RAPIDS projects before merging it

ajschmidt8

Approved with a question.

ci/gpu/build.sh

jjacobelli · 2022-05-10T11:21:53Z

conda/recipes/dask-cuda/meta.yaml


 source:
-  path: ../../..
+  git_url: ../../..


This change is required to prevent this issue:

AssertionError: Can't merge/copy source into subdirectory of itself. Please create separate spaces for these things. src: /workspace dst: /workspace/.conda-bld/work

jjacobelli · 2022-05-10T13:10:38Z

rerun tests

jjacobelli · 2022-05-10T13:50:09Z

rerun tests

Signed-off-by: Jordan Jacobelli <[email protected]>

pentschev · 2022-05-10T18:31:27Z

Failing test is due to dask/distributed#6320 , which was caused by dask/distributed#5910 merge.

jjacobelli · 2022-05-12T11:06:20Z

Failing test is due to dask/distributed#6320 , which was caused by dask/distributed#5910 merge.

Do you know if this will be fixed soon?

pentschev · 2022-05-12T20:19:13Z

Sorry @ethyling , I don't know if that is going to be fixed soon. I marked the test to xfail in #908 to unblock CI for now.

pentschev · 2022-05-12T20:19:17Z

rerun tests

pentschev

LGTM, thanks for the work here @ethyling .

pentschev · 2022-05-12T21:28:45Z

Feel free to merge if this is ready to go.

jjacobelli · 2022-05-13T08:11:55Z

Thank you for your help with this PR @pentschev!

jjacobelli · 2022-05-13T08:12:02Z

@gpucibot merge

jjacobelli added DO NOT MERGE Hold off on merging; see PR for details improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Apr 21, 2022

jjacobelli self-assigned this Apr 21, 2022

github-actions bot added the gpuCI gpuCI issue label Apr 21, 2022

jjacobelli force-pushed the remove-from-sources branch from ec21c59 to c917ad7 Compare April 21, 2022 08:23

jjacobelli force-pushed the remove-from-sources branch 2 times, most recently from b6360f6 to a7e30b3 Compare April 25, 2022 09:33

jjacobelli removed the DO NOT MERGE Hold off on merging; see PR for details label Apr 25, 2022

jjacobelli marked this pull request as ready for review April 25, 2022 09:38

jjacobelli requested a review from a team as a code owner April 25, 2022 09:38

jjacobelli force-pushed the remove-from-sources branch from a7e30b3 to 09e49fc Compare April 25, 2022 10:20

jjacobelli force-pushed the remove-from-sources branch from 09e49fc to 5d1db0f Compare April 26, 2022 09:14

jjacobelli force-pushed the remove-from-sources branch 2 times, most recently from d65e32f to 0d5b7b6 Compare April 27, 2022 11:08

ajschmidt8 approved these changes May 9, 2022

View reviewed changes

ci/gpu/build.sh Show resolved Hide resolved

ajschmidt8 requested changes May 9, 2022

View reviewed changes

ci/gpu/build.sh Outdated Show resolved Hide resolved

jjacobelli force-pushed the remove-from-sources branch 2 times, most recently from 772733a to b0024db Compare May 10, 2022 11:20

github-actions bot added the conda conda issue label May 10, 2022

jjacobelli commented May 10, 2022

View reviewed changes

jjacobelli force-pushed the remove-from-sources branch from b0024db to 601f9a7 Compare May 10, 2022 14:00

Use conda to build python packages during GPU tests

80fd80a

Signed-off-by: Jordan Jacobelli <[email protected]>

jjacobelli force-pushed the remove-from-sources branch from 601f9a7 to 80fd80a Compare May 10, 2022 14:18

ajschmidt8 approved these changes May 10, 2022

View reviewed changes

pentschev approved these changes May 12, 2022

View reviewed changes

rapids-bot bot merged commit 497180d into rapidsai:branch-22.06 May 13, 2022

This was referenced May 25, 2022

Move to Dask nightlies in CI rapidsai/cuml#4757

Open

Move to Dask nightlies in CI rapidsai/ucx-py#855

Closed

Move to Dask nightlies in CI rapidsai/raft#677

Open

pentschev mentioned this pull request May 25, 2022

[REVIEW] Pin dask & distributed for release #916

Merged

Use conda to build python packages during GPU tests #897

Use conda to build python packages during GPU tests #897

Uh oh!

Conversation

jjacobelli commented Apr 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pentschev commented Apr 21, 2022

Uh oh!

jjacobelli commented Apr 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Apr 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pentschev commented Apr 25, 2022

Uh oh!

pentschev commented Apr 26, 2022

Uh oh!

pentschev commented Apr 26, 2022

Uh oh!

pentschev commented Apr 26, 2022

Uh oh!

pentschev commented Apr 26, 2022

Uh oh!

jjacobelli commented Apr 27, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

pentschev commented May 4, 2022

Uh oh!

jjacobelli commented May 4, 2022

Uh oh!

ajschmidt8 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jjacobelli May 10, 2022

Choose a reason for hiding this comment

Uh oh!

jjacobelli commented May 10, 2022

Uh oh!

jjacobelli commented May 10, 2022

Uh oh!

pentschev commented May 10, 2022

Uh oh!

jjacobelli commented May 12, 2022

Uh oh!

pentschev commented May 12, 2022

Uh oh!

pentschev commented May 12, 2022

Uh oh!

pentschev left a comment

Choose a reason for hiding this comment

Uh oh!

pentschev commented May 12, 2022

Uh oh!

jjacobelli commented May 13, 2022

Uh oh!

jjacobelli commented May 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

jjacobelli commented Apr 21, 2022 •

edited

Loading

jjacobelli commented Apr 21, 2022 •

edited

Loading

codecov-commenter commented Apr 25, 2022 •

edited

Loading