cuML dask fixes to unblock CI

[dantegd]

PR includes commits from #6166 to run CI while that one is merged.

Dask nightlies break a few things in cuML, mainly the following two issues: #6168, #6169

Closes #6166.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[dantegd]

/ok to test

[betatim]

For the two dask related problems, do we have an upstream issue (in dask land) to track this? Even if they say "this is a feature, not a bug". I was mostly thinking that we are modifying the cuml tests here to make them pass, but presumably the tests got put here because someone thought users should be able to do this. So if we can do something simple to point users to an issue that explains why something that used to work suddenly has stopped working that would be great.

Otherwise this looks uncontroversial?!

[dantegd]

@betatim I agree we need to open them, haven't opened them yet because I don't have a repro without dask yet, but we should open them once we have some

[betatim]

                     from /home/coder/cuml/cpp/build/pip/cuda-12.5/release/_deps/rmm-src/include/rmm/mr/device/pool_memory_resource.hpp:34,
  
                     from /home/coder/cuml/cpp/build/pip/cuda-12.5/release/_deps/raft-src/cpp/include/raft/core/resource/device_memory_resource.hpp:26,
  
                     from /home/coder/cuml/cpp/build/pip/cuda-12.5/release/_deps/raft-src/cpp/include/raft/core/device_resources.hpp:32,
  
                     from /home/coder/cuml/cpp/build/pip/cuda-12.5/release/_deps/raft-src/cpp/include/raft/core/handle.hpp:19,
  
                     from /home/coder/cuml/python/cuml/build/pip/cuda-12.5/release/cuml/cluster/kmeans.cxx:1072:
    /home/coder/cuml/cpp/build/pip/cuda-12.5/release/_deps/cccl-src/thrust/thrust/cmake/../../thrust/system/cuda/config.h:122:2: error: #error The version of CUB in your include path is not compatible with this release of Thrust. CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB. Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this.
      122 | #error The version of CUB in your include path is not compatible with this release of Thrust. CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB. Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this.
          |  ^~~~~
    ninja: build stopped: subcommand failed.

This is the error from the "build in devcontainer" CI run. Any ideas?

[bdice]

This now looks good and CI should pass.

[bdice]

/merge

[bdice]

Thanks team! We got CI passing again! 🥳