Dependencies for Dask-expr and Rapids AI

[timmytwoteeth]

Hello,

Are you aware of any dependency conflicts between dask-expr and rapids ai that could cause substantial issues?

dask-expr 1.0.5 requires dask==2024.3.1, but you have dask 2024.1.1 which is incompatible.
dask-expr 1.0.5 requires pandas>=2, but you have pandas 1.5.3 which is incompatible.

Thank you!

[phofl]

cc @rjzamora

[rjzamora]

Thanks for raising @timmytwoteeth - Dask-expr support is a high priority for the 24.06 release of RAPIDS, but it is still a work in progress. What RAPIDS libraries are you working with?

In order to test/use dask-expr with rapids, I recommend that you work with dask.dataframe (or dask_cudf) after setting the "dataframe.query-planning" config to True. This means that dask.dataframe will be using dask-expr under the hood. You will likely run into some backend-dispatching problems if you try to use dask-expr directly with cudf.

The 24.04 release of RAPIDS has very minimal support for "dataframe.query-planning": True, and so it is technically pinned to dask 2024.1.1.

[timmytwoteeth]

Hi @rjzamora,

Thank you for the immediate response.

I am currently working largely with cuml and cudf from the rapids library. I will have to test with dask cudf setting dataframe query planning to true.

Thank you.

[rjzamora]

I am currently working largely with cuml and cudf from the rapids library

Okay, there are some known issues in cuml when query-planning is set to true, but I will be working on them very soon.

[timmytwoteeth]

I am currently working largely with cuml and cudf from the rapids library

Okay, there are some known issues in cuml when query-planning is set to true, but I will be working on them very soon.

I appreciate your help. Will be looking forward to it in the future.

Do you know whether dask-expr cuml will support binary quantization and int8?