Skip to content

Patch dask-expr var logic in dask-cudf#15347

Merged
rapids-bot[bot] merged 21 commits intorapidsai:branch-24.06from
rjzamora:fix-var
Apr 8, 2024
Merged

Patch dask-expr var logic in dask-cudf#15347
rapids-bot[bot] merged 21 commits intorapidsai:branch-24.06from
rjzamora:fix-var

Conversation

@rjzamora
Copy link
Copy Markdown
Member

@rjzamora rjzamora commented Mar 20, 2024

Description

The var logic in dask-expr relies on pandas -> numpy conversion that does not work for cudf -> cupy when null values are present. This PR copies over the custom var logic being used in dask-cudf for the legacy API.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@rjzamora rjzamora added the 2 - In Progress Currently a work in progress label Mar 20, 2024
@rjzamora rjzamora self-assigned this Mar 20, 2024
@github-actions github-actions bot added Python Affects Python cuDF API. ci labels Mar 20, 2024
@rjzamora rjzamora added bug Something isn't working non-breaking Non-breaking change labels Mar 20, 2024
@rjzamora rjzamora mentioned this pull request Mar 20, 2024
Open
28 tasks
bdice
bdice requested changes Mar 20, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add a test in dask-cudf for the functions used in the 10min notebook.

@rjzamora
Copy link
Copy Markdown
Member Author

rjzamora commented Mar 20, 2024

We need to add a test in dask-cudf for the functions used in the 10min notebook.

Agreed - This is still a draft.

bdice
bdice reviewed Mar 20, 2024
View reviewed changes
@rjzamora
Copy link
Copy Markdown
Member Author

rjzamora commented Mar 20, 2024 •
edited
Loading

Note that the custom VarCudf expression implemented in this PR will not always work without an upstream change like dask/dask#11013 (I used a "workaround" in this PR that is simple, but not very intuitive)

@rjzamora
Copy link
Copy Markdown
Member Author

rjzamora commented Mar 22, 2024

CI failures are due to a bad/old dask-expr version being pulled (dask/distributed#8574)

@rjzamora rjzamora marked this pull request as ready for review April 2, 2024 17:36
@rjzamora rjzamora requested review from a team as code owners April 2, 2024 17:36
@rjzamora
Copy link
Copy Markdown
Member Author

rjzamora commented Apr 2, 2024

The dask/dask-expr nightly-conda situation should be fixed now. So dask-related CI should start passing now.

@bdice
Copy link
Copy Markdown
Contributor

bdice commented Apr 3, 2024 •
edited
Loading

I reran the failed CI jobs (network issues). @rjzamora Is this ready for review?

bdice
bdice approved these changes Apr 3, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look fine to me.

@rjzamora
Copy link
Copy Markdown
Member Author

rjzamora commented Apr 3, 2024

Is this ready for review?

Yeah, thanks @bdice !

@rjzamora
Copy link
Copy Markdown
Member Author

rjzamora commented Apr 8, 2024

/merge

@rapids-bot rapids-bot bot merged commit 3896222 into rapidsai:branch-24.06 Apr 8, 2024
@rjzamora rjzamora deleted the fix-var branch April 8, 2024 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bdice bdice bdice approved these changes

@galipremsagar galipremsagar galipremsagar approved these changes

+1 more reviewer

@raydouglass raydouglass raydouglass approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@rjzamora rjzamora

Labels

2 - In Progress Currently a work in progress bug Something isn't working non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rjzamora @bdice @raydouglass @galipremsagar