Update versions/recipe logic.#109
Conversation
| "cupy-cuda11x", | ||
| ] | ||
| "pytest-rerunfailures", | ||
| ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`. |
There was a problem hiding this comment.
How did rapids-dependency-file-generator come up with this list? In particular the Dask/Distributed pins seem very arbitrary.
There was a problem hiding this comment.
These are coming from dependencies.yaml. That file is supposed to be a "single source of truth" for all dependencies and their pinnings (with the exception of meta.yaml conda recipes and CMakeLists.txt which are not implemented at this time). Today, rapids-dependency-file-generator generates pyproject.toml, developer conda environments, and can generate dependency lists on-the-fly that aren't committed to the repo but are used during various builds (e.g. devcontainers, rapids-compose).
There was a problem hiding this comment.
I just noticed that conda/environments/ucxx-cuda118_arch-x86_64.yaml is not being generated by dependencies.yaml so I'll try to adjust that.
There was a problem hiding this comment.
There are some features like channel pinnings (dask/label/dev::dask) and comments that would be lost in the developer environment. @pentschev Do you have thoughts on whether we should rework this to be auto-generated like other RAPIDS packages or would you prefer to maintain it separately?
There was a problem hiding this comment.
See what you think of commit 4fc2c97. I added a dev section with the dependencies like libtool and autoconf and dask-cuda that aren't otherwise specified by ucxx as build/run dependencies.
There was a problem hiding this comment.
These are coming from
dependencies.yaml. That file is supposed to be a "single source of truth" for all dependencies and their pinnings (with the exception of meta.yaml conda recipes and CMakeLists.txt which are not implemented at this time). Today,rapids-dependency-file-generatorgeneratespyproject.toml, developer conda environments, and can generate dependency lists on-the-fly that aren't committed to the repo but are used during various builds (e.g. devcontainers, rapids-compose).
Sorry, this was entirely my mistake. I knew it was coming from dependencies.yaml but I didn't realize it was being pinned to >=2023.1.1 there. I'm guessing when this was last adjusted python/pyproject.toml was not regenerated so I got confused by this seemingly arbitrary versioning.
There was a problem hiding this comment.
There are some features like channel pinnings (
dask/label/dev::dask) and comments that would be lost in the developer environment. @pentschev Do you have thoughts on whether we should rework this to be auto-generated like other RAPIDS packages or would you prefer to maintain it separately?
I guess that has been addressed by 4fc2c97, is that right? If so I think this question isn't relevant anymore.
See what you think of commit 4fc2c97. I added a
devsection with the dependencies likelibtoolandautoconfanddask-cudathat aren't otherwise specified by ucxx as build/run dependencies.
I see that it included dask/label/dev as well which is great and addresses your question from above, these changes are perfect, thanks for doing that Bradley!
| dev: | ||
| common: | ||
| - output_types: [conda] | ||
| packages: | ||
| # These packages are useful for development but not otherwise required to build/run | ||
| # RAPIDS | ||
| - pip | ||
| - dask-cuda==23.12.* | ||
| - dask-cudf==23.12.* | ||
| # UCX Build | ||
| - libtool | ||
| - automake | ||
| - autoconf | ||
| # UCXX Build | ||
| - pkg-config |
There was a problem hiding this comment.
Just to be sure I understand it correctly, the dev section is used to automatically generate conda/environments/*, is that it?
There was a problem hiding this comment.
The dev section is for dependencies that were added by hand to the previous conda environment (perhaps by you?) but do not fall into one of the other categories like build-time dependencies or run-time dependencies. I didn't want to lose that information but (if I understand correctly) none of the packages in dev are needed for CI to build and run tests.
| "cupy-cuda11x", | ||
| ] | ||
| "pytest-rerunfailures", | ||
| ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`. |
There was a problem hiding this comment.
These are coming from
dependencies.yaml. That file is supposed to be a "single source of truth" for all dependencies and their pinnings (with the exception of meta.yaml conda recipes and CMakeLists.txt which are not implemented at this time). Today,rapids-dependency-file-generatorgeneratespyproject.toml, developer conda environments, and can generate dependency lists on-the-fly that aren't committed to the repo but are used during various builds (e.g. devcontainers, rapids-compose).
Sorry, this was entirely my mistake. I knew it was coming from dependencies.yaml but I didn't realize it was being pinned to >=2023.1.1 there. I'm guessing when this was last adjusted python/pyproject.toml was not regenerated so I got confused by this seemingly arbitrary versioning.
| "cupy-cuda11x", | ||
| ] | ||
| "pytest-rerunfailures", | ||
| ] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../dependencies.yaml and run `rapids-dependency-file-generator`. |
There was a problem hiding this comment.
There are some features like channel pinnings (
dask/label/dev::dask) and comments that would be lost in the developer environment. @pentschev Do you have thoughts on whether we should rework this to be auto-generated like other RAPIDS packages or would you prefer to maintain it separately?
I guess that has been addressed by 4fc2c97, is that right? If so I think this question isn't relevant anymore.
See what you think of commit 4fc2c97. I added a
devsection with the dependencies likelibtoolandautoconfanddask-cudathat aren't otherwise specified by ucxx as build/run dependencies.
I see that it included dask/label/dev as well which is great and addresses your question from above, these changes are perfect, thanks for doing that Bradley!
|
/merge |
4 participants
Some improvements / modernizations for the package and dependencies. Depends on #108.