Skip to content

Update versions/recipe logic. #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 8 commits into from
Oct 24, 2023
Merged

Update versions/recipe logic. #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
db1aa60
Update versions/recipe logic.
bdice Oct 20, 2023
b66320c
Generate pyproject.toml, clean up dependencies.
bdice Oct 20, 2023
4fc2c97
Generate conda developer environment.
bdice Oct 20, 2023
02fce3e
Update pyproject.toml in update-version.sh.
bdice Oct 20, 2023
9916df0
Merge rapidsai/branch-0.35 into bdice/update-deps
jakirkham Oct 21, 2023
1da88f0
Remove dask/distributed lower bound.
bdice Oct 23, 2023
859e668
Merge remote-tracking branch 'upstream/branch-0.35' into update-deps
bdice Oct 23, 2023
b0b1e6d
Update dependencies with `rapids-dependency-file-generator`
pentschev Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions ci/release/update-version.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,19 @@ sed_runner "s/^version = .*/version = \"${NEXT_FULL_TAG}\"/g" python/pyproject.t
# bump RAPIDS libs
sed_runner "/- librmm =/ s/=.*/=${NEXT_RAPIDS_VERSION}/g" conda/recipes/ucxx/meta.yaml
sed_runner "/- rmm =/ s/=.*/=${NEXT_RAPIDS_VERSION}/g" conda/recipes/ucxx/meta.yaml
for FILE in conda/environments/*.yaml dependencies.yaml; do
sed_runner "/- cuda==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};
sed_runner "/- cudf==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};
sed_runner "/- dask-cuda==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};
sed_runner "/- dask-cudf==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};
sed_runner "/- librmm==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};
sed_runner "/- rmm==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};

DEPENDENCIES=(
cudf
dask-cuda
dask-cudf
librmm
rmm
)
for DEP in "${DEPENDENCIES[@]}"; do
for FILE in dependencies.yaml conda/environments/*.yaml; do
sed_runner "/-.* ${DEP}==/ s/==.*/==${NEXT_SHORT_TAG_PEP440}\.*/g" ${FILE};
done
sed_runner "/\"${DEP}==/ s/==.*\"/==${NEXT_SHORT_TAG_PEP440}\.*\"/g" python/pyproject.toml;
done

# rapids-cmake version
Expand Down
48 changes: 48 additions & 0 deletions conda/environments/all_cuda-118_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- autoconf
- automake
- c-compiler
- cloudpickle
- cmake>=3.26.4
- cuda-version=11.8
- cudatoolkit
- cudf==23.12.*
- cupy
- cxx-compiler
- cython>=3.0.0
- dask
- dask-cuda==23.12.*
- dask-cudf==23.12.*
- distributed
- fmt>=9.1.0,<10
- gmock>=1.13.0
- gtest>=1.13.0
- librmm==23.12.*
- libtool
- ninja
- numba>=0.57.1
- numpy>=1.21
- pip
- pkg-config
- pre-commit
- pynvml>=11.4.1
- pytest
- pytest-asyncio
- pytest-rerunfailures
- python>=3.9,<3.11
- rmm==23.12.*
- scikit-build>=0.13.1
- setuptools
- spdlog>=1.11.0,<1.12
- tomli
- ucx
- wheel
name: all_cuda-118_arch-x86_64
48 changes: 48 additions & 0 deletions conda/environments/all_cuda-120_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
- autoconf
- automake
- c-compiler
- cloudpickle
- cmake>=3.26.4
- cuda-cudart-dev
- cuda-version=12.0
- cudf==23.12.*
- cupy
- cxx-compiler
- cython>=3.0.0
- dask
- dask-cuda==23.12.*
- dask-cudf==23.12.*
- distributed
- fmt>=9.1.0,<10
- gmock>=1.13.0
- gtest>=1.13.0
- librmm==23.12.*
- libtool
- ninja
- numba>=0.57.1
- numpy>=1.21
- pip
- pkg-config
- pre-commit
- pynvml>=11.4.1
- pytest
- pytest-asyncio
- pytest-rerunfailures
- python>=3.9,<3.11
- rmm==23.12.*
- scikit-build>=0.13.1
- setuptools
- spdlog>=1.11.0,<1.12
- tomli
- ucx
- wheel
name: all_cuda-120_arch-x86_64
36 changes: 0 additions & 36 deletions conda/environments/ucxx-cuda118_arch-x86_64.yaml
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion conda/recipes/ucxx/conda_build_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ sysroot_linux_64:
- "2.17"

cmake:
- ">=3.23.1,!=3.25.0"
- ">=3.26.4"

python:
- 3.9
Expand Down
4 changes: 1 addition & 3 deletions conda/recipes/ucxx/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ package:
name: libucxx-split

source:
git_url: ../../..
path: ../../..

build:
script_env:
Expand Down Expand Up @@ -206,7 +206,6 @@ outputs:
host:
- python
- pip
- cmake
- scikit-build>=0.13.1
- setuptools
- cython >=3.0.0
Expand All @@ -220,7 +219,6 @@ outputs:
- cuda-version {{ cuda_version }}
run:
- python * *_cpython
- packaging
{% if cuda_major == "11" %}
- cudatoolkit
{% endif %}
Expand Down
2 changes: 1 addition & 1 deletion cpp/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: BSD 3-Clause License
# =================================================================================

cmake_minimum_required(VERSION 3.23.1 FATAL_ERROR)
cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR)

include(../fetch_rapids.cmake)
include(rapids-cmake)
Expand Down
82 changes: 69 additions & 13 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
# Dependency list for https://github.com/rapidsai/dependency-file-generator
files:
all:
output: none
output: conda
matrix:
cuda: ["11.8", "12.0"]
arch: [x86_64]
includes:
- build_cpp
- build_python
- checks
- cudatoolkit
- dev
- py_version
- run_python
- test_cpp
Expand All @@ -26,28 +31,60 @@ files:
includes:
- checks
- py_version
py_build:
output: pyproject
pyproject_dir: python
extras:
table: build-system
includes:
- build_python
py_run:
output: pyproject
pyproject_dir: python
extras:
table: project
includes:
- run_python
py_test:
output: pyproject
pyproject_dir: python
extras:
table: project.optional-dependencies
key: test
includes:
- test_python
channels:
- rapidsai
- rapidsai-nightly
- dask/label/dev
- conda-forge
- nvidia
dependencies:
build_python:
build_cpp:
common:
- output_types: [conda, requirements, pyproject]
packages:
- &cmake_ver cmake>=3.23.1,!=3.25.0
- rmm==23.12.*
- output_types: conda
packages:
- cython>=3.0.0
- c-compiler
- cxx-compiler
- &cmake_ver cmake>=3.26.4
- fmt>=9.1.0,<10
- &gmock gmock>=1.13.0
- &gtest gtest>=1.13.0
- librmm==23.12.*
- &numpy numpy>=1.21
- ninja
- spdlog>=1.11.0,<1.12
build_python:
common:
- output_types: [conda, requirements, pyproject]
packages:
- *cmake_ver
- cython>=3.0.0
- ninja
- rmm==23.12.*
- scikit-build>=0.13.1
- setuptools
- tomli
- wheel
checks:
common:
- output_types: [conda, requirements]
Expand Down Expand Up @@ -82,6 +119,21 @@ dependencies:
packages:
- cuda-version=12.0
- cuda-cudart-dev
dev:
common:
- output_types: [conda]
packages:
# These packages are useful for development but not otherwise required to build/run
# RAPIDS
- pip
- dask-cuda==23.12.*
- dask-cudf==23.12.*
# UCX Build
- libtool
- automake
- autoconf
# UCXX Build
- pkg-config
Comment on lines +122 to +136
Copy link
Member

@pentschev pentschev Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be sure I understand it correctly, the dev section is used to automatically generate conda/environments/*, is that it?

Copy link
Contributor Author

@bdice bdice Oct 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dev section is for dependencies that were added by hand to the previous conda environment (perhaps by you?) but do not fall into one of the other categories like build-time dependencies or run-time dependencies. I didn't want to lose that information but (if I understand correctly) none of the packages in dev are needed for CI to build and run tests.

py_version:
specific:
- output_types: conda
Expand All @@ -101,7 +153,10 @@ dependencies:
common:
- output_types: [conda, requirements, pyproject]
packages:
- *numpy
- numpy>=1.21
- pynvml>=11.4.1
- output_types: [conda]
packages:
- ucx
test_cpp:
common:
Expand All @@ -112,13 +167,14 @@ dependencies:
- *gmock
test_python:
common:
- output_types: [conda, requirements]
- output_types: [conda, requirements, pyproject]
packages:
- cloudpickle
- cudf==23.12.*
- cupy
- dask>=2023.1.1
- distributed>=2023.1.1
- numba>=0.57.0
- dask
- distributed
- numba>=0.57.1
- pytest
- pytest-asyncio
- pytest-rerunfailures
2 changes: 1 addition & 1 deletion python/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: BSD 3-Clause License
# =================================================================================

cmake_minimum_required(VERSION 3.23.1 FATAL_ERROR)
cmake_minimum_required(VERSION 3.26.4 FATAL_ERROR)

set(ucxx_version 0.35.00)

Expand Down
Loading