Skip to content

Commit 8ebff3b

Browse files
jameslambjameslamb
authored
introduce libcugraph wheels (#4804)
Replaces #4340, contributes to rapidsai/build-planning#33. Proposes packaging `libcugraph` as a wheel, which is then re-used by `cugraph-cu{11,12}` and `pylibcugraph-cu{11,12}` wheels. ## Notes for Reviewers ### Benefits of these changes * smaller wheels (see "Size Changes" below) - *no more `pylibcugraph` and `cugraph` both holding copies of libcugraph.so* * faster compile times - *no more re-compiling RAFT, thanks to rapidsai/raft#2531 - *no more recompiling libcugraph.so in both `pylibcugraph` and `cugraph` wheel builds* * other benefits mentioned in rapidsai/build-planning#33 ### Wheel contents `libcugraph`: * `libcugraph.so` (shared library) * cuGraph headers * vendored dependencies (`fmt`, `spdlog`, CCCL, `cuco`) `pylibcugraph`: * `pylibcugraph` Python / Cython code and compiled Cython extensions `cugraph`: * `cugraph` Python / Cython code and compiled Cython extension ### Dependency Flows In short.... `libcugraph` contains `libcugraph.so` and `libcugraph_c.so` dynamic libraries and the headers to link against it. * Anything that needs to link against cuGraph at build time pulls in `libcugraph` wheels as a build dependency. * Anything that needs cuGraph's symbols at runtime pulls it in as a runtime dependency, and calls `libcugraph.load_library()`. For more details and some flowcharts, see rapidsai/build-planning#33 (comment) ### Size changes (CUDA 12, Python 3.12, x86_64) | wheel | num files (before) | num files (this PR) | size (before) | size (this PR) | |:---------------:|------------------:|-----------------:|--------------:|-------------:| | `libcugraph` | --- | 1762 | --- | 903M | | `pylibcugraph` | 190 | 187 | 901M | 2M | | `cugraph` | 315 | 313 | 899M | 3M | |**TOTAL** | **505** | **2,262** | **1,800M** | **908M** | *NOTES: size = compressed, "before" = 2025-01-13 nightlies* *This is a cuGraph-specific slice of the table from rapidsai/raft#2531. See that PR for details.* ### How I tested this These other PRs: * rapidsai/devcontainers#435 * rapidsai/cugraph-gnn#110 Authors: - James Lamb (https://github.com/jameslamb) - Ralph Liu (https://github.com/nv-rliu) - Bradley Dice (https://github.com/bdice) Approvers: - Brad Rees (https://github.com/BradReesWork) - Bradley Dice (https://github.com/bdice) URL: #4804
1 parent 2848cd4 commit 8ebff3b

28 files changed

Lines changed: 558 additions & 241 deletions

‎.github/workflows/build.yaml‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,29 @@ jobs:
6767
node_type: "gpu-v100-latest-1"
6868
run_script: "ci/build_docs.sh"
6969
sha: ${{ inputs.sha }}
70+
wheel-build-libcugraph:
71+
secrets: inherit
72+
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.02
73+
with:
74+
build_type: ${{ inputs.build_type || 'branch' }}
75+
branch: ${{ inputs.branch }}
76+
sha: ${{ inputs.sha }}
77+
date: ${{ inputs.date }}
78+
script: ci/build_wheel_libcugraph.sh
79+
node_type: cpu32
80+
wheel-publish-libcugraph:
81+
needs: wheel-build-libcugraph
82+
secrets: inherit
83+
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-25.02
84+
with:
85+
build_type: ${{ inputs.build_type || 'branch' }}
86+
branch: ${{ inputs.branch }}
87+
sha: ${{ inputs.sha }}
88+
date: ${{ inputs.date }}
89+
package-name: libcugraph
90+
package-type: cpp
7091
wheel-build-pylibcugraph:
92+
needs: wheel-build-libcugraph
7193
secrets: inherit
7294
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.02
7395
with:
@@ -76,7 +98,6 @@ jobs:
7698
sha: ${{ inputs.sha }}
7799
date: ${{ inputs.date }}
78100
script: ci/build_wheel_pylibcugraph.sh
79-
node_type: cpu32
80101
wheel-publish-pylibcugraph:
81102
needs: wheel-build-pylibcugraph
82103
secrets: inherit
@@ -87,8 +108,9 @@ jobs:
87108
sha: ${{ inputs.sha }}
88109
date: ${{ inputs.date }}
89110
package-name: pylibcugraph
111+
package-type: python
90112
wheel-build-cugraph:
91-
needs: wheel-publish-pylibcugraph
113+
needs: wheel-build-pylibcugraph
92114
secrets: inherit
93115
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.02
94116
with:
@@ -107,3 +129,4 @@ jobs:
107129
sha: ${{ inputs.sha }}
108130
date: ${{ inputs.date }}
109131
package-name: cugraph
132+
package-type: python

‎.github/workflows/pr.yaml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ jobs:
2323
- conda-python-build
2424
- conda-python-tests
2525
- docs-build
26+
- wheel-build-libcugraph
2627
- wheel-build-pylibcugraph
2728
- wheel-tests-pylibcugraph
2829
- wheel-build-cugraph
@@ -149,14 +150,22 @@ jobs:
149150
arch: "amd64"
150151
container_image: "rapidsai/ci-conda:cuda11.8.0-ubuntu22.04-py3.10"
151152
run_script: "ci/build_docs.sh"
152-
wheel-build-pylibcugraph:
153+
wheel-build-libcugraph:
153154
needs: checks
154155
secrets: inherit
155156
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.02
157+
with:
158+
# build for every combination of arch and CUDA version, but only for the latest Python
159+
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
160+
build_type: pull-request
161+
script: ci/build_wheel_libcugraph.sh
162+
wheel-build-pylibcugraph:
163+
needs: wheel-build-libcugraph
164+
secrets: inherit
165+
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.02
156166
with:
157167
build_type: pull-request
158168
script: ci/build_wheel_pylibcugraph.sh
159-
node_type: cpu32
160169
wheel-tests-pylibcugraph:
161170
needs: [wheel-build-pylibcugraph, changed-files]
162171
secrets: inherit

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ repos:
5959
- id: sphinx-lint
6060
args: ["--enable=all", "--disable=line-too-long"]
6161
- repo: https://github.com/rapidsai/dependency-file-generator
62-
rev: v1.16.0
62+
rev: v1.17.0
6363
hooks:
6464
- id: rapids-dependency-file-generator
6565
args: ["--clean"]

‎build.sh‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/bin/bash
22

3-
# Copyright (c) 2019-2024, NVIDIA CORPORATION.
3+
# Copyright (c) 2019-2025, NVIDIA CORPORATION.
44

55
# cugraph build script
66

@@ -175,11 +175,6 @@ SKBUILD_EXTRA_CMAKE_ARGS="${EXTRA_CMAKE_ARGS}"
175175
# Replace spaces with semicolons in SKBUILD_EXTRA_CMAKE_ARGS
176176
SKBUILD_EXTRA_CMAKE_ARGS=$(echo ${SKBUILD_EXTRA_CMAKE_ARGS} | sed 's/ /;/g')
177177

178-
# Append `-DFIND_CUGRAPH_CPP=ON` to EXTRA_CMAKE_ARGS unless a user specified the option.
179-
if [[ "${EXTRA_CMAKE_ARGS}" != *"DFIND_CUGRAPH_CPP"* ]]; then
180-
SKBUILD_EXTRA_CMAKE_ARGS="${SKBUILD_EXTRA_CMAKE_ARGS};-DFIND_CUGRAPH_CPP=ON"
181-
fi
182-
183178
# If clean or uninstall targets given, run them prior to any other steps
184179
if hasArg uninstall; then
185180
if [[ "$INSTALL_PREFIX" != "" ]]; then

‎ci/build_wheel.sh‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
#!/bin/bash
2-
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
2+
# Copyright (c) 2023-2025, NVIDIA CORPORATION.
33

44
set -euo pipefail
55

66
package_name=$1
77
package_dir=$2
8+
package_type=$3
89

910
source rapids-configure-sccache
1011
source rapids-date-string
@@ -29,22 +30,31 @@ python -m pip wheel \
2930

3031
sccache --show-adv-stats
3132

32-
case "${RAPIDS_CUDA_VERSION}" in
33-
12.*)
34-
EXCLUDE_ARGS=(
35-
--exclude "libcublas.so.12"
36-
--exclude "libcublasLt.so.12"
37-
--exclude "libcurand.so.10"
38-
--exclude "libcusolver.so.11"
39-
--exclude "libcusparse.so.12"
40-
--exclude "libnvJitLink.so.12"
41-
)
42-
;;
43-
11.*)
44-
EXCLUDE_ARGS=()
45-
;;
46-
esac
33+
EXCLUDE_ARGS=(
34+
--exclude "libraft.so"
35+
)
36+
37+
# Avoid picking up dependencies on CUDA wheels that come through
38+
# transitively from 'libraft'.
39+
#
40+
# 'libraft' wheels are responsible for carrying a runtime dependency on
41+
# these based on RAFT's needs.
42+
EXCLUDE_ARGS+=(
43+
--exclude "libcublas.so.12"
44+
--exclude "libcublasLt.so.12"
45+
--exclude "libcurand.so.10"
46+
--exclude "libcusolver.so.11"
47+
--exclude "libcusparse.so.12"
48+
--exclude "libnvJitLink.so.12"
49+
)
50+
51+
if [[ "${package_dir}" != "python/libcugraph" ]]; then
52+
EXCLUDE_ARGS+=(
53+
--exclude "libcugraph_c.so"
54+
--exclude "libcugraph.so"
55+
)
56+
fi
4757

4858
mkdir -p final_dist
4959
python -m auditwheel repair -w final_dist "${EXCLUDE_ARGS[@]}" dist/*
50-
RAPIDS_PY_WHEEL_NAME="${package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 python final_dist
60+
RAPIDS_PY_WHEEL_NAME="${package_name}_${RAPIDS_PY_CUDA_SUFFIX}" rapids-upload-wheels-to-s3 "${package_type}" final_dist

‎ci/build_wheel_cugraph.sh‎

Lines changed: 12 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,25 @@
11
#!/bin/bash
2-
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
2+
# Copyright (c) 2023-2025, NVIDIA CORPORATION.
33

44
set -euo pipefail
55

66
package_dir="python/cugraph"
77

88
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
99

10-
# Download the pylibcugraph wheel built in the previous step and make it
10+
# Download the libcugraph and pylibcugraph wheels built in the previous step and make them
1111
# available for pip to find.
12-
#
13-
# ensure 'cugraph' wheel builds always use the 'pylibcugraph' just built in the same CI run
14-
#
15-
# using env variable PIP_CONSTRAINT is necessary to ensure the constraints
16-
# are used when creating the isolated build environment
17-
CPP_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 /tmp/pylibcugraph_dist)
12+
LIBCUGRAPH_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/libcugraph_dist)
13+
PYLIBCUGRAPH_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python /tmp/pylibcugraph_dist)
1814

19-
echo "pylibcugraph-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${CPP_WHEELHOUSE}/pylibcugraph_*.whl)" > ./constraints.txt
20-
export PIP_CONSTRAINT="${PWD}/constraints.txt"
21-
22-
PARALLEL_LEVEL=$(python -c \
23-
"from math import ceil; from multiprocessing import cpu_count; print(ceil(cpu_count()/4))")
24-
case "${RAPIDS_CUDA_VERSION}" in
25-
12.*)
26-
EXTRA_CMAKE_ARGS=";-DUSE_CUDA_MATH_WHEELS=ON"
27-
;;
28-
11.*)
29-
EXTRA_CMAKE_ARGS=";-DUSE_CUDA_MATH_WHEELS=OFF"
30-
;;
31-
esac
15+
cat >> ./constraints.txt <<EOF
16+
libcugraph-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUGRAPH_WHEELHOUSE}/libcugraph_*.whl)
17+
pylibcugraph-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${PYLIBCUGRAPH_WHEELHOUSE}/pylibcugraph_*.whl)
18+
EOF
3219

33-
export SKBUILD_CMAKE_ARGS="-DDETECT_CONDA_ENV=OFF;-DFIND_CUGRAPH_CPP=OFF${EXTRA_CMAKE_ARGS}"
34-
export SKBUILD_BUILD_TOOL_ARGS="-j${PARALLEL_LEVEL};-l${PARALLEL_LEVEL}"
20+
# Using env variable PIP_CONSTRAINT is necessary to ensure the constraints
21+
# are used when creating the isolated build environment.
22+
export PIP_CONSTRAINT="${PWD}/constraints.txt"
3523

36-
./ci/build_wheel.sh cugraph ${package_dir}
24+
./ci/build_wheel.sh cugraph ${package_dir} python
3725
./ci/validate_wheel.sh ${package_dir} final_dist

‎ci/build_wheel_libcugraph.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/bin/bash
2+
# Copyright (c) 2025, NVIDIA CORPORATION.
3+
4+
set -euo pipefail
5+
6+
package_name="libcugraph"
7+
package_dir="python/libcugraph"
8+
9+
rapids-logger "Generating build requirements"
10+
matrix_selectors="cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION};cuda_suffixed=true"
11+
12+
rapids-dependency-file-generator \
13+
--output requirements \
14+
--file-key "py_build_${package_name}" \
15+
--file-key "py_rapids_build_${package_name}" \
16+
--matrix "${matrix_selectors}" \
17+
| tee /tmp/requirements-build.txt
18+
19+
rapids-logger "Installing build requirements"
20+
python -m pip install \
21+
-v \
22+
--prefer-binary \
23+
-r /tmp/requirements-build.txt
24+
25+
rapids-logger "Done build requirements"
26+
27+
# build with '--no-build-isolation', for better sccache hit rate
28+
# 0 really means "add --no-build-isolation" (ref: https://github.com/pypa/pip/issues/5735)
29+
export PIP_NO_BUILD_ISOLATION=0
30+
31+
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
32+
33+
./ci/build_wheel.sh libcugraph ${package_dir} cpp
34+
./ci/validate_wheel.sh ${package_dir} final_dist

‎ci/build_wheel_pylibcugraph.sh‎

Lines changed: 13 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,23 @@
11
#!/bin/bash
2-
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
2+
# Copyright (c) 2023-2025, NVIDIA CORPORATION.
33

44
set -euo pipefail
55

66
package_dir="python/pylibcugraph"
77

8-
PARALLEL_LEVEL=$(python -c \
9-
"from math import ceil; from multiprocessing import cpu_count; print(ceil(cpu_count()/4))")
8+
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
109

11-
case "${RAPIDS_CUDA_VERSION}" in
12-
12.*)
13-
EXTRA_CMAKE_ARGS=";-DUSE_CUDA_MATH_WHEELS=ON"
14-
;;
15-
11.*)
16-
EXTRA_CMAKE_ARGS=";-DUSE_CUDA_MATH_WHEELS=OFF"
17-
;;
18-
esac
10+
# Download the libcugraph wheel built in the previous step and make it
11+
# available for pip to find.
12+
LIBCUGRAPH_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/libcugraph_dist)
1913

20-
export SKBUILD_CMAKE_ARGS="-DDETECT_CONDA_ENV=OFF;-DFIND_CUGRAPH_CPP=OFF${EXTRA_CMAKE_ARGS}"
21-
export SKBUILD_BUILD_TOOL_ARGS="-j${PARALLEL_LEVEL};-l${PARALLEL_LEVEL}"
14+
cat >> ./constraints.txt <<EOF
15+
libcugraph-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUGRAPH_WHEELHOUSE}/libcugraph_*.whl)
16+
EOF
2217

23-
./ci/build_wheel.sh pylibcugraph ${package_dir}
18+
# Using env variable PIP_CONSTRAINT is necessary to ensure the constraints
19+
# are used when creating the isolated build environment.
20+
export PIP_CONSTRAINT="${PWD}/constraints.txt"
21+
22+
./ci/build_wheel.sh pylibcugraph ${package_dir} python
2423
./ci/validate_wheel.sh ${package_dir} final_dist

‎ci/release/update-version.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
2-
# Copyright (c) 2018-2024, NVIDIA CORPORATION.
2+
# Copyright (c) 2018-2025, NVIDIA CORPORATION.
33
# Licensed under the Apache License, Version 2.0 (the "License");
44
# you may not use this file except in compliance with the License.
55
# You may obtain a copy of the License at
@@ -57,6 +57,7 @@ DEPENDENCIES=(
5757
dask-cuda
5858
dask-cudf
5959
libcudf
60+
libcugraph
6061
libraft
6162
librmm
6263
pylibcugraph
@@ -109,4 +110,3 @@ find .devcontainer/ -type f -name devcontainer.json -print0 | while IFS= read -r
109110
done
110111

111112
sed_runner "s/:[0-9][0-9]\.[0-9][0-9]/:${NEXT_SHORT_TAG}/" ./notebooks/README.md
112-
sed_runner "s/branch-[0-9][0-9].[0-9][0-9]/branch-${NEXT_SHORT_TAG}/" ./docs/cugraph/source/nx_cugraph/nx_cugraph.md

‎ci/test_wheel_cugraph.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,13 @@ set -eoxu pipefail
77
mkdir -p ./dist
88
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
99
RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
10+
RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./local-libcugraph-dep
1011
RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./local-pylibcugraph-dep
1112

1213
# echo to expand wildcard before adding `[extra]` requires for pip
1314
python -m pip install \
1415
"$(echo ./dist/cugraph*.whl)[test]" \
15-
./local-pylibcugraph-dep/pylibcugraph*.whl
16+
./local-pylibcugraph-dep/pylibcugraph*.whl \
17+
./local-libcugraph-dep/libcugraph*.whl
1618

1719
./ci/test_wheel.sh cugraph

0 commit comments

Comments
 (0)