Skip to content

Revert "Revert "Build C++ wheel (#2264)""#2306

Closed
vyasr wants to merge 1 commit intobranch-24.06from
revert-2305-chore/revert_cpp_wheel
Closed

Revert "Revert "Build C++ wheel (#2264)""#2306
vyasr wants to merge 1 commit intobranch-24.06from
revert-2305-chore/revert_cpp_wheel

Conversation

@vyasr
Copy link
Copy Markdown
Contributor

@vyasr vyasr commented May 10, 2024

Reverts #2305

Opening this preemptively for tracking, but no plans to try and merge this for a while (see rapidsai/build-planning#33 (comment)).

@vyasr vyasr mentioned this pull request May 10, 2024
Closed
11 tasks
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 10, 2024

Codecov Report

Attention: Patch coverage is 66.66667% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 87.27%. Comparing base (d11e78a) to head (d121816).
Report is 3 commits behind head on branch-24.06.

Files Patch % Lines
python/pylibraft/pylibraft/__init__.py 66.66% 2 Missing ⚠️
python/raft-dask/raft_dask/__init__.py 66.66% 2 Missing ⚠️
Additional details and impacted files 
@@              Coverage Diff              @@
##           branch-24.06    #2306   +/-   ##
=============================================
  Coverage         87.27%   87.27%           
=============================================
  Files                25       25           
  Lines               550      550           
=============================================
  Hits                480      480           
  Misses               70       70

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jameslamb
jameslamb reviewed Jul 3, 2024
View reviewed changes
Comment thread ci/build_wheel_python.sh
# Iterate over the array
for wheelhouse in "${WHEELHOUSES[@]}"; do
FIND_LINKS+="--find-links ${wheelhouse} "
done
Copy link
Copy Markdown
Member

@jameslamb jameslamb Jul 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whenever we pick this back up, I think it'd be preferable to switch away from FIND_LINKS towards a constraint file, as described in rapidsai/build-planning#69 (comment).

@cjnolet
Copy link
Copy Markdown
Member

cjnolet commented Nov 20, 2024

This PR has gotten stale. Closing. @vyasr please reopen if needed.

@cjnolet cjnolet closed this Nov 20, 2024
@vyasr
Copy link
Copy Markdown
Contributor Author

vyasr commented Nov 20, 2024

Thanks Corey, yes closing is fine. When we get back around to this we can rebase my branch and open a new PR.

@jameslamb jameslamb mentioned this pull request Jan 10, 2025
Merged
raydouglass pushed a commit that referenced this pull request Jan 16, 2025
@jameslamb
introduce libraft wheels (#2531)
8299f17 
Replaces #2306, contributes to
rapidsai/build-planning#33.

Proposes packaging `libraft` as a wheel, which is then re-used by:

* `pylibraft-cu{11,12}` and `raft-cu{11,12}` (this PR)
* `libcugraph-cu{11,12}`, `pylibcugraph-cu{11,12}`, and
`cugraph-cu{11,12}` in rapidsai/cugraph#4804
* `libcuml-cu{11,12}` and `cuml-cu{11,12}` in
rapidsai/cuml#6199

As part of this, also proposes:

* introducing a new CMake option, `RAFT_COMPILE_DYNAMIC_ONLY`, to allow
building/installing only the dynamic shared library (i.e. skipping the
static library)
* enforcing `rapids-cmake`'s preferred CMake style
(#2531 (comment))
* making wheel-building CI jobs always depend on other wheel-building CI
jobs, not tests or `*-publish` (to reduce end-to-end CI time)

## Notes for Reviewers

### Benefits of these changes

* smaller wheels (see "Size Changes" below)
* faster compile times (no more re-compiling RAFT in cuGraph and cuML
CI)
* other benefits mentioned in
rapidsai/build-planning#33

### Wheel contents

`libraft`:

* `libraft.so` (shared library)
* RAFT headers
* vendored dependencies (`fmt`, CCCL, `cuco`, `cute`, `cutlass`)

`pylibraft`:

* `pylibraft` Python / Cython code and compiled Cython extensions

`raft-dask`:

* `raft-dask` Python / Cython code and compiled Cython extension

### Dependency Flows

In short.... `libraft` contains a `libraft.so` dynamic library and the
headers to link against it.

* Anything that needs to link against RAFT at build time pulls in
`libraft` wheels as a build dependency.
* Anything that needs RAFT's symbols at runtime pulls it in as a runtime
dependency, and calls `libraft.load_library()`.

For more details and some flowcharts, see
rapidsai/build-planning#33 (comment)

### Size changes (CUDA 12, Python 3.12, x86_64)

| wheel | num files (before) | num files (these PRs) | size (before) |
size (these PRs) |

|:---------------:|------------------:|-----------------:|--------------:|-------------:|
| `libraft`. | --- | 3169 | --- | 19M |
| `pylibraft` | 64 | 63 | 11M | 1M |
| `raft-dask` | 29 | 28 | 188M | 188M |
| `libcugraph` | --- | 1762 | --- | 903M |
| `pylibcugraph` | 190 | 187 | 901M | 2M |
| `cugraph` | 315 | 313 | 899M | 3.0M |
| `libcuml` | --- | 1766 | --- | 289M |
| `cuml` | 442 | --- | 517M | --- |
|**TOTAL** | **1,040** | **7,268** | **2,516M** | **1,405M** |

*NOTES: size = compressed, "before" = 2025-01-13 nightlies*

<details><summary>how I calculated those (click me)</summary>

* `cugraph`: nightly commit =
rapidsai/cugraph@8507cbf,
PR = rapidsai/cugraph#4804
* `cuml`: nightly commit =
rapidsai/cuml@7c715c4,
PR = rapidsai/cuml#6199
* `raft`: nightly commit =
1b62c41,
PR = this PR

```shell
docker run \
    --rm \
    --network host \
    --env RAPIDS_NIGHTLY_DATE=2025-01-13 \
    --env CUGRAPH_NIGHTLY_SHA=8507cbf63db2f349136b266d3e6e787b189f45a0 \
    --env CUGRAPH_PR="pull-request/4804" \
    --env CUGRAPH_PR_SHA="2ef32eaa006a84c0bd16220bb8e8af34198fbee8" \
    --env CUML_NIGHTLY_SHA=7c715c494dff71274d0fdec774bdee12a7e78827 \
    --env CUML_PR="pull-request/6199" \
    --env CUML_PR_SHA="2ef32eaa006a84c0bd16220bb8e8af34198fbee8" \
    --env RAFT_NIGHTLY_SHA=1b62c4117a35b11ce3c830daae248e32ebf75e3f \
    --env RAFT_PR="pull-request/2531" \
    --env RAFT_PR_SHA="0d6597b08919f2aae8ac268f1a68d6a8fe5beb4e" \
    --env RAPIDS_PY_CUDA_SUFFIX=cu12 \
    --env WHEEL_DIR_BEFORE=/tmp/wheels-before \
    --env WHEEL_DIR_AFTER=/tmp/wheels-after \
    -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \
    bash

# --- nightly wheels --- #
mkdir -p ./wheels-before

export RAPIDS_BUILD_TYPE=branch
export RAPIDS_REF_NAME="branch-25.02"

# pylibraft
RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/raft \
RAPIDS_SHA=${RAFT_NIGHTLY_SHA} \
    rapids-download-wheels-from-s3 python ./wheels-before

# raft-dask
RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/raft \
RAPIDS_SHA=${RAFT_NIGHTLY_SHA} \
    rapids-download-wheels-from-s3 python ./wheels-before

# cugraph
RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cugraph \
RAPIDS_SHA=${CUGRAPH_NIGHTLY_SHA} \
    rapids-download-wheels-from-s3 python ./wheels-before

# pylibcugraph
RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cugraph \
RAPIDS_SHA=${CUGRAPH_NIGHTLY_SHA} \
    rapids-download-wheels-from-s3 python ./wheels-before

# cuml
RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cuml \
RAPIDS_SHA=${CUML_NIGHTLY_SHA} \
    rapids-download-wheels-from-s3 python ./wheels-before

# --- wheels from CI --- #
mkdir -p ./wheels-after

export RAPIDS_BUILD_TYPE="pull-request"

# libraft
RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/raft \
RAPIDS_REF_NAME="${RAFT_PR}" \
RAPIDS_SHA="${RAFT_PR_SHA}" \
    rapids-download-wheels-from-s3 cpp ./wheels-after

# pylibraft
RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/raft \
RAPIDS_REF_NAME="${RAFT_PR}" \
RAPIDS_SHA="${RAFT_PR_SHA}" \
    rapids-download-wheels-from-s3 python ./wheels-after

# raft-dask
RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/raft \
RAPIDS_REF_NAME="${RAFT_PR}" \
RAPIDS_SHA="${RAFT_PR_SHA}" \
    rapids-download-wheels-from-s3 python ./wheels-after

# libcugraph
RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cugraph \
RAPIDS_REF_NAME="${CUGRAPH_PR}" \
RAPIDS_SHA="${CUGRAPH_PR_SHA}" \
    rapids-download-wheels-from-s3 cpp ./wheels-after

# pylibcugraph
RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cugraph \
RAPIDS_REF_NAME="${CUGRAPH_PR}" \
RAPIDS_SHA="${CUGRAPH_PR_SHA}" \
    rapids-download-wheels-from-s3 python ./wheels-after

# cugraph
RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cugraph \
RAPIDS_REF_NAME="${CUGRAPH_PR}" \
RAPIDS_SHA="${CUGRAPH_PR_SHA}" \
    rapids-download-wheels-from-s3 python ./wheels-after

# libcuml
RAPIDS_PY_WHEEL_NAME="libcuml_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cuml \
RAPIDS_REF_NAME="${CUML_PR}" \
RAPIDS_SHA="${CUML_PR_SHA}" \
    rapids-download-wheels-from-s3 cpp ./wheels-after

# cuml
RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \
RAPIDS_REPOSITORY=rapidsai/cuml \
RAPIDS_REF_NAME="${CUML_PR}" \
RAPIDS_SHA="${CUML_PR_SHA}" \
    rapids-download-wheels-from-s3 python ./wheels-after

pip install pydistcheck
pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ./wheels-before/*.whl \
| grep -E '^checking|files: | compressed' \
> ./before.txt

# get more exact sizes
du -sh ./wheels-before/*

pydistcheck \
    --inspect \
    --select 'distro-too-large-compressed' \
    ./wheels-after/*.whl \
| grep -E '^checking|files: | compressed' \
> ./after.txt

# get more exact sizes
du -sh ./wheels-after/*
```

</details>

### How I tested this

These other PRs:

* rapidsai/devcontainers#435
* rapidsai/cugraph-gnn#110
* rapidsai/cuml#6199
* rapidsai/cugraph#4804
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jameslamb jameslamb jameslamb left review comments

Assignees

@vyasr vyasr

Labels

ci CMake cpp python

Projects

Unstructured Data Processing
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vyasr @codecov-commenter @cjnolet @jameslamb