feat(libcuvs): port libcuvs to rattler-build by gforsyth · Pull Request #751 · rapidsai/cuvs

gforsyth · 2025-03-06T17:57:09Z

This PR ports cuVS over to rattler-build for building conda packages.

Some changes to the build process:

cuvs-bench was recompiling the entire library separately -- caching was hopefully helping with this but it was an unnecessary double-compile. Instead, I've added the bench-ann to the pre-build stage in libcuvs and create a libcuvs-bench output there, that cuvs-bench then depends on to provide the necessary shared objects.
Edited build.sh so that if -n is passed, the cuvs-bench Python package is not installed.
libcuvs-static seems to be a leftover from a previous incarnation -- the existing libcuvs-static packages on rapidsai-nightly consist only of build metadata files and are otherwise empty. I've removed the unused --compile-static-lib option from build.sh

Add source and fix nccl spec Add pre-build stage, remove static output, add tests and examples fix(libcuvs): handle most overlinking errors fix(libcuvs): install cuvs, c_api, and hnswlib components feat(libcuvs): remove defunct build scripts

copy-pr-bot · 2025-03-06T17:57:12Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

gforsyth · 2025-03-06T19:26:41Z

/ok to test

gforsyth · 2025-03-06T21:54:48Z

/ok to test

This reverts commit b597650.

gforsyth · 2025-03-06T22:00:43Z

/ok to test

gforsyth · 2025-03-06T22:33:06Z

/ok to test

This reverts commit 2eb7ddd.

bdice

A few comments but overall this is great. I'm really glad this rattler conversion is getting easier as we progress through more RAPIDS libraries.

gforsyth · 2025-03-13T18:10:25Z

This is ready for another look @bdice

bdice

All looks good to me!

bdice · 2025-03-19T16:53:47Z

+      then:
+        - libaio
+        - libboost-devel=1.87
+        - mkl-devel=2023


For follow-up: Why is this pinned to mkl 2023? The latest is 2025.0.1. https://anaconda.org/conda-forge/mkl/files

vyasr

Couple notes for future work, but LGTM.

vyasr · 2025-03-19T17:17:47Z

+      # Remove `-fdebug-prefix-map` line from CFLAGS and CXXFLAGS so the
+      # incrementing version number in the compile line doesn't break the
+      # cache
+      set -x
+      export CFLAGS=$(echo $CFLAGS | sed -E 's@\-fdebug\-prefix\-map[^ ]*@@g')
+      export CXXFLAGS=$(echo $CXXFLAGS | sed -E 's@\-fdebug\-prefix\-map[^ ]*@@g')
+      set +x


Suggestion: rapidsai/rapids-cmake#798 (comment)

vyasr · 2025-03-19T17:25:14Z

Future suggestion: rather than maintaining separate cuvs-bench and cuvs-bench-cpu recipes, we should parametrize the recipe. The two should be almost identical.

gforsyth · 2025-03-19T17:41:46Z

/merge

gforsyth · 2025-03-19T17:46:25Z

Followups recorded in rapidsai/build-planning#47 (comment)

This PR ports cuVS over to `rattler-build` for building conda packages. Some changes to the build process: * `cuvs-bench` was recompiling the entire library separately -- caching was hopefully helping with this but it was an unnecessary double-compile. Instead, I've added the `bench-ann` to the pre-build stage in `libcuvs` and create a `libcuvs-bench` output there, that `cuvs-bench` then depends on to provide the necessary shared objects. * Edited `build.sh` so that if `-n` is passed, the `cuvs-bench` Python package is not installed. * `libcuvs-static` seems to be a leftover from a previous incarnation -- the existing `libcuvs-static` packages on `rapidsai-nightly` consist only of build metadata files and are otherwise empty. I've removed the unused `--compile-static-lib` option from `build.sh` Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: rapidsai#751

gforsyth added 12 commits March 5, 2025 10:34

feat(libcudf): port libcuvs to rattler-build

1986c44

Add source and fix nccl spec Add pre-build stage, remove static output, add tests and examples fix(libcuvs): handle most overlinking errors fix(libcuvs): install cuvs, c_api, and hnswlib components feat(libcuvs): remove defunct build scripts

fix(examples): define hasArg function in examples/build.sh

821af31

feat(cuvs): port cuvs to rattler-build

92524e4

fix(cuvs-bench): skip recompiling C library for bench-ann

64e54f6

feat(cuvs-bench): port cuvs-bench to rattler-build

dd1cf19

feat(cuvs-bench): add libcuvs-bench-ann output to cpp recipe

e0220b5

chore(build): revert changes to build.sh

9a14480

feat(build.sh): allow skipping python install for bench-ann

622b3b7

feat(libcuvs): add benchmark c++ output

20aeba6

fix(cuvs-bench): remove build environment (now in libcuvs)

c0c69fd

Port cuvs-bench-cpu to rattler-build

cc42cbb

refactor: remove conda-build recipes and unused build scripts

1e6e721

gforsyth added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Mar 6, 2025

github-actions Bot added the Python label Mar 6, 2025

refactor(build.sh): remove defunct static libs flag

a3a994b

gforsyth changed the title ~~feat(libcudf): port libcuvs to rattler-build~~ feat(libcuvs): port libcuvs to rattler-build Mar 6, 2025

refactor(cuvs-bench-cpu): add import tests and remove cuda variable

fb62054

switch over to rattler-build in CI

060860c

github-actions Bot added the ci label Mar 6, 2025

disable most workflows for testing rattler

b597650

Revert "disable most workflows for testing rattler"

50e81ce

This reverts commit b597650.

Add openblas to pre-build stage for bench-ann

1e2bf00

gforsyth added 2 commits March 7, 2025 05:27

fix(libcuvs): gate x86 deps in libcuvs-bench output

5896758

chore: disable wheel and other workflows again

e105ec7

gforsyth requested a review from AyodeAwe March 10, 2025 17:44

Reapply "remove datasets from gitignore"

17085c8

This reverts commit 2eb7ddd.

gforsyth commented Mar 10, 2025

View reviewed changes

Comment thread .gitignore Outdated

Constrain .gitignore entry to only top-level datasets folder

6933581

gforsyth force-pushed the rattler-build branch from 170663e to 6933581 Compare March 11, 2025 14:01

gforsyth added 2 commits March 11, 2025 14:21

Add additional run_exports to avoid overconstraining outputs

6ec34e6

Merge branch 'branch-25.04' into rattler-build

c84e471

bdice reviewed Mar 12, 2025

View reviewed changes

gforsyth commented Mar 12, 2025

View reviewed changes

Comment thread conda/recipes/libcuvs/recipe.yaml

gforsyth added 3 commits March 12, 2025 17:35

Fix cuda 11 runtime deps and relax mkl pin

0c84f88

Only require mkl on linux64

5d8c652

ignore run export for cudatoolkit in libcuvs

08373fd

gforsyth mentioned this pull request Mar 13, 2025

[FEA] Add CMake component that contains libdiskann.so #760

Open

gforsyth added 2 commits March 13, 2025 09:46

Fix the other yaml comment

2e230c7

Make test for datasets.yaml existence more robust

6ef134e

gforsyth mentioned this pull request Mar 13, 2025

[discussion] should we update our s3 sccache prefixes? rapidsai/build-planning#161

Closed

gforsyth added 2 commits March 14, 2025 10:53

Disable binary prefix replacement

851825f

refactor: make sccache prefix keys more granular

21caaee

gforsyth force-pushed the rattler-build branch from 9be2686 to 21caaee Compare March 18, 2025 12:58

Merge branch 'branch-25.04' into rattler-build

95897cb

gforsyth mentioned this pull request Mar 18, 2025

Replace conda-build with rattler-build rapidsai/build-planning#47

Open

43 tasks

bdice approved these changes Mar 19, 2025

View reviewed changes

vyasr approved these changes Mar 19, 2025

View reviewed changes

rapids-bot Bot merged commit 2c3df75 into rapidsai:branch-25.04 Mar 19, 2025

gforsyth deleted the rattler-build branch March 19, 2025 17:42

Conversation

gforsyth commented Mar 6, 2025

Uh oh!

copy-pr-bot Bot commented Mar 6, 2025

Uh oh!

gforsyth commented Mar 6, 2025

Uh oh!

gforsyth commented Mar 6, 2025

Uh oh!

gforsyth commented Mar 6, 2025

Uh oh!

gforsyth commented Mar 6, 2025

Uh oh!

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gforsyth commented Mar 13, 2025

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

bdice Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

vyasr left a comment

Choose a reason for hiding this comment

Uh oh!

vyasr Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

vyasr Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

gforsyth commented Mar 19, 2025

Uh oh!

gforsyth commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants