Skip to content

Diskann Benchmarking Wrapper#260

Merged
rapids-bot[bot] merged 169 commits intorapidsai:branch-25.04from
tarang-jain:diskann-wrapper
Feb 27, 2025
Merged

Diskann Benchmarking Wrapper#260
rapids-bot[bot] merged 169 commits intorapidsai:branch-25.04from
tarang-jain:diskann-wrapper

Conversation

@tarang-jain
Copy link
Copy Markdown
Contributor

@tarang-jain tarang-jain commented Jul 29, 2024 •
edited
Loading

Brings DiskANN into cuvs-bench

  • Build and search in-memory DiskANN index
  • Build and search SSD DiskANN index
  • Build a cuvs Vamana index on GPU and serialize it in DiskANN format. Search on CPU using in-memory DiskANN search API.

@tarang-jain tarang-jain added feature request New feature or request non-breaking Introduces a non-breaking change labels Jul 29, 2024
@tarang-jain tarang-jain self-assigned this Jul 29, 2024
@github-actions github-actions Bot added the Python label Aug 3, 2024
tarang-jain and others added 12 commits August 5, 2024 11:10
@tarang-jain
rm num_threads_
dbc84cc
@dantegd
FEA Add cuvs-bench to dependencies and conda environments
7e37218
@dantegd
FIX add missing deps
b2aef6d
@tarang-jain
Merge branch 'fea-add-bench-deps' of https://github.com/dantegd/cuvs …
b9762d5 
…into diskann-wrapper
@dantegd
FIX version and other improvements
bf75242
@dantegd
FEA Add cuvs_bench.run
a8bcdef
@tarang-jain
update patch;build command
3818da9
@tarang-jain
Merge branch 'cuvsbench-run' of https://github.com/dantegd/cuvs into …
cd8bfe5 
…diskann-wrapper
@dantegd
FIX some cuvs_bench python build dependencies
ec6d70c
@tarang-jain
Merge branch 'cuvsbench-run' of https://github.com/dantegd/cuvs into …
c9f797a 
…diskann-wrapper
@dantegd
FIX add missing algorithms.yaml
585ad53
@tarang-jain
Merge branch 'cuvsbench-run' of https://github.com/dantegd/cuvs into …
441ab2a 
…diskann-wrapper
@cjnolet cjnolet requested a review from a team as a code owner February 21, 2025 16:14
@cjnolet
Copy link
Copy Markdown
Member

cjnolet commented Feb 21, 2025

/ok to test

@tarang-jain
Copy link
Copy Markdown
Contributor Author

tarang-jain commented Feb 24, 2025

/ok to test

@tarang-jain
Copy link
Copy Markdown
Contributor Author

tarang-jain commented Feb 24, 2025

/rerun tests

@tarang-jain
Copy link
Copy Markdown
Contributor Author

tarang-jain commented Feb 24, 2025

/ok to test

@tarang-jain
Copy link
Copy Markdown
Contributor Author

tarang-jain commented Feb 24, 2025

/ok to test

@tarang-jain
Copy link
Copy Markdown
Contributor Author

tarang-jain commented Feb 27, 2025

/ok to test

jakirkham
jakirkham reviewed Feb 27, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@jakirkham jakirkham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Tarang! 🙏

Had a couple questions below

Note: There are other occurrences like those commented on. Just picked one of each to simplify the discussion

Comment thread conda/recipes/cuvs-bench-cpu/meta.yaml Outdated
Comment thread conda/recipes/cuvs-bench-cpu/meta.yaml Outdated
tarang-jain and others added 2 commits February 27, 2025 12:14
@tarang-jain @jakirkham
Update conda/recipes/cuvs-bench-cpu/meta.yaml
2ef2e35 
Remove libaio from run environment (meta.yaml)

Co-authored-by: jakirkham <jakirkham@gmail.com>
@tarang-jain @jakirkham
Update conda/recipes/cuvs-bench-cpu/meta.yaml
18e2de4 
"rm devel packages from run environment"

Co-authored-by: jakirkham <jakirkham@gmail.com>
@cjnolet
Copy link
Copy Markdown
Member

cjnolet commented Feb 27, 2025

/ok to test

@tarang-jain
Copy link
Copy Markdown
Contributor Author

tarang-jain commented Feb 27, 2025

/ok to test

@cjnolet cjnolet removed request for a team and jameslamb February 27, 2025 16:46
@cjnolet
Copy link
Copy Markdown
Member

cjnolet commented Feb 27, 2025

/merge

@rapids-bot rapids-bot Bot merged commit ce321a9 into rapidsai:branch-25.04 Feb 27, 2025
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Unstructured Data Processing Feb 27, 2025
This was referenced Mar 5, 2025
Closed
Merged
rapids-bot Bot pushed a commit that referenced this pull request Mar 6, 2025
@jameslamb
cuvs-bench-cpu: avoid 'mkl' dependency on aarch64 (#750)
f14266c 
Fixes rapidsai/docker#739

#260 introduced a runtime dependency on `mkl` for the `cuvs-bench-cpu` conda package. There are not aarch64 packages for `mkl` on conda-forge, so this makes `cuvs-bench-cpu` impossible to install on aarch64.

This fixes that, by applying the same "only add on x86_64" guard used for `mkl` everywhere else in this project, e.g. like this:

https://github.com/rapidsai/cuvs/blob/89b03493b487910d2125fde6680590adde8e2a95/conda/recipes/cuvs-bench-cpu/meta.yaml#L51

It also proposes adding import tests to the `cuvs-bench-cpu` conda recipe, so issues like this can be caught in CI in the future.

## Notes for Reviewers

I searched for references like this

```shell
git grep -i mkl
```

### How I tested this

Saw lines like this in `conda-python-build` logs:

```text
BUILD START: ['cuvs-bench-cpu-25.04.00a96-py312_250305_g94340bc_96.conda']
...
TEST START: /tmp/conda-bld-output/linux-aarch64/cuvs-bench-cpu-25.04.00a96-py312_250305_g94340bc_96.conda
...
import: 'cuvs_bench'
import: 'cuvs_bench.generate_groundtruth'
import: 'cuvs_bench.get_dataset'
import: 'cuvs_bench.plot'
import: 'cuvs_bench.run'
import: 'cuvs_bench.run.data_export'
import: 'cuvs_bench.split_groundtruth'
import: 'cuvs_bench'
import: 'cuvs_bench.generate_groundtruth'
import: 'cuvs_bench.get_dataset'
import: 'cuvs_bench.plot'
import: 'cuvs_bench.run'
import: 'cuvs_bench.run.data_export'
import: 'cuvs_bench.split_groundtruth'
...
TEST END: /tmp/conda-bld-output/linux-aarch64/cuvs-bench-cpu-25.04.00a96-py312_250305_g94340bc_96.conda
```

([build link](https://github.com/rapidsai/cuvs/actions/runs/13687659537/job/38276160494?pr=750#step:9:5961))

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Gil Forsyth (https://github.com/gforsyth)
  - Divye Gala (https://github.com/divyegala)

URL: #750
jiangyinzuo pushed a commit to jiangyinzuo/cuvs that referenced this pull request Mar 27, 2025
@jameslamb @jiangyinzuo
cuvs-bench-cpu: avoid 'mkl' dependency on aarch64 (rapidsai#750)
cedd565 
Fixes rapidsai/docker#739

rapidsai#260 introduced a runtime dependency on `mkl` for the `cuvs-bench-cpu` conda package. There are not aarch64 packages for `mkl` on conda-forge, so this makes `cuvs-bench-cpu` impossible to install on aarch64.

This fixes that, by applying the same "only add on x86_64" guard used for `mkl` everywhere else in this project, e.g. like this:

https://github.com/rapidsai/cuvs/blob/89b03493b487910d2125fde6680590adde8e2a95/conda/recipes/cuvs-bench-cpu/meta.yaml#L51

It also proposes adding import tests to the `cuvs-bench-cpu` conda recipe, so issues like this can be caught in CI in the future.

## Notes for Reviewers

I searched for references like this

```shell
git grep -i mkl
```

### How I tested this

Saw lines like this in `conda-python-build` logs:

```text
BUILD START: ['cuvs-bench-cpu-25.04.00a96-py312_250305_g94340bc_96.conda']
...
TEST START: /tmp/conda-bld-output/linux-aarch64/cuvs-bench-cpu-25.04.00a96-py312_250305_g94340bc_96.conda
...
import: 'cuvs_bench'
import: 'cuvs_bench.generate_groundtruth'
import: 'cuvs_bench.get_dataset'
import: 'cuvs_bench.plot'
import: 'cuvs_bench.run'
import: 'cuvs_bench.run.data_export'
import: 'cuvs_bench.split_groundtruth'
import: 'cuvs_bench'
import: 'cuvs_bench.generate_groundtruth'
import: 'cuvs_bench.get_dataset'
import: 'cuvs_bench.plot'
import: 'cuvs_bench.run'
import: 'cuvs_bench.run.data_export'
import: 'cuvs_bench.split_groundtruth'
...
TEST END: /tmp/conda-bld-output/linux-aarch64/cuvs-bench-cpu-25.04.00a96-py312_250305_g94340bc_96.conda
```

([build link](https://github.com/rapidsai/cuvs/actions/runs/13687659537/job/38276160494?pr=750#step:9:5961))

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Gil Forsyth (https://github.com/gforsyth)
  - Divye Gala (https://github.com/divyegala)

URL: rapidsai#750
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jakirkham jakirkham jakirkham left review comments

@bdice bdice bdice left review comments

@achirkin achirkin achirkin approved these changes

@cjnolet cjnolet cjnolet approved these changes

@gforsyth gforsyth gforsyth approved these changes

+1 more reviewer

@bkarsin bkarsin bkarsin approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@tarang-jain tarang-jain

Labels

benchmarking CMake cpp feature request New feature or request non-breaking Introduces a non-breaking change Python

Projects

Unstructured Data Processing
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@tarang-jain @cjnolet @gforsyth @jakirkham @bdice @bkarsin @achirkin @dantegd