Add filter for cagra::merge by benfred · Pull Request #1496 · rapidsai/cuvs

benfred · 2025-11-05T05:59:00Z

Add an optional filter argument to cagra::merge, so that we can compact the index to remove deleted items when merging.

Also remove the 'LOGICAL_MERGE' strategy from the cagra::merge api, since this is only supported with the composite merge api. (The cagra::merge function ignore this value if set, and only does a logical merge).

Add an optional filter argument to cagra::merge, so that we can compact the index to remove deleted items when merging

copy-pr-bot · 2025-11-05T05:59:03Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

cjnolet · 2025-11-19T20:14:58Z

+  cagra::index_params output_index_params;
+
+  /// Strategy for merging. Defaults to `MergeStrategy::MERGE_STRATEGY_PHYSICAL`.
+  cuvs::neighbors::MergeStrategy merge_strategy =


Should we just remove the merge params altogether? I don't know if we need them anymore. Maybe we do- that's why I'm asking. Might help to think forward to when we have a SMART merge option (maye you've thought of this already). But maybe we don't need a "merge strategy" for that? Maybe we just have a flag or parameter for "rebuild" vs "combine"?

cjnolet · 2025-11-19T20:16:42Z

-        reinterpret_cast<uintptr_t>(_merge<int8_t>(res, *params, indices, num_indices));
+        reinterpret_cast<uintptr_t>(_merge<int8_t>(res, *params, indices, num_indices, filter));
    } else if (dtype.code == kDLUInt && dtype.bits == 8) {
      output_index->addr =


Ah- I was trying to figure out how you were going to support allowing a user to pass in the "index" to populate in the C layer while only have the option to return a newly created index in the C++ layer. Not opposed to this solution at all. Just good that we're not doing a copy!

cjnolet · 2025-11-19T20:18:06Z

+        auto out_layout          = raft::make_strided_layout(filtered_dataset.view().extents(),
+                                                    std::array<int64_t, 2>{stride, 1});
+
+        merged_index.update_dataset(handle, owning_t{std::move(filtered_dataset), out_layout});


We really need to prioritize having the indexes all accept "Dataset" instead of mdspans. This is going to get out of control :-(

cjnolet · 2025-11-19T20:21:33Z

 template <typename T, typename IdxT>
 struct index;
-struct merge_params;
+struct index_params;


These include files have gotten super lightweight now that they are only including prototypes and declarations. I'm not a huge fan of all of the CAGRA headers we're creating. Can we just consolidate this into the main header?

Also- for some reason, we have a header for cagra_optimize.hpp and I think this was a misunderstanding. Would you be able to copy that into the main cagra.hpp in this PR? It should be a straightforward copy/paste... just trying to keep the number of headers manageable. The whole benefit to no longer being header only is that the headers are super super lightweight.

benfred · 2025-11-19T22:24:55Z

+                                  0.006,
+                                  min_recall));
+
+      /* TODO: eval_distances doesn't work, potentially because of id translation mismatch


fwiw, This needs either fixed (by adding id translations) - or removed before this PR should be merged

copy-pr-bot · 2025-12-01T20:51:54Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

robertmaynard

Currently breaks ABI stability between 25.12 and 26.02

If we are planning on getting this into 26.02 as is we will need to bump our ABI. If we want to keep ABI compat we need a cuvsCagraMerge_v2

benfred · 2026-01-29T17:49:52Z

/merge

Add an optional filter argument to cagra::merge, so that we can compact the index to remove deleted items when merging. Also remove the 'LOGICAL_MERGE' strategy from the `cagra::merge` api, since this is only supported with the composite merge api. (The `cagra::merge` function ignore this value if set, and only does a logical merge). Authors: - Ben Frederickson (https://github.com/benfred) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Robert Maynard (https://github.com/robertmaynard) URL: rapidsai#1496

robertmaynard · 2026-02-26T16:11:31Z

@benfred We are going to have to open a new PR and introduce cuvsCagraMerge_v2 with this new signature as this breaks ABI between 26.02 and 26.04

benfred · 2026-02-26T16:31:49Z

@robertmaynard this change is in 26.02 - I messed up the merge on this pr, but we merged into 26.02 with #1755

robertmaynard · 2026-02-26T16:42:07Z

@robertmaynard this change is in 26.02 - I messed up the merge on this pr, but we merged into 26.02 with #1755

@benfred That is great news! Thanks

This adds a script that can check for changes that would break our C-ABI. It works by comparing two sets of header files: the `c/include` header files for the published ABI, as well as the `c/include` header files inside a new pull request. Each set of header files is parsed using libclang, and the differences between the old and new header files are examined for changes that would cause a breaking ABI change. Currently the following checks are made and flagged by this tool: * Functions that have been removed from the C-ABI * Functions that have extra parameters added * Functions that have parameters removed * Functions that have the type of any parameter changed * Structs that have been removed * Structs that have have members removed * Structs that have the types of members changed * Enum values that have been removed * Enum values that have their value changed This is just currently a POC of using libclang to flag breaking c-abi changes - and is meant as an alternative to tools that require debug symbols for detecting abi breaks from the compiled library, and isn't ready to merge just yet. To fully get this integrated, we will need to store the headers for the stable c-abi to use as a baseline somewhere, and then download the stable c-abi headers on each PR and use them to run in a GHA workflow to flag breaking changes. When run on a branch that has a breaking ABI change (#1496 which has a number of breaking c-abi changes), the output is: ``` $ ./check_c_abi.py ~/code/cuvs_stable_abi/c/include ~/code/cuvs/c/include Error: Function has been removed. Symbol cuvsCagraMergeParamsCreate from /home/ben/code/cuvs_stable_abi/c/include/cuvs/neighbors/cagra.h:584 Error: Function has been removed. Symbol cuvsCagraMergeParamsDestroy from /home/ben/code/cuvs_stable_abi/c/include/cuvs/neighbors/cagra.h:587 Error: Function has a changed argument type 'cuvsCagraMergeParams_t' to 'cuvsCagraIndexParams_t for argument 'params'. Symbol cuvsCagraMerge from /home/ben/code/cuvs/c/include/cuvs/neighbors/cagra.h:882 Error: Function has a changed argument type 'cuvsCagraIndex_t' to 'cuvsFilter for argument 'output_index'. Symbol cuvsCagraMerge from /home/ben/code/cuvs/c/include/cuvs/neighbors/cagra.h:882 Error: Function has a new argument 'cuvsCagraIndex_t output_index'. Symbol cuvsCagraMerge from /home/ben/code/cuvs/c/include/cuvs/neighbors/cagra.h:882 Error: Struct has been removed. Symbol cuvsCagraMergeParams from /home/ben/code/cuvs_stable_abi/c/include/cuvs/neighbors/cagra.h:576 ``` Also, this script runs in < 100ms on my workstation, and we could potentially add this as a pre-commit hook closes #1739 Authors: - Ben Frederickson (https://github.com/benfred) - Mike Sarahan (https://github.com/msarahan) Approvers: - Mike Sarahan (https://github.com/msarahan) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #1749

This adds a script that can check for changes that would break our C-ABI. It works by comparing two sets of header files: the `c/include` header files for the published ABI, as well as the `c/include` header files inside a new pull request. Each set of header files is parsed using libclang, and the differences between the old and new header files are examined for changes that would cause a breaking ABI change. Currently the following checks are made and flagged by this tool: * Functions that have been removed from the C-ABI * Functions that have extra parameters added * Functions that have parameters removed * Functions that have the type of any parameter changed * Structs that have been removed * Structs that have have members removed * Structs that have the types of members changed * Enum values that have been removed * Enum values that have their value changed This is just currently a POC of using libclang to flag breaking c-abi changes - and is meant as an alternative to tools that require debug symbols for detecting abi breaks from the compiled library, and isn't ready to merge just yet. To fully get this integrated, we will need to store the headers for the stable c-abi to use as a baseline somewhere, and then download the stable c-abi headers on each PR and use them to run in a GHA workflow to flag breaking changes. When run on a branch that has a breaking ABI change (rapidsai#1496 which has a number of breaking c-abi changes), the output is: ``` $ ./check_c_abi.py ~/code/cuvs_stable_abi/c/include ~/code/cuvs/c/include Error: Function has been removed. Symbol cuvsCagraMergeParamsCreate from /home/ben/code/cuvs_stable_abi/c/include/cuvs/neighbors/cagra.h:584 Error: Function has been removed. Symbol cuvsCagraMergeParamsDestroy from /home/ben/code/cuvs_stable_abi/c/include/cuvs/neighbors/cagra.h:587 Error: Function has a changed argument type 'cuvsCagraMergeParams_t' to 'cuvsCagraIndexParams_t for argument 'params'. Symbol cuvsCagraMerge from /home/ben/code/cuvs/c/include/cuvs/neighbors/cagra.h:882 Error: Function has a changed argument type 'cuvsCagraIndex_t' to 'cuvsFilter for argument 'output_index'. Symbol cuvsCagraMerge from /home/ben/code/cuvs/c/include/cuvs/neighbors/cagra.h:882 Error: Function has a new argument 'cuvsCagraIndex_t output_index'. Symbol cuvsCagraMerge from /home/ben/code/cuvs/c/include/cuvs/neighbors/cagra.h:882 Error: Struct has been removed. Symbol cuvsCagraMergeParams from /home/ben/code/cuvs_stable_abi/c/include/cuvs/neighbors/cagra.h:576 ``` Also, this script runs in < 100ms on my workstation, and we could potentially add this as a pre-commit hook closes rapidsai#1739 Authors: - Ben Frederickson (https://github.com/benfred) - Mike Sarahan (https://github.com/msarahan) Approvers: - Mike Sarahan (https://github.com/msarahan) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: rapidsai#1749

Add filter for cagra::merge

c91e609

Add an optional filter argument to cagra::merge, so that we can compact the index to remove deleted items when merging

benfred self-assigned this Nov 5, 2025

github-project-automation Bot added this to Unstructured Data Processing Nov 5, 2025

github-project-automation Bot moved this to Todo in Unstructured Data Processing Nov 5, 2025

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Nov 19, 2025

cjnolet moved this from Todo to In Progress in Unstructured Data Processing Nov 19, 2025

cjnolet reviewed Nov 19, 2025

View reviewed changes

benfred commented Nov 19, 2025

View reviewed changes

benfred added 2 commits December 1, 2025 11:07

Merge branch 'release/25.12' into filtered_merge

6b18ce8

unittest fix

5987408

benfred changed the base branch from main to release/25.12 December 1, 2025 20:52

Merge branch 'release/25.12' into filtered_merge

c962fd6

benfred marked this pull request as ready for review December 1, 2025 20:53

benfred requested review from a team as code owners December 1, 2025 20:53

benfred added 3 commits December 1, 2025 17:37

fix ann-bench

6c614c2

fix java bindings

3f32d50

Merge branch 'main' into filtered_merge

ec75ff2

benfred requested review from a team as code owners January 6, 2026 00:45

benfred removed request for a team and gforsyth January 6, 2026 00:49

benfred added 5 commits January 6, 2026 10:10

update copyright dates

87e00c4

.

a1fa3b0

Merge branch 'main' into filtered_merge

7d77193

fix java help string

c51567b

Handle null params object being passed from java to cuvsCagraMerge

638ae68

cjnolet approved these changes Jan 9, 2026

View reviewed changes

cjnolet and others added 4 commits January 9, 2026 13:50

Merge branch 'main' into filtered_merge

48fc9f8

Merge branch 'main' into filtered_merge

abc4b85

Fix java cuvsCagraMerge

55f0ec0

Merge branch 'main' into filtered_merge

072e719

robertmaynard requested changes Jan 15, 2026

View reviewed changes

Merge branch 'main' into filtered_merge

e782348

benfred mentioned this pull request Jan 28, 2026

Add a script to check for breaking C ABI changes #1749

Merged

cjnolet changed the base branch from main to release/26.02 January 29, 2026 03:14

cjnolet changed the base branch from release/26.02 to main January 29, 2026 03:15

robertmaynard approved these changes Jan 29, 2026

View reviewed changes

rapids-bot Bot merged commit a883a25 into rapidsai:main Jan 29, 2026
180 of 193 checks passed

github-project-automation Bot moved this from In Progress to Done in Unstructured Data Processing Jan 29, 2026

benfred mentioned this pull request Jan 29, 2026

Add filter for cagra::merge #1755

Merged

benfred deleted the filtered_merge branch January 29, 2026 23:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filter for cagra::merge#1496

Add filter for cagra::merge#1496
rapids-bot[bot] merged 17 commits into
rapidsai:mainfrom
benfred:filtered_merge

benfred commented Nov 5, 2025

Uh oh!

copy-pr-bot Bot commented Nov 5, 2025

Uh oh!

cjnolet Nov 19, 2025

Uh oh!

cjnolet Nov 19, 2025

Uh oh!

cjnolet Nov 19, 2025

Uh oh!

cjnolet Nov 19, 2025

Uh oh!

benfred Nov 19, 2025

Uh oh!

copy-pr-bot Bot commented Dec 1, 2025

Uh oh!

robertmaynard left a comment

Uh oh!

benfred commented Jan 29, 2026

Uh oh!

Uh oh!

robertmaynard commented Feb 26, 2026

Uh oh!

benfred commented Feb 26, 2026

Uh oh!

robertmaynard commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

benfred commented Nov 5, 2025

Uh oh!

copy-pr-bot Bot commented Nov 5, 2025

Uh oh!

cjnolet Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

cjnolet Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

cjnolet Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

cjnolet Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

benfred Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

copy-pr-bot Bot commented Dec 1, 2025

Uh oh!

robertmaynard left a comment

Choose a reason for hiding this comment

Uh oh!

benfred commented Jan 29, 2026

Uh oh!

Uh oh!

robertmaynard commented Feb 26, 2026

Uh oh!

benfred commented Feb 26, 2026

Uh oh!

robertmaynard commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants