[FEA] CAGRA ACE Follow up tracker

[tfeher]

#1404 implements a new build method for CAGRA indices. Here we list follow up issues that are out of scope of the original PR.

• Default build config should select ACE build when regular build would go OOM
• cuvs-bench param parser to accept graph_build_algo: "ACE" (and set other params automatically if needed)
• Set ace_params.npartitions default value 0 for auto select?
• do we need to expose build_knn_graph for ivf_pq method only?
• does get_device_for_address (used in src/neighbors/cagra.cuh) work as intended on systems with AST or HMM? #1721
• hnsw::from_cagra does only serialize the index when it is on_disk. Is this the right behavior?
• python API tests
• [BUG] sample_rows + balanced k-means leads to imbalanced clusters on BIGANN 1B #1461
• writing numpy headers for files is repetitive in cagra_build.cuh is repetitive, create better helpers.
• File management: shall we use file descriptors instead of path? Who shall delete temporary files? (Do we handle it correctly if there was a crash that left files on disk?) Which files do we consider temporary? Does all user need the (reordered) dataset?

[julianmi]

I have removed the outdated Java and Python interfaces. If Java and Python support for ACE builds is needed, we will need to implement them based on the final APIs and add tests.

Edit Nov. 6: I have added interfaces that match the C interface and respective tests.

[julianmi]

hnsw::from_cagra does only serialize the index when it is on_disk. Is this the right behavior?

I propose to move to hnsw::build in #1597. This would build a CAGRA index on disk using CAGRA ACE followed by serializing an HNSW index to disk. A user would deserialize this index to make it searchable. However, users might expect hnsw::build to be searchable. We could rename the API to something more explicit such as hnsw::build_and_serialize. @tfeher What do you think?

[julianmi]

Default build config should select ACE build when regular build would go OOM

We might want to wait for the ACE implementation to mature before switching automatically.

cuvs-bench param parser to accept graph_build_algo: "ACE" (and set other params automatically if needed)

This was done in the original PR #1404.

Set ace_params.npartitions default value 0 for auto select?

This is addressed in #1603.

writing numpy headers for files is repetitive in cagra_build.cuh is repetitive, create better helpers.

This is addressed in #1645.

File management: shall we use file descriptors instead of path? Who shall delete temporary files? (Do we handle it correctly if there was a crash that left files on disk?) Which files do we consider temporary? Does all user need the (reordered) dataset?

We switched to file handles in the original PR #1404. The temporary files are cleaned up in case of a crash. I believe the reordered dataset is always necessary to make use of the search graph built since it uses the reordered index space.

[tfeher]

do we need to expose build_knn_graph for ivf_pq method only?
We select automatically the KNN graph build method and its config. We do not plan to expose fine tuning parameters

[tfeher]

We have additional issues with the two remaining open point (those are more general than ACE). Closing this tracker.