[WIP] feature : add zero copy support for vector index by foxspy · Pull Request #1032 · zilliztech/knowhere

foxspy · 2025-01-15T12:28:05Z

issue: #1031

sre-ci-robot · 2025-01-15T12:28:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: foxspy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [foxspy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2025-01-15T12:28:41Z

@foxspy 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

If you're fixing a bug, label it as kind/bug.
For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

codecov · 2025-01-15T15:31:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.81%. Comparing base (3c46f4c) to head (771d36e).
Report is 344 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #1032       +/-   ##
=========================================
+ Coverage      0   72.81%   +72.81%     
=========================================
  Files         0       82       +82     
  Lines         0     7529     +7529     
=========================================
+ Hits          0     5482     +5482     
- Misses        0     2047     +2047

see 82 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

alexanderguzhva

one topic to discuss. Otherwise, lgtm

alexanderguzhva · 2025-01-15T22:54:53Z

thirdparty/faiss/faiss/Index.h

    MetricType metric_type;
    float metric_arg; ///< argument of the metric type

+    std::shared_ptr<MmappedFileMappingOwner> mmap_owner;


this is a very tricky point. Although, I understand why this was introduced, I'd like to avoid adding it here. Basically, the approach is similar to a std::string_view, which does not have a business to maintain the memory. Similar, here: It is not Index's instance business to maintain the view.

I'm open for discussions.

@foxspy any comments on this? Basically, I doubt that we'll able to introduce this variable in vanilla Faiss when porting back

alexanderguzhva · 2025-01-15T22:56:19Z

thirdparty/faiss/faiss/impl/index_read.cpp

+        }
+    }
+
+    ZeroCopyIOReader* zr = dynamic_cast<ZeroCopyIOReader*>(f);


this is a todo for the future: basically, a reader need to instantiate an object. But this is a later business, because it needs to be properly synchronized with the vanilla faiss first.
So, let's keep the change in the way you proposed it.

Yes, so I put the PR up first; zero copy is needed here mainly for

Optimizing memory expansion caused by copying during loading

The memory will be managed uniformly through Eyrie later, and the index does not hold data, but only views

The implementation of mmap and zero copy can be unified (mmap externally first, load through zero copy)
If you have any ideas about this, please feel free to communicate; has faiss considered supporting this mode?

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>

alexanderguzhva · 2025-02-16T23:12:24Z

src/index/flat/flat.cc


-        MemoryIOReader reader(binary->data.get(), binary->size);
+        int io_flags = faiss::IO_FLAG_ZERO_COPY;
+        faiss::ZeroCopyIOReader reader(binary->data.get(), binary->size);


Such an approach won't lead to a memory leak, correct?

The binary will be held by the index; so there will be no memory leak

alexanderguzhva · 2025-02-16T23:14:27Z

thirdparty/faiss/faiss/Index.h

    MetricType metric_type;
    float metric_arg; ///< argument of the metric type

+    std::shared_ptr<MmappedFileMappingOwner> mmap_owner;


@foxspy any comments on this? Basically, I doubt that we'll able to introduce this variable in vanilla Faiss when porting back

alexanderguzhva · 2025-02-16T23:16:20Z

thirdparty/faiss/faiss/impl/zerocopy_io.cpp

+
+namespace faiss {
+
+ZeroCopyIOReader::ZeroCopyIOReader(uint8_t* data, size_t size) : data_(data), rp_(0), total_(size) {


A duplicate code, similar to src/io/memory_io.h

This is because Knowhere is a hub that is adapted to multiple indexes. For indexes other than faiss, it uses the memory_io.h. Faiss needs to remain independent zerocopy_io.cpp and not rely on related objects of knowhere.

foxspy · 2025-02-24T09:18:24Z

Zero copy may cause data misalignment, because data serialization in Faiss does not require alignment, which may cause performance degradation or even crashes in some scenarios. @alexanderguzhva Do you have any suggestions for this?

alexanderguzhva · 2025-02-24T13:19:24Z

Zero copy may cause data misalignment, because data serialization in Faiss does not require alignment, which may cause performance degradation or even crashes in some scenarios. @alexanderguzhva Do you have any suggestions for this?

I'm not sure I follow. Any examples?
As far as was able to understand your code, it is about avoiding memory duplication, so I'm not sure where a data misalignment and crash might occur

alexanderguzhva · 2025-02-24T13:46:42Z

I mean that this PR totally makes sense, with the exception of adding a memory owner directly to the faiss::Index object

alexanderguzhva · 2025-02-24T14:15:08Z

thirdparty/faiss/faiss/impl/index_read.cpp

+            return;
+        }
+    }
+    target.resize(size);


this particular line 118 is not needed

foxspy · 2025-02-25T02:42:36Z

Zero copy may cause data misalignment, because data serialization in Faiss does not require alignment, which may cause performance degradation or even crashes in some scenarios. @alexanderguzhva Do you have any suggestions for this?

I'm not sure I follow. Any examples? As far as was able to understand your code, it is about avoiding memory duplication, so I'm not sure where a data misalignment and crash might occur

A simple example is, if https://github.com/zilliztech/knowhere/pull/1032/files#diff-d05f16d9b1473d49c536582f9433cca57d6a2909a5a729dec54c383f598b57c4R119 assign 0x5ffffff3 to float *, it will cause misalignment of memory access. Although I have tried it on x64 and arm machines, it will not cause a crash, but I know that this will generally lead to performance degradation (may cause multiple memory accesses); some older machines may also have strange problems; for example, scann also has SIMD requirements for alignment https://github.com/zilliztech/knowhere/blob/main/thirdparty/faiss/faiss/utils/AlignedTable.h#L115, which will cause memory expansion (duplicated binary + memory copy) if not handled

Summary: This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces. The root underlying idea is that we replace certain `std::vector<>` containers with a custom `faiss::MaybeOwnedVector<>` container, which may behave either as `std::vector<>`, or as a view of a certain pointer / descriptor. We don't replace all the instances of `std::vector<>`, but the largest ones. This change affects `IndexFlatCodes`-based and `IndexHNSW` CPU indices. (done) alter IVF lists as well. (done) alter binary indices as well. Memory-mapped index works like this: ```C++ std::unique_ptr<faiss::Index> index_mm( faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC)); ``` In theory, it should be ready to be used from Python. All the descriptor management should be working. Zero-copy index works like this: ```C++ #include <faiss/impl/zerocopy_io.h> faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size()); std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader)); ``` All the pointer management for `faiss::ZeroCopyIOReader` should be handled manually. I'm not sure how to plug this into Python yet, maybe, some ref-counting is required. (done) some refactoring Pull Request resolved: #4199 Reviewed By: mengdilin Differential Revision: D69972250 Pulled By: mdouze fbshipit-source-id: 98a3f94d6884814873d3534ee25f960892ef1076

Summary: This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces. The root underlying idea is that we replace certain `std::vector<>` containers with a custom `faiss::MaybeOwnedVector<>` container, which may behave either as `std::vector<>`, or as a view of a certain pointer / descriptor. We don't replace all the instances of `std::vector<>`, but the largest ones. This change affects `IndexFlatCodes`-based and `IndexHNSW` CPU indices. (done) alter IVF lists as well. (done) alter binary indices as well. Memory-mapped index works like this: ```C++ std::unique_ptr<faiss::Index> index_mm( faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC)); ``` In theory, it should be ready to be used from Python. All the descriptor management should be working. Zero-copy index works like this: ```C++ #include <faiss/impl/zerocopy_io.h> faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size()); std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader)); ``` All the pointer management for `faiss::ZeroCopyIOReader` should be handled manually. I'm not sure how to plug this into Python yet, maybe, some ref-counting is required. (done) some refactoring Pull Request resolved: facebookresearch/faiss#4199 Reviewed By: mengdilin Differential Revision: D69972250 Pulled By: mdouze fbshipit-source-id: 98a3f94d6884814873d3534ee25f960892ef1076

Summary: This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces. The root underlying idea is that we replace certain `std::vector<>` containers with a custom `faiss::MaybeOwnedVector<>` container, which may behave either as `std::vector<>`, or as a view of a certain pointer / descriptor. We don't replace all the instances of `std::vector<>`, but the largest ones. This change affects `IndexFlatCodes`-based and `IndexHNSW` CPU indices. (done) alter IVF lists as well. (done) alter binary indices as well. Memory-mapped index works like this: ```C++ std::unique_ptr<faiss::Index> index_mm( faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC)); ``` In theory, it should be ready to be used from Python. All the descriptor management should be working. Zero-copy index works like this: ```C++ #include <faiss/impl/zerocopy_io.h> faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size()); std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader)); ``` All the pointer management for `faiss::ZeroCopyIOReader` should be handled manually. I'm not sure how to plug this into Python yet, maybe, some ref-counting is required. (done) some refactoring Pull Request resolved: facebookresearch#4199 Reviewed By: mengdilin Differential Revision: D69972250 Pulled By: mdouze fbshipit-source-id: 98a3f94d6884814873d3534ee25f960892ef1076

sre-ci-robot requested review from PwzXxm and cqy123456 January 15, 2025 12:28

sre-ci-robot added the approved label Jan 15, 2025

foxspy changed the title ~~feature : add zero copy support for vector index~~ [WIP] feature : add zero copy support for vector index Jan 15, 2025

sre-ci-robot added do-not-merge/work-in-progress size/XL labels Jan 15, 2025

mergify bot added the dco-passed label Jan 15, 2025

mergify bot added the do-not-merge/missing-related-issue label Jan 15, 2025

foxspy force-pushed the zero_copy_support branch 3 times, most recently from 5385cae to dc5a6ac Compare January 15, 2025 13:03

alexanderguzhva reviewed Jan 15, 2025

View reviewed changes

foxspy force-pushed the zero_copy_support branch from dc5a6ac to 7dc2032 Compare January 16, 2025 02:39

foxspy force-pushed the zero_copy_support branch 2 times, most recently from b58a68a to 91cfcde Compare February 6, 2025 08:06

add zero copy support for faiss

771d36e

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>

foxspy force-pushed the zero_copy_support branch from 91cfcde to 771d36e Compare February 6, 2025 08:09

mergify bot added the ci-passed label Feb 6, 2025

alexanderguzhva reviewed Feb 16, 2025

View reviewed changes

alexanderguzhva mentioned this pull request Feb 19, 2025

Memory-mapping and Zero-copy deserializers facebookresearch/faiss#4199

Closed

foxspy closed this Feb 24, 2025

foxspy reopened this Feb 24, 2025

mergify bot removed the ci-passed label Feb 24, 2025

alexanderguzhva reviewed Feb 24, 2025

View reviewed changes

thirdparty/faiss/faiss/impl/index_read.cpp

return;

}

}

target.resize(size);

Copy link

Collaborator

alexanderguzhva Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this particular line 118 is not needed

github-actions bot added the stale label Mar 28, 2025

github-actions bot closed this Apr 4, 2025


		namespace faiss {

		ZeroCopyIOReader::ZeroCopyIOReader(uint8_t* data, size_t size) : data_(data), rp_(0), total_(size) {

Conversation

foxspy commented Jan 15, 2025

Uh oh!

sre-ci-robot commented Jan 15, 2025

Uh oh!

mergify bot commented Jan 15, 2025

Uh oh!

codecov bot commented Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alexanderguzhva left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foxspy Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foxspy commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexanderguzhva commented Feb 24, 2025

Uh oh!

alexanderguzhva commented Feb 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foxspy commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Jan 15, 2025 •

edited

Loading

foxspy Jan 16, 2025 •

edited

Loading

foxspy commented Feb 24, 2025 •

edited

Loading