Removing from IndexIVFFlat with a IndexIDMap2 wrapper fails

[dbiton]

Hello,
In Python 3.12.5, with faiss-cpu 1.11.0.post1, I have the following scheme:

# Generate some embeds with SBERT...
dim = 384
count_cells = 10 
coarse_quantizer = faiss.IndexFlatL2(dim)
ivf_index = faiss.IndexIVFFlat(coarse_quantizer, dim, nlist)
train_embeds = embeds[:5000]
ivf_index.train(train_embeds)
index = faiss.IndexIDMap2(ivf_index)

Afterwards, I have a loop where I add a few embeds each time like this:

# embeds - a list of sbert embeds of dim 384, embeds_ids - their unique ids
self.index.add_with_ids(embeds, embeds_ids)

The ids are for sure unique - I use the index of the embeds, and iterate over the embeds.
Once the index reaches a certain size, I start removing embeds by ids - not in a sliding window fashion.

I added some safety code as advised by an LLM, although note that my code works fine with a naive index, i.e. faiss.IndexIDMap2(faiss.IndexFlatL2(dim)), by just calling remove_ids with the array of ids.

ids64 = np.ascontiguousarray(np.asarray(embeds_ids, dtype=np.int64))

sel = faiss.IDSelectorBatch(ids64.size, faiss.swig_ptr(ids64))
before = self.index.ntotal
removed = int(self.index.remove_ids(sel))
after = self.index.ntotal

assert after == before - len(embeds_ids), (before, after)
return removed

This function can remove a single ID 3 times, passing the assertion, and than gets the following assertion failure (btw - throwing an exception would be nice, unless there's some good reason to use an assertion instead).

Faiss assertion 'j == index->ntotal' failed in virtual size_t __cdecl faiss::IndexIDMapTemplate<faiss::Index>::remove_ids(const IDSelector &) [IndexT = faiss::Index] at D:\a\faiss-wheels\faiss-wheels\faiss\faiss\IndexIDMap.cpp:197

I've been troubleshooting for quite some time, and I suspect there might be a bug - since I think (hope) my usage is fine.

[dbiton]

I've installed faiss-cpu using:
python -m pip install faiss-cpu

To make sure there's nothing weird related to GPU drivers / experimental versions.

Using Window 11.

Made sure with:
faiss.vector_to_array(self.index.id_map)

That the item id requested for deletion is present in the index's id_map.

[dbiton]

Yes, the checklist would be useful. I can post more details about the bug and check when the invariant breaks

[dbiton]

I think @onestardao is a chat bot of some sort, since the link is unrelated.
And also in hindsight, the comment is pretty odd.
Anyhow, seems like the bug is not resolved.

[dbiton]

Looked into tests here:
https://github.com/facebookresearch/faiss/blob/main/tests/test_index_composite.py#L25

Seems like IVF does not need to be wrapped with an IndexIDMap2, which was my problem.
I think the interface can be made clearer, maybe with more explicit assertions or exceptions.

[mnorris11]

That user is indeed some chatbot. We banned them.

[mnorris11]

Our oncall will try to check into the issue soon

[mnorris11]

Sorry for the delay, fell through the cracks.
How did you install Faiss?
I cannot reproduce. Below is a self contained example using SyntheticDataset:

import faiss
import numpy as np
from faiss.contrib.datasets import SyntheticDataset


d = 128
nt = 5_000
nb = 10_000
nq = 100
nlist = 1024
rs = np.random.RandomState(1234)

ds = SyntheticDataset(d, nt, nb, nq)

coarse_quantizer = faiss.IndexFlatL2(d)
ivf_index = faiss.IndexIVFFlat(coarse_quantizer, d, nlist)
ivf_index.train(ds.get_train())
index = faiss.IndexIDMap2(ivf_index)
# random unique ids 0 to 1 million
ids = np.random.choice(np.arange(1_000_000, dtype=np.int32), size=nb, replace=False)

index.add_with_ids(ds.get_database(), ids)


# pyre-ignore
def remove(index, ids) -> None:
    ids_to_remove = ids[0:10]
    ids_to_remove = np.ascontiguousarray(ids_to_remove, dtype="int64")
    print(f"ids_to_remove: {ids_to_remove}")
    sel = faiss.IDSelectorBatch(ids_to_remove.size, faiss.swig_ptr(ids_to_remove))
    before = index.ntotal
    removed = int(index.remove_ids(sel))
    after = index.ntotal
    print(f"before: {before}, after: {after}, removed: {removed}")


remove(index, ids)
remove(index, ids)
remove(index, ids)
remove(index, ids)
remove(index, ids)

Sample output:

ids_to_remove: [250961 179372 884886 515364 383434 747379  61367 713033 256264 638369]
before: 10000, after: 9990, removed: 10
ids_to_remove: [250961 179372 884886 515364 383434 747379  61367 713033 256264 638369]
before: 9990, after: 9990, removed: 0
ids_to_remove: [250961 179372 884886 515364 383434 747379  61367 713033 256264 638369]
before: 9990, after: 9990, removed: 0
ids_to_remove: [250961 179372 884886 515364 383434 747379  61367 713033 256264 638369]
before: 9990, after: 9990, removed: 0
ids_to_remove: [250961 179372 884886 515364 383434 747379  61367 713033 256264 638369]
before: 9990, after: 9990, removed: 0