[Question] Difference between two different ways to train multi-GPU index

[aengle1429]

Summary

I see two different examples of ways to train IVF ANN on multiple GPUs. I'm wondering what is the difference between the two.

Platform

OS:

Faiss version:

Installed from:

Faiss compilation options:

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

The first example is from the benchmarks

https://github.com/facebookresearch/faiss/blob/master/benchs/bench_gpu_sift1m.py

index = faiss.index_factory(d, "IVF4096,PQ64")
co = faiss.GpuClonerOptions()
co.useFloat16 = True
index = faiss.index_cpu_to_gpu(res, 0, index, co)
index.train(xt)

and the second from a popular gist https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6

index2 = faiss.index_factory(128, "PCA64,IVF16384_HNSW32,Flat")
index_ivf = faiss.extract_index_ivf(index2)
clustering_index = faiss.index_cpu_to_all_gpus(faiss.IndexFlatL2(64))
index_ivf.clustering_index = clustering_index
index2.train(xt)

I'm wondering why we call index.train on the return value of index_cpu_to_gpu in the first example and call index.train directly on the index_factory in the second example.

[mdouze]

The first example builds an index and moves it to GPU prior to training.
The second example builds and index and trains it, using a GPU index for clustering. index2 is not on GPU.

[aengle1429]

Oh I see, I did not realize that index2 was not on the GPU.

I noticed that some indices are not yet possible to build on GPUs; for example, it seems using method 1 with "PCA64,L2norm,IVF16384_HNSW32,Flat" is not yet possible, but it is with method 2.

Would a general rule of thumb to be use method 1 whenever possible, and fall back to method 2 if not? If not, is there some other guidance one can follow? Thank you!