RAM usage during training of the IVF65536,PQ64 index

[XingxingZhang]

Summary

I intended to train an "IVF65536,PQ64" index with 20 million 1024d vectors on a server with 400G RAM and 4 x V100 (16G) GPUs.
It ran out of memory. I know these vectors already take ~80G RAM even without doing anything.

My questions are

1. How much RAM is needed to train such an index (i.e., "IVF65536,PQ64") with 20M 1024d vectors?
2. Are there ways to optimize the RAM usage?

Thanks

Platform

OS: Ubuntu 18.04.6 LTS

Faiss version: 1.7.2

Installed from: pip install faiss-gpu

Faiss compilation options:

Running on:

• GPU

Interface:

• Python

Reproduction instructions

dim = 1024
index_type = "IVF65536,PQ64"
index = faiss.index_factory(dim, index_type)

# move to gpus
co = faiss.GpuMultipleClonerOptions()
co.useFloat16 = True
gpu_index = faiss.index_cpu_to_all_gpus(index, co)

# train
# x.shape = (20*1000*1000, 1024)
gpu_index.train(x)
# Error Message from OS: "Killed PID"

[mdouze]

2 remarks:

• since you do a PQ64, then it is better to reduce the dimension of the vectors to ~256D with "OPQ64_256,IVF65536,PQ64"

• to train 65536 centroids, 3-4 M training points are enough (=50*65536)

[XingxingZhang]

Thanks for the very useful remarks, @mdouze!

Both the "OPQ64_256,IVF65536,PQ64" and "OPQ64_512,IVF65536,PQ64" index with 20M vectors were trained successfully on my machine.

So which one do you recommend? If I understand correctly, the rotation matrix of "OPQ64_512" index (1024 x 512) is larger than the matrix of "OPQ64_256" index (1024 x 256). Will larger rotation matrix (more trainable parameters) lead to better kNN search accuracy?

Thanks