About training, history data and daily updated data.

[splinter21]

Running on:

• [√ ] CPU
• GPU

Interface:

• C++
• [√ ] Python

First we add history data (about 150kk) into the index, and randomly select some to train the index.
And then we have newly updated data everyday (about 400k/day).
How should I train the index everyday? Should I only randomly select {the daily updated data} for training, or randomly select {the daily updated data and all history data}for training?

[beauby]

If the distribution of your data does not change, there is no need to re-train the index.

[splinter21]

If the distribution of your data does not change, there is no need to re-train the index.

If there is some change of the distribution everday, and I need train the index everyday, should I sample the training data from {all history data and the daily updated data} instead of training only from the sample of {the daily updated data}?

[mdouze]

We can't give a generic answer to this question: it depends by how much the data distribution changes!