How to control the total number of merging threads when vector data merging easily leads to memory overflow and high CPU cost

[weizijun]

Description

When there are many shards to merge, vector data merging can easily lead to memory overflow and high CPU cost.
The index.merge.scheduler.max_thread_count parameter can't control the merge thread count, it only pause the writeByte by MergeRateLimiter when the merge thread is bigger then max_thread_count.
But OnHeapHnswGraph has been built during the pause phase, and it will take up so much memory that the Java heap is not enough.
This problem can easily be caused when a datanode with a 32G heap size holds 2-3TB of vector documents(with bbq, the node can contain these data).
The PR #14527 can reduce the heap size, but it don't solve the problem totally.
Is there any solution to this problem?

[jpountz]

It doesn't solve the whole problem, but #13883 would at least help better keep the total number of merge threads across shards under control.

[karlney]

We likely have the same problem in our env (Elasticsearch 8.18.0)
We have ~1B vectors on each node (256g RAM, 64 cores, 3TB disk) . The data is in approx 100 shards (lucene indices) and a few 100s of segments.
We got heap OOM issues during merges with a 70g heap (and ongoing indexing)
Now we are trying with a 100g heap to see if that is enough for our data.

A preferable solution would be to have some max settings on total heap used for merges of the HNSW graphs

[weizijun]

We likely have the same problem in our env (Elasticsearch 8.18.0)

This PR : #14527 can significantly help reduce heap memory.