[Usage]: Spread two big LLMs across four GPUs

[fozziethebeat]

Your current environment

This is about API usage rather than an environment issue.

How would you like to use vllm

I want to run batch inference using two models in the same process and spread them across distinct GPUs.

Here's an example code snippet i've been trying to figure out how to get correct:

from vllm import LLM

llm1 = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=2
)
llm2 = LLM(
    model="CohereForAI/c4ai-command-r-v01",
    tensor_parallel_size=2,
)

In this scenario, if I have 4 A100 80Gs on a single node, I should be able to fit model one on GPUs 1 and 2, then fit model two on GPUs 3 and 4. I was hoping that Ray's backend would get this right but instead this doesn't work. I typically run out of vRAM.

Is there a way to configure two LLM instances such that they work on distinct subsets of my available GPUs all while running the same process? Do I need to use device right?

[orellavie1212]

I'd say first set only cuda 0 and 1 available and load llm 1, and then cuda 2 and 3 only and load llm2, and then just set all cudas available and keep going with that.

[fozziethebeat]

Can you elaborate this suggestion? The issue we saw when doing something like the code I wrote was ray runtime engine stalling when initializing a second time

[youkaichao]

Why not running them in two processes with different CUDA_VISIBLE_DEVICES?

[fozziethebeat]

Yes that is my backup plan if I need to. I was just hoping I'd be able to do this without sub-processes

[youkaichao]

No, you don't want to do it in one process. It gives you no speedup compared with two processes. And it might be prone to bugs in distributed inference.

[stas00]

xref a similar discussion: #6155

[m1chae11u]

Hey youkaichao, I'm facing the same problem where I have 4 A100 80Gs on a single node, and my script needs to be able to run two different models in one process. Would this be possible? I added you on Linkedin, maybe we can chat further there?