[Usage]: if I want to run a 34B model，like yi-34B-chat,how can I use multi GPU,I just have A100 40G

[hellostronger]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

[hellostronger]

i have tried engine_use_ray = True in AsyncEngineArgs or CUDA_VISIBLE_DEVICES = 0,1 but it still does not work ,GPU0 OOM

[kir1to00]

use tensor-parallel-size

[hellostronger]

thk, the problem that GPU0 OOM is solved, but new problem comes ,it shows " ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 33024 " ,anybody can load the yi-34B-chat with lora successfuly? hoping your suggestion

[kir1to00]

thk, the problem that GPU0 OOM is solved, but new problem comes ,it shows " ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 33024 " ,anybody can load the yi-34B-chat with lora successfuly? hoping your suggestioncu

maybe use max-model-len?

[jeejeelee]

thk, the problem that GPU0 OOM is solved, but new problem comes ,it shows " ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 33024 " ,anybody can load the yi-34B-chat with lora successfuly? hoping your suggestion

"ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 33024" has already been resolved by #4015. You can build the source from the main branch or wait for the 0.4.1 release. After resolving this issue, Yi-34 can load LoRA successfully.

[riverind]

use tensor-parallel-size

however，use tensor-parallel-size=2 is also oom，H800-80G*2

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!