[Bugfix] Specify device when loading LoRA and embedding tensors #7129
Conversation
- Add map_location="device" when loading LoRA tensors from .bin files - Add map_location="device" when loading new embeddings from .bin files
jischein
changed the title
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
jischein
changed the title
github-actions
bot
added
the
ready
|
friendly bump! @youkaichao (believe you had reviewed the previous push to fix this) |
vllm/lora/models.py
Outdated
| f" but received {unexpected_modules}." | ||
| f" Please verify that the loaded LoRA module is correct") | ||
| tensors = torch.load(lora_bin_file_path) | ||
| tensors = torch.load(lora_bin_file_path, map_location="device") |
There was a problem hiding this comment.
did you test it? I don't think "device" works. this is a string.
youkaichao
left a comment
youkaichao
left a comment
There was a problem hiding this comment.
thanks for the contribution!
…-project#7129) Co-authored-by: Jacob Schein <[email protected]> Signed-off-by: Alvant <[email protected]>
…-project#7129) Co-authored-by: Jacob Schein <[email protected]> Signed-off-by: LeiWang1999 <[email protected]>
2 participants
[Bugfix] Assign device when loading LoRA modules from file
Fixes #3374. Note — existing PR to address this issue got stale; so bumping with some light updates.
This PR addresses the issue of CUDA device mismatch when loading LoRA modules from files. It fixes the
Attempting to deserialize object on CUDA device X but torch.cuda.device_count() is Yerror by explicitly specifying the device during tensor loading.Changes:
map_location="device"when loading LoRA tensors from .bin filesmap_location="device"when loading new embeddings from .bin files