Fast Inference with vLLM for VLMs #2975

Datta0 · 2025-07-15T17:59:25Z

Notebooks:

Qwen 2.5 7B VL on Colab T4
Gemma 3 4B on Colab L4 (vllm gemma doesn't support FP16)

Editing icon sizes

Updating icon sizes

* add deepseek v3 * add deepseek r1 base * add deepseek r1 zero * add deepseek distill llama * add deepseek distill models * remove redundant code when constructing model names * add mistral small to registry * rename model registration methods * rename deepseek registration methods * refactor naming for mistral and phi * add global register models * refactor model registration tests for new registry apis * add model search method * remove deprecated registration api * add quant type test * add registry readme * make llama registration more specific * clear registry when executing individual model registration file * more registry readme updates * Update _auto_install.py * Llama4 * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Synthetic data * Update mapper.py * Xet and Synthetic * Update synthetic.py * Update loader.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update pyproject.toml * Delete .gitignore * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update _utils.py * Update pyproject.toml * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update synthetic.py * Update chat_templates.py * Seasame force float16 / float32 * Fix Seasame * Update loader.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * is_multimodal * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update vision.py * Update vision.py * Update vision.py * UNSLOTH_DISABLE_STATIC_GENERATION * Update vision.py * Auto vision detection * Sesame * Whisper * Update loader.py * Update loader.py * Update loader.py * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update vision.py * Update loader.py * Update loader.py * Update loader.py * Update loader.py * Update _utils.py * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl.py * Update rl.py * Update rl.py * logging * Update pyproject.toml * Update rl.py * versioning * Update rl.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * Update rl.py * Update rl_replacements.py * Update rl_replacements.py * logits / temperature * Update rl_replacements.py * Update pyproject.toml * Update rl_replacements.py * Update rl_replacements.py * Debugging only * Update llama.py * Update llama.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Update rl_replacements.py * Generic efficient GRPO * Update rl_replacements.py * Update rl_replacements.py * Remove debugging * Update rl_replacements.py * Update rl_replacements.py * Update vision.py * Update llama.py * Update rl_replacements.py * versioning * Update _utils.py * Update vision.py * Update mapper.py * Update loader.py * Update mapper.py * Update vision.py * Update loader.py * Update vision.py * Update loader.py * Update _utils.py * Update vision.py * gradient checkpointing * Gemma 3N fixes * Update loader.py * Versioning * Gemma 3N fixes * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Fix setup.py * setup.py * Prints * Update setup.py * Update setup.py * Update setup.py * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update vision.py * Update vision.py * Update pyproject.toml * Update vision.py * Update _utils.py * Update __init__.py * Update __init__.py --------- Co-authored-by: jeromeku <[email protected]> Co-authored-by: Michael Han <[email protected]>

…othai#2912)

…slothai#2911)

* fix for casual mask * use un_casual in sdpa * add missing mask * fix for type

…falcon h1 (unslothai#2913)

* Move tensors to right devices * fix multi gpu for non mistral models * multi GPU RoPE for gemma2 * Finish up multi GPU inference * Make multiGPU rope a list * Remove unnecessary transfer to CPU * Remove unnecessary move to CPU * Donot move inputs to device yet will be handled separately in another PR * Move inputs to appropriate decoder device * Make device count global variable * Cleanup RoPE device code * Fixup num_gpu to device count * Cleanup device counts * Use device index for RoPE get_cache * Donot typecast * Use tuple instead of list for tensors. Use device index directly * fixup move to device logic

unsloth/models/_utils.py

* Fix mamba * Update loader.py * Update vision.py * Update loader.py * Filter vLLM standby logs (unslothai#3131) * filter vLLM standby logs * safeguard standby logger patch * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py --------- Co-authored-by: Daniel Han <[email protected]> * Update loader.py * Add scaler * Update llama.py * Update _utils.py * Versioning * GPT OSS fix * GPT OSS fix * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Versioning * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Upcast norms * Update loader.py * Update vision.py * Upcast layernorms * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update save.py * Update rl.py * Update pyproject.toml * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Torch 2.8 * Update rl_replacements.py --------- Co-authored-by: Datta Nimmaturi <[email protected]>

* Fix mamba * Update loader.py * Update vision.py * Update loader.py * Filter vLLM standby logs (unslothai#3131) * filter vLLM standby logs * safeguard standby logger patch * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py * Update unsloth/models/_utils.py --------- Co-authored-by: Daniel Han <[email protected]> * Update loader.py * Add scaler * Update llama.py * Update _utils.py * Versioning * GPT OSS fix * GPT OSS fix * Update loader.py * Update vision.py * Update vision.py * Update loader.py * Update vision.py * Update vision.py * Update llama.py * Update llama.py * Update llama.py * Versioning * Update mapper.py * Update vision.py * Update vision.py * Update vision.py * Upcast norms * Update loader.py * Update vision.py * Upcast layernorms * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update llama.py * Update save.py * Update rl.py * Update pyproject.toml * Update rl.py * Update rl_replacements.py * Update rl.py * Update rl.py * Update rl.py * Update _utils.py * Update __init__.py * Torch 2.8 * Update rl_replacements.py * Update loader.py * UNSLOTH_ENABLE_CCE * Fix * Update loader.py * Update loader.py * Update __init__.py * Update __init__.py * Update __init__.py * Update __init__.py * Import fixes * Update loader.py * Fix aimv2 issue * Update loader.py * Update import_fixes.py * Update import_fixes.py * Update loader.py * Update loader.py * Update loader.py * Upgrade * Update loader.py * Update loader.py * Update loader.py * Update loader.py --------- Co-authored-by: Datta Nimmaturi <[email protected]>

danielhanchen · 2025-09-08T06:58:11Z

unsloth/models/loader.py

+
+        # Add the missing vLLM/inference parameters
+        fast_inference             = False, # uses vLLM
+        gpu_memory_utilization     = 0.5,


Change to 0.9 if standby seen

made the change in load_vllm on unsloth-zoo that makes it work both for FastVisionModel and FastLanguageModel

unsloth/models/vision.py

danielhanchen · 2025-09-08T07:01:21Z

unsloth/models/vision.py

        if os.environ.get("UNSLOTH_MODEL_NAME", "") == "":
            os.environ["UNSLOTH_MODEL_NAME"] = model_name.lower()

+        is_vlm = (auto_model in [AutoModelForVision2Seq, AutoModelForImageTextToText])


I think HF is deprecating AutoModelForVision2Seq - best to add a try except import, then replace it with AutoModelForImageTextToText

We already have it up top in L22 when we initially import it here

danielhanchen

Nice work

Datta0 and others added 19 commits July 6, 2025 07:48

[WIP] use vLLM for vision language models

9ce943b

Update README.md

71b4d7b

Editing icon sizes

Update README.md

a1f84bb

Updating icon sizes

Update README.md (unslothai#2885)

4df0b67

MoE kernels AGPLv3

5986b1f

versioning

cf40750

silienty skip falcon h1 import is transformers_version < 4.53.0 (unsl…

6c32cae

…othai#2912)

Dynamically adjust get_per_token_logps function and patch as well (un…

631ab63

…slothai#2911)

add intel gpu with vllm support (unslothai#2903)

637ec67

[bugs] fix for casual mask (unslothai#2868)

b7879ca

* fix for casual mask * use un_casual in sdpa * add missing mask * fix for type

Explicitly check if xformers exists for attention (unslothai#2889)

3dc2095

Update __init__.py

9d507ce

Update llama.py

100a006

if mlp doesn't exist in layer module check for feed_forward name for …

7855ac6

…falcon h1 (unslothai#2913)

WIP VLM vLLM

a959b53

Make vLLM patch a function

2f39018

Add save and load lora functions

6fedbd0

Datta0 mentioned this pull request Jul 15, 2025

Fast Inference with vLLM for VLMs unslothai/unsloth-zoo#202

Merged

2 tasks

Datta0 added 8 commits July 15, 2025 18:07

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

275f0fe

Make fast_inference setup depend on the flag

a712e4c

Improve fast inference patching mechanism

2f71ad8

Make vision setting depend on checks in fastbasemodel

a91c3e9

Check LoRA and vLLM intercompatibility for vision models

5a5c033

Comment pointing to vLLM LoRA check

04f060d

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

5d79305

Improve lora validation on vLLM

307a111

Datta0 force-pushed the vlm_fast_infer branch from c278812 to 307a111 Compare July 21, 2025 05:15

danielhanchen reviewed Jul 23, 2025

View reviewed changes

unsloth/models/_utils.py Outdated Show resolved Hide resolved

Datta0 and others added 20 commits August 19, 2025 05:41

Better attn_impl handling

820f2cf

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

74eca60

Patch TF protobuf incompatability

9cdc96b

Update _auto_install.py

b5d2175

Update pyproject.toml

01f915a

Update rl.py

4428029

Protobuf issue

2ca62f4

Update pyproject.toml

21675c2

Fix extras transformers typo in pyproject.toml

16ba6c9

Update _utils.py

be65943

adallow float32 dtype in FastLanguageModel (unslothai#3204)

f2c5651

Update loader.py

92d267e

Update vision.py

3c70edd

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

1bb8624

Suppress message and use unsloth sampling params

0dbf1cd

Use trl sampling params for now

3f07ea4

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

2245004

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

54e8402

danielhanchen reviewed Sep 8, 2025

View reviewed changes

unsloth/models/vision.py Outdated Show resolved Hide resolved

danielhanchen reviewed Sep 8, 2025

View reviewed changes

danielhanchen requested changes Sep 8, 2025

View reviewed changes

Datta0 added 3 commits September 8, 2025 07:50

Improve error message

849e7db

Merge remote-tracking branch 'origin/main' into vlm_fast_infer

f7bf0e2

fixup quantized fast inference model name

a2ebc1d

Datta0 force-pushed the vlm_fast_infer branch from 935f19e to a2ebc1d Compare September 9, 2025 08:26

pluesclues mentioned this pull request Sep 16, 2025

official VLM GRPO notebook adapted to TRL unslothai/notebooks#105

Merged

danielhanchen merged commit a6176ad into unslothai:main Sep 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fast Inference with vLLM for VLMs #2975

Fast Inference with vLLM for VLMs #2975

Datta0 commented Jul 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

danielhanchen Sep 8, 2025

Uh oh!

Datta0 Sep 8, 2025

Uh oh!

Uh oh!

danielhanchen Sep 8, 2025

Uh oh!

Datta0 Sep 8, 2025 •

edited

Loading

Uh oh!

danielhanchen left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Fast Inference with vLLM for VLMs #2975

Fast Inference with vLLM for VLMs #2975

Conversation

Datta0 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

danielhanchen Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Datta0 Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danielhanchen Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Datta0 Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielhanchen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Datta0 commented Jul 15, 2025 •

edited

Loading

Datta0 Sep 8, 2025 •

edited

Loading