Feature Request: support qwen3-vl series

[wsbagnsv1]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Qwen has released two new SOTA open source mutlimodal models:
https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Thinking
https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct

Motivation

new SOTA vision models

Possible Implementation

huggingface/transformers#40795
sgl-project/sglang#10323

[arcral]

lol

[gileneusz]

👀

[wsbagnsv1]

New 30b models were just released 😉

https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct
https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Thinking

[AbdullahMPrograms]

New 30b models were just released 😉

https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Thinking

Fantastic models which would be amazing in llama.cpp!!

[gileneusz]

👀

[armand0e]

Hopefully coming soon 😄

[gileneusz]

👀

https://x.com/Alibaba_Qwen/status/1974289216113947039

[am17an]

The language model should already be supported, but creating a mmproj might require some changes. Specifically, there is deepstack_merger_list which doesn't have support. @CISC do you have any idea how this might be done?

[CISC]

The language model should already be supported, but creating a mmproj might require some changes. Specifically, there is deepstack_merger_list which doesn't have support. @CISC do you have any idea how this might be done?

I have not looked into it, nor mtmd in general yet, so probably not much help.

[Thireus]

Draft mapping, which most likely contains mistakes (e.g. some gate tensors are missing I believe, such as for merger.linear_fc1):

model.language_model.embed_tokens.weight              -> TOKEN_EMBD -> token_embd.weight
model.language_model.layers.0.mlp.gate.weight         -> FFN_GATE_INP -> blk.0.ffn_gate_inp.weight
model.language_model.layers.0.self_attn.k_norm.weight -> ATTN_K_NORM -> blk.0.attn_k_norm.weight
model.language_model.layers.0.self_attn.k_proj.weight -> ATTN_K -> blk.0.attn_k.weight
model.language_model.layers.0.self_attn.o_proj.weight -> ATTN_OUT -> blk.0.attn_output.weight
model.language_model.layers.0.self_attn.q_norm.weight -> ATTN_Q_NORM -> blk.0.attn_q_norm.weight
model.language_model.layers.0.self_attn.q_proj.weight -> ATTN_Q -> blk.0.attn_q.weight
model.language_model.layers.0.self_attn.v_proj.weight -> ATTN_V -> blk.0.attn_v.weight

model.visual.blocks.0.attn.proj.bias   -> V_ENC_ATTN_O -> v.blk.0.attn_out.bias
model.visual.blocks.0.attn.proj.weight -> V_ENC_ATTN_O -> v.blk.0.attn_out.weight

--SPLIT model.visual.blocks.0.attn.qkv.bias & model.visual.blocks.0.attn.qkv.weight:
model.visual.blocks.0.attn.q.bias   -> V_ENC_ATTN_Q -> v.blk.0.attn_q.bias
model.visual.blocks.0.attn.k.bias   -> V_ENC_ATTN_K -> v.blk.0.attn_k.bias
model.visual.blocks.0.attn.v.bias   -> V_ENC_ATTN_V -> v.blk.0.attn_v.bias
model.visual.blocks.0.attn.q.weight -> V_ENC_ATTN_Q -> v.blk.0.attn_q.weight
model.visual.blocks.0.attn.k.weight -> V_ENC_ATTN_K -> v.blk.0.attn_k.weight
model.visual.blocks.0.attn.v.weight -> V_ENC_ATTN_V -> v.blk.0.attn_v.weight

model.visual.blocks.0.mlp.linear_fc1.bias   -> V_ENC_FFN_UP -> v.blk.0.ffn_up.bias
model.visual.blocks.0.mlp.linear_fc1.weight -> V_ENC_FFN_UP -> v.blk.0.ffn_up.weight
model.visual.blocks.0.mlp.linear_fc2.bias   -> V_ENC_FFN_DOWN -> v.blk.0.ffn_down.bias
model.visual.blocks.0.mlp.linear_fc2.weight -> V_ENC_FFN_DOWN -> v.blk.0.ffn_down.weight
model.visual.blocks.0.norm1.bias            -> V_ENC_INPUT_NORM -> v.blk.0.ln1.bias
model.visual.blocks.0.norm1.weight          -> V_ENC_INPUT_NORM -> v.blk.0.ln1.weight
model.visual.blocks.0.norm2.bias            -> V_ENC_POST_ATTN_NORM -> v.blk.0.ln2.bias
model.visual.blocks.0.norm2.weight          -> V_ENC_POST_ATTN_NORM -> v.blk.0.ln2.weight

model.visual.deepstack_merger_list.0.linear_fc1.bias   -> NEW TENSOR CATEGORY! V_DEEPSTACK_MERGER_FFN_UP -> v.deepstack_merger_list.0.ffn_up.bias
model.visual.deepstack_merger_list.0.linear_fc1.weight -> NEW TENSOR CATEGORY! V_DEEPSTACK_MERGER_FFN_UP -> v.deepstack_merger_list.0.ffn_up.weight
model.visual.deepstack_merger_list.0.linear_fc2.bias   -> NEW TENSOR CATEGORY! V_DEEPSTACK_MERGER_FFN_DOWN -> v.deepstack_merger_list.0.ffn_down.bias
model.visual.deepstack_merger_list.0.linear_fc2.weight -> NEW TENSOR CATEGORY! V_DEEPSTACK_MERGER_FFN_DOWN -> v.deepstack_merger_list.0.ffn_down.weight
model.visual.deepstack_merger_list.0.norm.bias         -> NEW TENSOR CATEGORY! V_DEEPSTACK_MERGER_FFN_NORM -> v.deepstack_merger_list.0.ffn_norm.bias
model.visual.deepstack_merger_list.0.norm.weight       -> NEW TENSOR CATEGORY! V_DEEPSTACK_MERGER_FFN_NORM -> v.deepstack_merger_list.0.ffn_norm.weight

model.visual.merger.linear_fc1.bias   -> NEW TENSOR CATEGORY! V_MERGER_FFN_UP -> v.merger.ffn_up.bias
model.visual.merger.linear_fc1.weight -> NEW TENSOR CATEGORY! V_MERGER_FFN_UP -> v.merger.ffn_up.weight
model.visual.merger.linear_fc2.bias   -> NEW TENSOR CATEGORY! V_MERGER_FFN_DOWN -> v.merger.ffn_down.bias
model.visual.merger.linear_fc2.weight -> NEW TENSOR CATEGORY! V_MERGER_FFN_DOWN -> v.merger.ffn_down.weight
model.visual.merger.norm.bias         -> NEW TENSOR CATEGORY! V_MERGER_FFN_NORM -> v.merger.ffn_norm.bias
model.visual.merger.norm.weight       -> NEW TENSOR CATEGORY! V_MERGER_FFN_NORM -> v.merger.ffn_norm.weight

model.visual.patch_embed.proj.bias   -> V_ENC_EMBD_PATCH -> v.patch_embd.bias
model.visual.patch_embed.proj.weight -> V_ENC_EMBD_PATCH -> v.patch_embd.weight
model.visual.pos_embed.weight        -> V_ENC_EMBD_POS -> v.position_embd.weight

model.language_model.layers.0.input_layernorm.weight             -> ATTN_NORM -> blk.0.attn_norm.weight
model.language_model.layers.0.mlp.experts.down_proj              -> FFN_DOWN_EXP -> blk.0.ffn_down_exps.weight

--SPLIT model.language_model.layers.0.mlp.experts.gate_up_proj:
model.language_model.layers.0.mlp.experts.gate_proj           -> FFN_GATE_EXP -> blk.0.ffn_gate_exps.weight
model.language_model.layers.0.mlp.experts.up_proj             -> FFN_UP_EXP -> blk.0.ffn_up_exps.weight

model.language_model.layers.0.post_attention_layernorm.weight    -> FFN_NORM -> blk.0.ffn_norm.weight
model.language_model.norm.weight                                 -> OUTPUT_NORM -> output_norm.weight
lm_head.weight                                                   -> OUTPUT -> output.weight

If that can help anyone.

[theo77186]

found on Reddit, there is a patch that just got posted. Wonder if the original author will make a PR to upstream that.
https://huggingface.co/yairpatch/Qwen3-VL-30B-A3B-Thinking-GGUF/blob/main/qwen3vl-implementation.patch

[Thireus]

found on Reddit, there is a patch that just got posted. Wonder if the original author will make a PR to upstream that. https://huggingface.co/yairpatch/Qwen3-VL-30B-A3B-Thinking-GGUF/blob/main/qwen3vl-implementation.patch

I've produced builds if anyone wants to test: https://github.com/Thireus/llama.cpp/releases/tag/tr-qwen3-vl-b6906-26dd953. Please report back how it performs.

Test GGUFs are on the author's repo: https://huggingface.co/yairpatch/Qwen3-VL-30B-A3B-Thinking-GGUF/tree/main

As mentioned on the reddit post, run it with: --mmproj mmproj-Qwen3-VL-30B-A3B-F16.gguf --jinja