model: support gemma 4 (vision + moe, no audio)#21309
Merged
ngxson merged 2 commits intoggml-org:masterfrom Apr 2, 2026
Merged
model: support gemma 4 (vision + moe, no audio)#21309ngxson merged 2 commits intoggml-org:masterfrom
ngxson merged 2 commits intoggml-org:masterfrom
Conversation
danbev
approved these changes
Apr 2, 2026
Contributor
Author
|
Running CI on my fork for faster result: ngxson#95 |
ggerganov
approved these changes
Apr 2, 2026
ngxson
requested review from
danbev and
ggerganov
and removed request for
JohannesGaessler
danbev
approved these changes
Apr 2, 2026
ggerganov
approved these changes
Apr 2, 2026
|
lgtm |
CISC
reviewed
Apr 2, 2026
Comment on lines
+7477
to
+7478
| self.gguf_writer.add_add_space_prefix(False) | ||
| self.gguf_writer.add_add_bos_token(False) # already added via the chat template |
Member
There was a problem hiding this comment.
If already added in the chat template that means it should be True, or is it not a single token?
Contributor
Author
There was a problem hiding this comment.
the chat template already had {{ bos_token }}, so add_bos_token is not necessary (though even if it set to True here, llama.cpp do have a logic to avoid double BOS)
I'm explicitly setting it to False here for clarity though
Member
There was a problem hiding this comment.
I think you misunderstand the purpose of this field, it's to explicitly signal that a model requires BOS, ie. for correct behavior when not using a chat template (completion or FIM).
Contributor
Author
There was a problem hiding this comment.
According to the HF implementation, BOS is defined but not being added automatically when I try tokenizer("my prompt", return_tensors="pt"), so I hope this is correct
Contributor
Author
There was a problem hiding this comment.
(For context, this is to match the behavior of HF transformers when I compare the activations via llama-eval-callback)
Member
There was a problem hiding this comment.
OK, so False is probably the correct value then.
|
thank you guys |
ngxson
changed the title
github-actions
bot
added
model
4 tasks
danielhanchen
added a commit
to unslothai/unsloth
that referenced
this pull request
Apr 2, 2026
#4790) The latest ggml-org/llama.cpp release (b8635) does not include Gemma 4 support (ggml-org/llama.cpp#21309 merged after the release was cut). This causes `llama-server` to fail with "unknown model architecture: gemma4" when loading Gemma 4 GGUFs. Temporarily default _DEFAULT_LLAMA_TAG to "master" so all new installs build from the llama.cpp master branch which includes Gemma 4 support. Once a new upstream release is cut with Gemma 4, this can be reverted back to "latest". Changes: - setup.sh: add _DEFAULT_LLAMA_TAG="master" maintainer default - setup.ps1: add $DefaultLlamaTag="master" maintainer default - install_llama_prebuilt.py: change DEFAULT_LLAMA_TAG fallback to "master" Users can still override via UNSLOTH_LLAMA_TAG env var.
Vect0rM
pushed a commit
to AtomicBot-ai/atomic-llama-cpp-turboquant
that referenced
this pull request
Apr 2, 2026
4 tasks
danielhanchen
added a commit
to unslothai/unsloth
that referenced
this pull request
Apr 2, 2026
ggml-org/llama.cpp b8637 includes Gemma 4 support (ggml-org/llama.cpp#21309). Revert the temporary "master" default back to a pinned release tag. This eliminates the HTTP 422 errors from the prebuilt resolver (which could not find a release matching "master"), avoids unnecessary source builds, and restores prebuilt binary downloads on all platforms. Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
|
no audio :( |
5 tasks
Contributor
Author
|
Audio will be a follow-up PR, I am working on that |
Wowfunhappy
added a commit
to Wowfunhappy/llama.cpp
that referenced
this pull request
Apr 3, 2026
Merges ggml-org/llama.cpp upstream (d23355a..7992aa7) including: - Gemma 4 model support (PR ggml-org#21309) - KV cache rotation for better quantization (ggml-org#21038) - Auto GPU memory fitting (llama_params_fit) - Many new model architectures (Qwen3.5, Kimi K2, LFM2, etc.) C++14/CUDA 7.5 compatibility fixes applied to merged code: - Replaced if constexpr with runtime if across CUDA files - Replaced constexpr __device__ functions with macros - Replaced structured bindings with .first/.second access - Replaced std::string_view/std::optional with std::string - Template specializations for ggml_cuda_cast (convert.cuh) - BF16 flash attention guarded behind CUDART_VERSION >= 11000 - Eager CUDA context init restored for accurate VRAM on non-VMM GPUs - Jinja C++17 structured bindings fixed (caused Qwen 3.5 segfault) Build system updates: - Added hf-cache-stub.cpp, server-tools-stub.cpp for C++14 compat - Added mtmd-image.cpp, httplib.cpp to build - convert_hf_to_gguf.py patched for PyTorch 1.13 compatibility - gguf vocab.py fallback for old tokenizers library Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
luvwinnie
added a commit
to luvwinnie/llama.cpp
that referenced
this pull request
Apr 4, 2026
Merge 59 upstream commits including: - model: support gemma 4 (vision + moe, no audio) (ggml-org#21309) - kv-cache: do not quantize SWA KV cache (ggml-org#21277) - Preserve RotorQuant exclusion from Hadamard rotation
wordingone
pushed a commit
to wordingone/llama-cpp-turboquant-cuda
that referenced
this pull request
Apr 6, 2026
kgluszczyk
added a commit
to Creatiwi-ai/llama.cpp
that referenced
this pull request
Apr 6, 2026
Rebased onto upstream master (b8672+) which includes Gemma 4 model support (PR ggml-org#21309, ggml-org#21326, ggml-org#21418). This enables loading Gemma 4 E2B/E4B GGUF models on-device via llama.cpp. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spiritbuun
added a commit
to spiritbuun/buun-llama-cpp
that referenced
this pull request
Apr 6, 2026
Cherry-pick Gemma 4 (26B MoE + 31B dense) from upstream PR ggml-org#21309: - ISWA dual-cache (5:1 SWA:global ratio) - Variable head_dim (256 SWA / 512 global) - MoE with 128 experts top-8 + shared expert - K=V on global layers (attention_k_eq_v) - Gemma 4 tokenizer (byte_encode support) Head padding: pad heads to nearest multiple of 128 for FWHT alignment. Enables turbo quants on Phi-3 (96→128), Qwen3-0.6B (64→128), etc. Zero padding preserves inner products (Parseval's theorem). FA VEC dispatch: add head_dim=512 instances for all turbo + q8_0 + f16 type combinations, needed for Gemma 4 global attention layers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fix a bug where model with both vision/audio cannot be converted properly
Requirements