Skip to content

clip: move model cgraphs into their own files#17965

Merged
ngxson merged 7 commits intoggml-org:masterfrom
ngxson:xsn/clip_refactor_smaller_files
Dec 12, 2025
Merged

clip: move model cgraphs into their own files#17965
ngxson merged 7 commits intoggml-org:masterfrom
ngxson:xsn/clip_refactor_smaller_files

Conversation

@ngxson
Copy link
Contributor

@ngxson ngxson commented Dec 12, 2025

Some models like deepseek-ocr (#17400), gemma3n (#17961) and LFM2-audio (#17694) adds quite a lot of code into clip.cpp making it hard to track changes.

This PR moves cgraph builder of all vision/audio models into their own file, mirroring the recent refactoring in libllama by @pwilkin

Migration guide for pending PRs

  1. Create a new file under mtmd/models/your_model.cpp
  2. Implement the code inside newly created file (simply copy your existing build_*() function from clip.cpp into here)
    NOTE: You won't be able to access clip_ctx directly from here. However, you can add custom fields into clip_graph and modify its constructor inside clip.cpp to copy fields into clip_graph
  3. Add class definition into models.h
  4. Modify clip_image_build_graph() in clip.cpp to use the new cgraph

@ngxson
Copy link
Contributor Author

ngxson commented Dec 12, 2025

Test results:

[vision] OK:   ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
[vision] OK:   THUDM/glm-edge-v-5b-gguf:Q4_K_M
[vision] OK:   second-state/Llava-v1.5-7B-GGUF:Q2_K
[vision] OK:   cjpais/llava-1.6-mistral-7b-gguf:Q3_K_M
[vision] OK:   ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
[vision] OK:   second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
[vision] OK:   openbmb/MiniCPM-V-2_6-gguf:Q2_K
[vision] OK:   openbmb/MiniCPM-o-2_6-gguf:Q4_0
[vision] OK:   bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/InternVL2_5-1B-GGUF:Q8_0
[vision] OK:   ggml-org/InternVL3-1B-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[vision] OK:   ggml-org/LFM2-VL-450M-GGUF:Q8_0
[vision] OK:   ggml-org/granite-docling-258M-GGUF:Q8_0
[vision] OK:   ggml-org/LightOnOCR-1B-1025-GGUF:Q8_0
[audio]  OK:   ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF:Q8_0
[audio]  OK:   ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[audio]  OK:   ggml-org/Voxtral-Mini-3B-2507-GGUF:Q4_K_M
[vision] OK:   ggml-org/pixtral-12b-GGUF:Q4_K_M
[vision] OK:   ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF
[vision] OK:   ggml-org/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2-VL-7B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2.5-VL-7B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen3-VL-2B-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/InternVL3-8B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/InternVL3-14B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2.5-Omni-7B-GGUF:Q4_K_M
[audio]  OK:   ggml-org/ultravox-v0_5-llama-3_1-8b-GGUF:Q4_K_M
[audio]  OK:   ggml-org/Qwen2.5-Omni-7B-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2.5-VL-72B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUF:IQ1_S

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filenaming convention is to use dashes instead of underscores for C++ source files

@ngxson ngxson merged commit e39a2ce into ggml-org:master Dec 12, 2025
68 of 69 checks passed
sfallah added a commit to sfallah/llama.cpp that referenced this pull request Dec 13, 2025
Anico2 added a commit to Anico2/llama.cpp that referenced this pull request Jan 15, 2026
* clip: move model cgraphs into their own files

* more explicit enums

* fix linux build

* fix naming

* missing headers

* nits: add comments for contributors
blime4 referenced this pull request in blime4/llama.cpp Feb 5, 2026
* clip: move model cgraphs into their own files

* more explicit enums

* fix linux build

* fix naming

* missing headers

* nits: add comments for contributors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants