Feature Request: Support Kimi K2.5

[Skelectric]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Requesting support for Kimi K2.5's image encoding.

Motivation

Multimodal support for the best open-weights model currently available

Possible Implementation

Llama.cpp support was just merged: ggml-org/llama.cpp#19127

[ubergarm]

I haven't tried @AesSedai 's mmproj files yet: https://huggingface.co/AesSedai/Kimi-K2.5-GGUF/tree/main , but he handled the Q4_X this time as I was out.

[Lissanro]

I can confirm that his mmproj files work well, but I made my own Q4_X quant out of BF16 that I regenerated with mainline llama.cpp after applying the patch to support the vision tower (currently the patch is merged to the mainline). However, I made my Q4_X based on his recipe, so it should be the same. It would be great to see this integrated to ik_llama.cpp if possible since prompt processing speed in the mainline is noticeably slower.

[ubergarm]

@Lissanro

I made my Q4_X based on his recipe, so it should be the same.

Right, the Q4_X is a combination of that recipe using Q4_0 for routed exps, no imatrix, and patching ggml/src/ggml-quants.c quantize_row_q4_0_ref() to use the older max / -7; and recompiling before quantizing.

<         const float d  = max / -8;
---
>         const float d  = max / -7;

Thanks for confirming you're having luck with the mmproj file on mainline as its on my TODO list to try it still and it would be cool if it lands here too!

[Skelectric]

I've been trying to use a patch shared by user gghfez on hf (https://huggingface.co/jukofyork/creative-writing-control-vectors-v3.0/discussions/15#698e943a9dc9e4d73f2b3533) but haven't had any luck, just keep getting a segfault. Has anyone else tried it?

log

``` $ LD_LIBRARY_PATH=/run/opengl-driver/lib:$LD_LIBRARY_PATH \ ~/dev/ik_llama.cpp/build/bin/llama-server \ --model ~/models/Kimi-K2.5/UD-Q4_K_XL/Kimi-K2.5-UD-Q4_K_XL-00001-of-00013.gguf \ --mmproj ~/models/Kimi-K2.5/mmproj-Kimi-K2.5-Q8_0.gguf \ --host 0.0.0.0 \ --port 8081 \ --jinja \ --chat-template-file ~/models/Kimi-K2.5/chat_template.jinja \ -amb 512 \ -ctk q8_0 \ -ngl 99 \ -ot exps=CPU \ --ctx-size 256000 \ --batch-size 16384 \ --ubatch-size 16384 \ --threads 64 \ --webui llamacpp ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes, VRAM: 97247 MiB INFO [ main] build info | tid="140316620816384" timestamp=1771130489 build=4195 commit="f5fe33b7a" INFO [ main] system info | tid="140316620816384" timestamp=1771130489 n_threads=64 n_threads_batch=-1 total_threads=128 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | " CUDA0: using device CUDA0 - 96688 MiB free llama_model_loader: additional 12 GGUFs metadata loaded. llama_model_loader: loaded meta data with 64 key-value pairs and 1096 tensors from /home/user/models/Kimi-K2.5/UD-Q4_K_XL/Kimi-K2.5-UD-Q4_K_XL-00001-of-00013.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = deepseek2 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Kimi-K2.5 llama_model_loader: - kv 3: general.basename str = Kimi-K2.5 llama_model_loader: - kv 4: general.quantized_by str = Unsloth llama_model_loader: - kv 5: general.size_label str = 384x14B llama_model_loader: - kv 6: general.license str = other llama_model_loader: - kv 7: general.license.name str = modified-mit llama_model_loader: - kv 8: general.repo_url str = https://huggingface.co/unsloth llama_model_loader: - kv 9: general.base_model.count u32 = 1 llama_model_loader: - kv 10: general.base_model.0.name str = Kimi K2.5 llama_model_loader: - kv 11: general.base_model.0.organization str = Moonshotai llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/moonshotai/Kim... llama_model_loader: - kv 13: general.tags arr[str,2] = ["unsloth", "image-text-to-text"] llama_model_loader: - kv 14: deepseek2.block_count u32 = 61 llama_model_loader: - kv 15: deepseek2.context_length u32 = 262144 llama_model_loader: - kv 16: deepseek2.embedding_length u32 = 7168 llama_model_loader: - kv 17: deepseek2.feed_forward_length u32 = 18432 llama_model_loader: - kv 18: deepseek2.attention.head_count u32 = 64 llama_model_loader: - kv 19: deepseek2.attention.head_count_kv u32 = 1 llama_model_loader: - kv 20: deepseek2.rope.scaling.type str = yarn llama_model_loader: - kv 21: deepseek2.rope.scaling.factor f32 = 64.000000 llama_model_loader: - kv 22: deepseek2.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 23: deepseek2.rope.scaling.yarn_beta_fast f32 = 32.000000 llama_model_loader: - kv 24: deepseek2.rope.scaling.yarn_beta_slow f32 = 1.000000 llama_model_loader: - kv 25: deepseek2.rope.freq_base f32 = 50000.000000 llama_model_loader: - kv 26: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 27: deepseek2.expert_used_count u32 = 8 llama_model_loader: - kv 28: deepseek2.expert_group_count u32 = 1 llama_model_loader: - kv 29: deepseek2.expert_group_used_count u32 = 1 llama_model_loader: - kv 30: deepseek2.expert_gating_func u32 = 2 llama_model_loader: - kv 31: deepseek2.leading_dense_block_count u32 = 1 llama_model_loader: - kv 32: deepseek2.vocab_size u32 = 163840 llama_model_loader: - kv 33: deepseek2.attention.q_lora_rank u32 = 1536 llama_model_loader: - kv 34: deepseek2.attention.kv_lora_rank u32 = 512 llama_model_loader: - kv 35: deepseek2.attention.key_length u32 = 576 llama_model_loader: - kv 36: deepseek2.attention.value_length u32 = 512 llama_model_loader: - kv 37: deepseek2.attention.key_length_mla u32 = 192 llama_model_loader: - kv 38: deepseek2.attention.value_length_mla u32 = 128 llama_model_loader: - kv 39: deepseek2.expert_feed_forward_length u32 = 2048 llama_model_loader: - kv 40: deepseek2.expert_count u32 = 384 llama_model_loader: - kv 41: deepseek2.expert_shared_count u32 = 1 llama_model_loader: - kv 42: deepseek2.expert_weights_scale f32 = 2.827000 llama_model_loader: - kv 43: deepseek2.expert_weights_norm bool = true llama_model_loader: - kv 44: deepseek2.rope.dimension_count u32 = 64 llama_model_loader: - kv 45: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000 llama_model_loader: - kv 46: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 47: tokenizer.ggml.pre str = kimi-k2 llama_model_loader: - kv 48: tokenizer.ggml.tokens arr[str,163840] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 49: tokenizer.ggml.token_type arr[i32,163840] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 50: tokenizer.ggml.merges arr[str,163328] = ["Ġ Ġ", "ĠĠ ĠĠ", "Ġ t", "i n",... llama_model_loader: - kv 51: tokenizer.ggml.bos_token_id u32 = 163584 llama_model_loader: - kv 52: tokenizer.ggml.eos_token_id u32 = 163586 llama_model_loader: - kv 53: tokenizer.ggml.padding_token_id u32 = 163839 llama_model_loader: - kv 54: tokenizer.chat_template str = {%- macro render_content(msg) -%}\n ... llama_model_loader: - kv 55: general.quantization_version u32 = 2 llama_model_loader: - kv 56: general.file_type u32 = 15 llama_model_loader: - kv 57: quantize.imatrix.file str = Kimi-K2.5-GGUF/imatrix_unsloth.gguf llama_model_loader: - kv 58: quantize.imatrix.dataset str = unsloth_calibration_Kimi-K2.5.txt llama_model_loader: - kv 59: quantize.imatrix.entries_count u32 = 789 llama_model_loader: - kv 60: quantize.imatrix.chunks_count u32 = 50 llama_model_loader: - kv 61: split.no u16 = 0 llama_model_loader: - kv 62: split.tensors.count i32 = 1096 llama_model_loader: - kv 63: split.count u16 = 13 llama_model_loader: - type f32: 365 tensors llama_model_loader: - type f16: 3 tensors llama_model_loader: - type q8_0: 193 tensors llama_model_loader: - type q4_K: 420 tensors llama_model_loader: - type q5_K: 15 tensors llama_model_loader: - type q6_K: 100 tensors ================= Adjusted mainline llama.cpp MLA tensors to ik_llama.cpp load: printing all EOG tokens: load: - 163586 ('<|im_end|>') load: special tokens cache size = 256 load: token to piece cache size = 1.0606 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = deepseek2 llm_load_print_meta: n_ctx_train = 262144 llm_load_print_meta: n_embd = 7168 llm_load_print_meta: n_layer = 61 llm_load_print_meta: n_head = 64 llm_load_print_meta: n_head_kv = 64 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_swa_pattern = 1 llm_load_print_meta: n_embd_head_k = 192 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 12288 llm_load_print_meta: n_embd_v_gqa = 8192 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 18432 llm_load_print_meta: n_expert = 384 llm_load_print_meta: n_expert_used = 8 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = yarn llm_load_print_meta: freq_base_train = 50000.0 llm_load_print_meta: freq_scale_train = 0.015625 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 671B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 1.026 T llm_load_print_meta: model size = 579.281 GiB (4.848 BPW) llm_load_print_meta: repeating layers = 577.769 GiB (4.846 BPW, 1024.059 B parameters) llm_load_print_meta: general.name = Kimi-K2.5 llm_load_print_meta: n_layer_dense_lead = 1 llm_load_print_meta: n_lora_q = 1536 llm_load_print_meta: n_lora_kv = 512 llm_load_print_meta: n_ff_exp = 2048 llm_load_print_meta: n_expert_shared = 1 llm_load_print_meta: expert_weights_scale = 2.8 llm_load_print_meta: expert_weights_norm = 1 llm_load_print_meta: expert_gating_func = sigmoid llm_load_print_meta: rope_yarn_log_mul = 0.1000 print_info: vocab type = BPE print_info: n_vocab = 163840 print_info: n_merges = 163328 print_info: BOS token = 163584 '[BOS]' print_info: EOS token = 163586 '<|im_end|>' print_info: EOT token = 163586 '<|im_end|>' print_info: PAD token = 163839 '[PAD]' print_info: LF token = 198 'Ċ' print_info: EOG token = 163586 '<|im_end|>' print_info: max token length = 512 llm_load_tensors: ggml ctx size = 0.90 MiB Tensor blk.1.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.1.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.1.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.2.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.2.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.2.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.3.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.3.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.3.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.4.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.4.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.4.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.5.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.5.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.5.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.6.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.6.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.6.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.7.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.7.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.7.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.8.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.8.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.8.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.9.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.9.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.40.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.40.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.40.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.41.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.41.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.41.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.42.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.42.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.42.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.43.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.43.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.43.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.44.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.44.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.44.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.45.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.45.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.45.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.46.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.46.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.46.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.47.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.47.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.47.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.48.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.48.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.48.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.49.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.49.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.49.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.50.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.50.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.50.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.51.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.51.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.51.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.52.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.52.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.52.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.53.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.53.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.53.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.54.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.54.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.54.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.55.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.55.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.55.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.56.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.56.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.56.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.57.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.57.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.57.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.58.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.58.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.58.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.59.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.59.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.59.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.60.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.60.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.60.ffn_down_exps.weight buffer type overriden to CPU llm_load_tensors: offloading 61 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 62/62 layers to GPU llm_load_tensors: CPU buffer size = 42216.68 MiB llm_load_tensors: CPU buffer size = 46945.80 MiB llm_load_tensors: CPU buffer size = 47249.59 MiB llm_load_tensors: CPU buffer size = 45581.82 MiB llm_load_tensors: CPU buffer size = 44812.49 MiB llm_load_tensors: CPU buffer size = 45516.18 MiB llm_load_tensors: CPU buffer size = 45572.82 MiB llm_load_tensors: CPU buffer size = 47224.58 MiB llm_load_tensors: CPU buffer size = 45518.46 MiB llm_load_tensors: CPU buffer size = 47167.94 MiB llm_load_tensors: CPU buffer size = 45511.54 MiB llm_load_tensors: CPU buffer size = 45248.05 MiB llm_load_tensors: CPU buffer size = 42190.00 MiB llm_load_tensors: CPU buffer size = 630.00 MiB llm_load_tensors: CUDA0 buffer size = 7367.78 MiB .................................................................................................... ============ llm_prepare_mla: need to compute 61 wkv_b tensors Computed blk.0.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.1.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.2.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.3.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.4.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.5.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.6.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.7.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.8.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.9.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.10.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.11.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.12.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.13.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.14.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.15.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.16.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.17.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.18.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.19.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.20.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.21.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.22.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.23.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.24.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.25.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.26.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.27.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.28.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.29.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.30.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.31.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.32.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.33.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.34.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.35.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.36.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.37.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.38.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.39.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.40.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.41.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.42.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.43.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.44.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.45.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.46.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.47.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.48.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.49.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.50.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.51.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.52.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.53.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.54.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.55.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.56.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.57.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.58.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.59.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Computed blk.60.attn_kv_b.weight as 512 x 16384 and stored in buffer CUDA0 Adjust batch size for mtmd: u_batch = 16384, batch = 16384 ===================================== llama_init_from_model: f16 llama_init_from_model: n_ctx = 256000 llama_init_from_model: n_batch = 16384 llama_init_from_model: n_ubatch = 16384 llama_init_from_model: flash_attn = 1 llama_init_from_model: mla_attn = 3 llama_init_from_model: attn_max_b = 512 llama_init_from_model: fused_moe = 1 llama_init_from_model: grouped er = 0 llama_init_from_model: fused_up_gate = 1 llama_init_from_model: fused_mmad = 1 llama_init_from_model: rope_cache = 0 llama_init_from_model: graph_reuse = 1 llama_init_from_model: k_cache_hadam = 0 llama_init_from_model: split_mode_graph_scheduling = 0 llama_init_from_model: reduce_type = f16 llama_init_from_model: sched_async = 0 llama_init_from_model: ser = -1, 0 llama_init_from_model: freq_base = 50000.0 llama_init_from_model: freq_scale = 0.015625 llama_kv_cache_init: CUDA0 KV buffer size = 9114.29 MiB llama_init_from_model: KV self size = 9114.26 MiB, c^KV (q8_0): 9114.26 MiB, kv^T: not used llama_init_from_model: CUDA_Host output buffer size = 0.62 MiB llama_init_from_model: CUDA0 compute buffer size = 23905.69 MiB llama_init_from_model: CUDA_Host compute buffer size = 8448.19 MiB llama_init_from_model: graph nodes = 4014 llama_init_from_model: graph splits = 122 XXXXXXXXXXXXXXXXXXXXX Setting only active experts offload clip_model_loader: model name: Kimi K2.5 clip_model_loader: description: clip_model_loader: GGUF version: 3 clip_model_loader: alignment: 32 clip_model_loader: n_tensors: 335 clip_model_loader: n_kv: 28

clip_model_loader: has vision encoder
clip_ctx: have 2 back-ends:
0: CPU
1: CUDA0
ggml_backend_cuda_context: a context for device 0 already exists?
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector: kimik25
load_hparams: n_embd: 1152
load_hparams: n_head: 16
load_hparams: n_ff: 4304
load_hparams: n_layer: 27
load_hparams: ffn_op: gelu
load_hparams: projection_dim: 7168

--- vision hparams ---
load_hparams: image_size: 896
load_hparams: patch_size: 14
load_hparams: has_llava_proj: 0
load_hparams: minicpmv_version: 0
load_hparams: n_merge: 2
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels: 6272
load_hparams: image_max_pixels: 3211264

load_hparams: model size: 613.37 MiB
load_hparams: metadata size: 0.12 MiB
alloc_compute_meta: warmup with image size = 448 x 448
Segmentation fault (core dumped) LD_LIBRARY_PATH=/run/opengl-driver/lib:$LD_LIBRARY_PATH ~/dev/ik_llama.cpp/build/bin/llama-server --model ~/models/Kimi-K2.5/UD-Q4_K_XL/Kimi-K2.5-UD-Q4_K_XL-00001-of-00013.gguf --mmproj ~/models/Kimi-K2.5/mmproj-Kimi-K2.5-Q8_0.gguf --host 0.0.0.0 --port 8081 --jinja --chat-template-file ~/models/Kimi-K2.5/chat_template.jinja -amb 512 -ctk q8_0 -ngl 99 -ot exps=CPU --ctx-size 256000 --batch-size 16384 --ubatch-size 16384 --threads 64 --webui llamacpp

</details>

[SamuelOliveirads]

What is the current state of multimodal/vision support here?

Does the current codebase support the vision architecture used by models like Qwen-VL or would it require a significant port of the core vision logic before model-specific support can be added?

[saood06]

What is the current state of multimodal/vision support here?

There is support, for some PRs see #798 #803 #883 #901.