Eval bug: Qwen 3.5 27B GGUF from unsloth hard crash

[snapo]

Name and Version

./llama.cpp/llama-cli --version
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
  Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
version: 8138 (b68a83e64)
built with GNU 11.4.0 for Linux x86_64

./llama.cpp/llama-server --model "models/Qwen3.5-27B-UD-Q4_K_XL.gguf" --alias "Qwen3.5 27B" --temp 0.6 --top-p 0.95 --min-p 0.01 --top-k 20 --port 16384 --host 0.0.0.0 --ctx-size 4096 --cache-type-k q8_0 --cache-type-v q8_0 --parallel 1 --cont-batching

Operating systems

Linux

GGML backends

CUDA

Hardware

2 x RTX 2080 Ti 22GB vram , 64GB systemram

Models

unsloth/Qwen3.5-27B-GGUF

Same behaviour with :
Qwen3.5-27B-UD-Q4_K_XL.gguf
Qwen3.5-27B-UD-Q5_K_XL.gguf
Qwen3.5-27B-UD-Q8_K_XL.gguf

Problem description & steps to reproduce

hard crash even on ctx size 512 and 1 input token

First Bad Commit

No response

Relevant log output

Logs

ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
build: 8138 (b68a83e) with GNU 11.4.0 for Linux x86_64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12

system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

Running without SSL
init: using 11 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model 'models/Qwen3.5-27B-UD-Q4_K_XL.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 2080 Ti): 22002 total, 7835 used, 13931 free vs. target of 1024
llama_params_fit_impl: - CUDA1 (NVIDIA GeForce RTX 2080 Ti): 22002 total, 9101 used, 12670 free vs. target of 1024
llama_params_fit_impl: projected to use 16936 MiB of device memory vs. 43538 MiB of free device memory
llama_params_fit_impl: targets for free memory can be met on all devices, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.65 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 2080 Ti) (0000:23:00.0) - 21844 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 2080 Ti) (0000:2d:00.0) - 21844 MiB free
llama_model_loader: loaded meta data with 42 key-value pairs and 851 tensors from models/Qwen3.5-27B-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen35
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 0.600000
llama_model_loader: - kv 5: general.name str = Qwen3.5-27B
llama_model_loader: - kv 6: general.basename str = Qwen3.5-27B
llama_model_loader: - kv 7: general.quantized_by str = Unsloth
llama_model_loader: - kv 8: general.size_label str = 27B
llama_model_loader: - kv 9: general.license str = apache-2.0
llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-2...
llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 12: general.tags arr[str,1] = ["image-text-to-text"]
llama_model_loader: - kv 13: qwen35.block_count u32 = 64
llama_model_loader: - kv 14: qwen35.context_length u32 = 262144
llama_model_loader: - kv 15: qwen35.embedding_length u32 = 5120
llama_model_loader: - kv 16: qwen35.feed_forward_length u32 = 17408
llama_model_loader: - kv 17: qwen35.attention.head_count u32 = 24
llama_model_loader: - kv 18: qwen35.attention.head_count_kv u32 = 4
llama_model_loader: - kv 19: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llama_model_loader: - kv 20: qwen35.rope.freq_base f32 = 10000000.000000
llama_model_loader: - kv 21: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 22: qwen35.attention.key_length u32 = 256
llama_model_loader: - kv 23: qwen35.attention.value_length u32 = 256
llama_model_loader: - kv 24: qwen35.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 25: qwen35.ssm.state_size u32 = 128
llama_model_loader: - kv 26: qwen35.ssm.group_count u32 = 16
llama_model_loader: - kv 27: qwen35.ssm.time_step_rank u32 = 48
llama_model_loader: - kv 28: qwen35.ssm.inner_size u32 = 6144
llama_model_loader: - kv 29: qwen35.full_attention_interval u32 = 4
llama_model_loader: - kv 30: qwen35.rope.dimension_count u32 = 64
llama_model_loader: - kv 31: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 32: tokenizer.ggml.pre str = qwen35
llama_model_loader: - kv 33: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 34: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 35: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 36: tokenizer.ggml.eos_token_id u32 = 248046
llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 248044
llama_model_loader: - kv 38: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 39: tokenizer.chat_template str = {%- set image_count = namespace(value...
llama_model_loader: - kv 40: general.quantization_version u32 = 2
llama_model_loader: - kv 41: general.file_type u32 = 15
llama_model_loader: - type f32: 353 tensors
llama_model_loader: - type q4_K: 359 tensors
llama_model_loader: - type q5_K: 96 tensors
llama_model_loader: - type q6_K: 43 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 15.57 GiB (4.97 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch = qwen35
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 5120
print_info: n_embd_inp = 5120
print_info: n_layer = 64
print_info: n_head = 24
print_info: n_head_kv = 4
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 256
print_info: n_embd_head_v = 256
print_info: n_gqa = 6
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 17408
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 6144
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 48
print_info: ssm_n_group = 16
print_info: ssm_dt_b_c_rms = 0
print_info: model type = ?B
print_info: model params = 26.90 B
print_info: general.name = Qwen3.5-27B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 11 ','
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 63 repeating layers to GPU
load_tensors: offloaded 65/65 layers to GPU
load_tensors: CPU_Mapped model buffer size = 682.03 MiB
load_tensors: CUDA0 model buffer size = 7353.08 MiB
load_tensors: CUDA1 model buffer size = 7908.45 MiB
............................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 4096
llama_context: n_ctx_seq = 4096
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = false
llama_context: freq_base = 10000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (4096) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 0.95 MiB
llama_kv_cache: CUDA0 KV buffer size = 128.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 128.00 MiB
llama_kv_cache: size = 256.00 MiB ( 4096 cells, 16 layers, 1/1 seqs), K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_memory_recurrent: CUDA0 RS buffer size = 77.93 MiB
llama_memory_recurrent: CUDA1 RS buffer size = 71.70 MiB
llama_memory_recurrent: size = 149.62 MiB ( 1 cells, 64 layers, 1 seqs), R (f32): 5.62 MiB, S (f32): 144.00 MiB
llama_context: pipeline parallelism enabled
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: CUDA0 compute buffer size = 228.06 MiB
sched_reserve: CUDA1 compute buffer size = 905.32 MiB
sched_reserve: CUDA_Host compute buffer size = 52.08 MiB
sched_reserve: graph nodes = 12729 (with bs=512), 4713 (with bs=1)
sched_reserve: graph splits = 5
sched_reserve: reserve took 49.71 ms, sched copies = 4
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv load_model: initializing slots, n_slots = 1
common_speculative_is_compat: the target context does not support partial sequence removal
srv load_model: speculative decoding not supported by this context
slot load_model: id 0 | task -1 | new slot, n_ctx = 4096
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use --cache-ram 0 to disable the prompt cache
srv load_model: for more info see https://github.com//pull/16391
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://0.0.0.0:16384
main: starting the main loop...
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, task.n_tokens = 568
slot update_slots: id 0 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_tokens = 56, batch.n_tokens = 56, progress = 0.098592
slot update_slots: id 0 | task 0 | n_tokens = 56, memory_seq_rm [56, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_tokens = 568, batch.n_tokens = 512, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_tokens = 568, batch.n_tokens = 512
slot init_sampler: id 0 | task 0 | init sampler, took 0.08 ms, tokens: text = 568, total = 568
/home/snapo/llama/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
ggml_cuda_compute_forward: SET failed
CUDA error: an illegal memory access was encountered
current device: 1, in function ggml_cuda_compute_forward at /home/snapo/llama/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2761
err
[New LWP 3283938]
[New LWP 3283952]
[New LWP 3283953]
[New LWP 3283954]
[New LWP 3283955]
[New LWP 3283956]
[New LWP 3283957]
[New LWP 3283958]
[New LWP 3283959]
[New LWP 3283960]
[New LWP 3283961]
[New LWP 3283962]
[New LWP 3283963]
[New LWP 3283964]
[New LWP 3283965]
[New LWP 3283966]
[New LWP 3283967]
[New LWP 3283968]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007be4be0ea42f in __GI___wait4 (pid=3284307, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
#0 0x00007be4be0ea42f in __GI___wait4 (pid=3284307, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x00005da4f81ac86b in ggml_print_backtrace ()
#2 0x00005da4f81ac9fe in ggml_abort ()
#3 0x00005da4f779fff6 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#4 0x00005da4f77b0e1f in ggml_cuda_compute_forward(ggml_backend_cuda_context&, ggml_tensor*) ()
#5 0x00005da4f77b4e7e in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) ()
#6 0x00005da4f77b722a in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#7 0x00005da4f81ccdc9 in ggml_backend_sched_compute_splits(ggml_backend_sched*) ()
#8 0x00005da4f7579611 in llama_context::graph_compute(ggml_cgraph*, bool) ()
#9 0x00005da4f7579765 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) ()
#10 0x00005da4f7583293 in llama_context::decode(llama_batch const&) ()
#11 0x00005da4f75848d0 in llama_decode ()
#12 0x00005da4f725c841 in server_context_impl::update_slots() ()
#13 0x00005da4f72991b0 in server_queue::start_loop(long) ()
#14 0x00005da4f7159cde in main ()
[Inferior 1 (process 3283937) detached]

[vikaskumarsingh123]

I have the same issue. Tested with and without flash attention; also with and without GGML_CUDA_DISABLE_GRAPHS (using env vars).

Always crashes as soon as I send the first message ("hi") on the chat interface.

The error:

CUDA error: an illegal memory access was encountered
current device: 1, in function ggml_cuda_compute_forward at /home/thewoodcutter/Development/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2761

It was working last when I pulled the branch pr/19747-alt. I will get the last commit of that branch before merge; and then test again.

Full Log

`/Development/llama.cpp/build/bin/llama-server --flash-attn 0 --host 0.0.0.0 -m /mnt/md127/models/AesSedai/Qwen3.5-122B-A10B-GGUF/Qwen3.5-122B-A10B-IQ4_XS-00001-of-00003.gguf --mmproj /mnt/md127/models/unsloth/Qwen3.5-122B-A10B-GGUF/mmproj-F32.gguf --parallel 1 --reasoning-budget 0 -ngl 999 -b 1024 -ub 32 -fitt 2048,256,256 --ctx-size 2000 ggml_cuda_init: found 3 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes Device 2: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes build: 8157 (2943210) with GNU 13.3.0 for Linux x86_64 system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 750,860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

Running without SSL
init: using 23 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/mnt/md127/models/AesSedai/Qwen3.5-122B-A10B-GGUF/Qwen3.5-122B-A10B-IQ4_XS-00001-of-00003.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 3090) : 24126 total, 21132 used, 2662 free vs. target of 2048
llama_params_fit_impl: - CUDA1 (NVIDIA GeForce RTX 2080 Ti): 22001 total, 18783 used, 3002 free vs. target of 256
llama_params_fit_impl: - CUDA2 (NVIDIA GeForce RTX 2080 Ti): 22001 total, 17207 used, 4586 free vs. target of 256
llama_params_fit_impl: projected to use 57124 MiB of device memory vs. 67376 MiB of free device memory
llama_params_fit_impl: targets for free memory can be met on all devices, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 3.32 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:09:00.0) - 23853 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 2080 Ti) (0000:06:00.0) - 21836 MiB free
llama_model_load_from_file_impl: using device CUDA2 (NVIDIA GeForce RTX 2080 Ti) (0000:0c:00.0) - 21836 MiB free
llama_model_loader: additional 2 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 50 key-value pairs and 879 tensors from /mnt/md127/models/AesSedai/Qwen3.5-122B-A10B-GGUF/Qwen3.5-122B-A10B-IQ4_XS-00001-of-00003.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen35moe
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 0.600000
llama_model_loader: - kv 5: general.name str = Qwen3.5 122B A10B
llama_model_loader: - kv 6: general.basename str = Qwen3.5
llama_model_loader: - kv 7: general.size_label str = 122B-A10B
llama_model_loader: - kv 8: general.license str = apache-2.0
llama_model_loader: - kv 9: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-1...
llama_model_loader: - kv 10: general.tags arr[str,1] = ["image-text-to-text"]
llama_model_loader: - kv 11: qwen35moe.block_count u32 = 48
llama_model_loader: - kv 12: qwen35moe.context_length u32 = 262144
llama_model_loader: - kv 13: qwen35moe.embedding_length u32 = 3072
llama_model_loader: - kv 14: qwen35moe.attention.head_count u32 = 32
llama_model_loader: - kv 15: qwen35moe.attention.head_count_kv u32 = 2
llama_model_loader: - kv 16: qwen35moe.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llama_model_loader: - kv 17: qwen35moe.rope.freq_base f32 = 10000000.000000
llama_model_loader: - kv 18: qwen35moe.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 19: qwen35moe.expert_count u32 = 256
llama_model_loader: - kv 20: qwen35moe.expert_used_count u32 = 8
llama_model_loader: - kv 21: qwen35moe.attention.key_length u32 = 256
llama_model_loader: - kv 22: qwen35moe.attention.value_length u32 = 256
llama_model_loader: - kv 23: qwen35moe.expert_feed_forward_length u32 = 1024
llama_model_loader: - kv 24: qwen35moe.expert_shared_feed_forward_length u32 = 1024
llama_model_loader: - kv 25: qwen35moe.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 26: qwen35moe.ssm.state_size u32 = 128
llama_model_loader: - kv 27: qwen35moe.ssm.group_count u32 = 16
llama_model_loader: - kv 28: qwen35moe.ssm.time_step_rank u32 = 64
llama_model_loader: - kv 29: qwen35moe.ssm.inner_size u32 = 8192
llama_model_loader: - kv 30: qwen35moe.full_attention_interval u32 = 4
llama_model_loader: - kv 31: qwen35moe.rope.dimension_count u32 = 64
llama_model_loader: - kv 32: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 33: tokenizer.ggml.pre str = qwen35
llama_model_loader: - kv 34: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 35: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 36: tokenizer.ggml.merges arr[str,247587] = ["\u0120 \u0120", "\u0120\u0120 \u0120\u0120", "i n", "\u0120 t",...
llama_model_loader: - kv 37: tokenizer.ggml.eos_token_id u32 = 248046
llama_model_loader: - kv 38: tokenizer.ggml.padding_token_id u32 = 248044
llama_model_loader: - kv 39: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 40: tokenizer.chat_template str = {%- set image_count = namespace(value...
llama_model_loader: - kv 41: general.quantization_version u32 = 2
llama_model_loader: - kv 42: general.file_type u32 = 7
llama_model_loader: - kv 43: quantize.imatrix.file str = /mnt/srv/snowdrift/fp16/Qwen3.5-122B-...
llama_model_loader: - kv 44: quantize.imatrix.dataset str = /mnt/srv/host/resources/KLD/calibrati...
llama_model_loader: - kv 45: quantize.imatrix.entries_count u32 = 612
llama_model_loader: - kv 46: quantize.imatrix.chunks_count u32 = 100
llama_model_loader: - kv 47: split.no u16 = 0
llama_model_loader: - kv 48: split.tensors.count i32 = 879
llama_model_loader: - kv 49: split.count u16 = 3
llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q8_0: 374 tensors
llama_model_loader: - type iq3_s: 96 tensors
llama_model_loader: - type iq4_xs: 48 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q8_0
print_info: file size = 56.25 GiB (3.96 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch = qwen35moe
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 3072
print_info: n_embd_inp = 3072
print_info: n_layer = 48
print_info: n_head = 32
print_info: n_head_kv = 2
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 256
print_info: n_embd_head_v = 256
print_info: n_gqa = 16
print_info: n_embd_k_gqa = 512
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 0
print_info: n_expert = 256
print_info: n_expert_used = 8
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 8192
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 64
print_info: ssm_n_group = 16
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 80B.A3B
print_info: model params = 122.11 B
print_info: general.name = Qwen3.5 122B A10B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 11 ','
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 '\u010a'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 47 repeating layers to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors: CPU_Mapped model buffer size = 772.97 MiB
load_tensors: CUDA0 model buffer size = 21027.23 MiB
load_tensors: CUDA1 model buffer size = 18686.36 MiB
load_tensors: CUDA2 model buffer size = 17118.47 MiB
....................................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 2048
llama_context: n_ctx_seq = 2048
llama_context: n_batch = 1024
llama_context: n_ubatch = 32
llama_context: causal_attn = 1
llama_context: flash_attn = disabled
llama_context: kv_unified = false
llama_context: freq_base = 10000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (2048) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 0.95 MiB
llama_kv_cache: CUDA0 KV buffer size = 16.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 16.00 MiB
llama_kv_cache: CUDA2 KV buffer size = 16.00 MiB
llama_kv_cache: size = 48.00 MiB ( 2048 cells, 12 layers, 1/1 seqs), K (f16): 24.00 MiB, V (f16): 24.00 MiB
llama_memory_recurrent: CUDA0 RS buffer size = 57.97 MiB
llama_memory_recurrent: CUDA1 RS buffer size = 49.69 MiB
llama_memory_recurrent: CUDA2 RS buffer size = 41.41 MiB
llama_memory_recurrent: size = 149.06 MiB ( 1 cells, 48 layers, 1 seqs), R (f32): 5.06 MiB, S (f32): 144.00 MiB
sched_reserve: reserving ...
sched_reserve: CUDA0 compute buffer size = 31.64 MiB
sched_reserve: CUDA1 compute buffer size = 31.64 MiB
sched_reserve: CUDA2 compute buffer size = 31.64 MiB
sched_reserve: CUDA_Host compute buffer size = 1.13 MiB
sched_reserve: graph nodes = 7556 (with bs=32), 5324 (with bs=1)
sched_reserve: graph splits = 4
sched_reserve: reserve took 85.28 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
clip_model_loader: model name: Qwen3.5-122B-A10B
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment: 32
clip_model_loader: n_tensors: 334
clip_model_loader: n_kv: 35

clip_model_loader: has vision encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: #16842

load_hparams: projector: qwen3vl_merger
load_hparams: n_embd: 1152
load_hparams: n_head: 16
load_hparams: n_ff: 4304
load_hparams: n_layer: 27
load_hparams: ffn_op: gelu
load_hparams: projection_dim: 3072

--- vision hparams ---
load_hparams: image_size: 768
load_hparams: patch_size: 16
load_hparams: has_llava_proj: 0
load_hparams: minicpmv_version: 0
load_hparams: n_merge: 2
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels: 8192
load_hparams: image_max_pixels: 4194304

load_hparams: model size: 1721.54 MiB
load_hparams: metadata size: 0.12 MiB
warmup: warmup with image size = 1472 x 1472
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4657.81 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 4884066560
alloc_compute_meta: graph splits = 1, nodes = 877
warmup: flash attention is disabled
srv load_model: loaded multimodal model, '/mnt/md127/models/unsloth/Qwen3.5-122B-A10B-GGUF/mmproj-F32.gguf'
srv load_model: initializing slots, n_slots = 1
common_speculative_is_compat: the target context does not support partial sequence removal
srv load_model: speculative decoding not supported by this context
slot load_model: id 0 | task -1 | new slot, n_ctx = 2048
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use --cache-ram 0 to disable the prompt cache
srv load_model: for more info see #16391
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
srv init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv update_slots: all slots are idle
srv params_from_: Chat format: peg-constructed
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 2048, n_keep = 0, task.n_tokens = 12
slot update_slots: id 0 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot init_sampler: id 0 | task 0 | init sampler, took 0.01 ms, tokens: text = 12, total = 12
slot update_slots: id 0 | task 0 | prompt processing done, n_tokens = 12, batch.n_tokens = 12
srv log_server_r: done request: POST /v1/chat/completions 100.68.172.91 200
/home/thewoodcutter/Development/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
ggml_cuda_compute_forward: MUL failed
CUDA error: an illegal memory access was encountered
current device: 1, in function ggml_cuda_compute_forward at /home/thewoodcutter/Development/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2761
err
[New LWP 15428]
[New LWP 15427]
[New LWP 15426]
[New LWP 15425]
[New LWP 15424]
[New LWP 15423]
[New LWP 15422]
[New LWP 15421]
[New LWP 15420]
[New LWP 15419]
[New LWP 15418]
[New LWP 15417]
[New LWP 15416]
[New LWP 15415]
[New LWP 15414]
[New LWP 15413]
[New LWP 15412]
[New LWP 15411]
[New LWP 15410]
[New LWP 15409]
[New LWP 15408]
[New LWP 15407]
[New LWP 15406]
[New LWP 15405]
[New LWP 15404]
[New LWP 15403]
[New LWP 15402]
[New LWP 15401]
[New LWP 15400]
[New LWP 15399]
[New LWP 15398]
[New LWP 15397]
[New LWP 15386]

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libopenblas.so.0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007baea5510813 in __GI___wait4 (pid=15429, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0 0x00007baea5510813 in __GI___wait4 (pid=15429, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x00007baea5b6d823 in ggml_print_backtrace () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-base.so.0
#2 0x00007baea5b6d9cb in ggml_abort () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-base.so.0
#3 0x00007bae8f366b67 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-cuda.so.0
#4 0x00007bae8f377160 in ggml_cuda_compute_forward(ggml_backend_cuda_context&, ggml_tensor*) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-cuda.so.0
#5 0x00007bae8f37af1e in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-cuda.so.0
#6 0x00007bae8f37d20e in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-cuda.so.0
#7 0x00007baea5b8a0f7 in ggml_backend_sched_graph_compute_async () from /home/thewoodcutter/Development/llama.cpp/build/bin/libggml-base.so.0
#8 0x00007baea5cc15b1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libllama.so.0
#9 0x00007baea5cc36c4 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libllama.so.0
#10 0x00007baea5cca286 in llama_context::decode(llama_batch const&) () from /home/thewoodcutter/Development/llama.cpp/build/bin/libllama.so.0
#11 0x00007baea5ccbd1f in llama_decode () from /home/thewoodcutter/Development/llama.cpp/build/bin/libllama.so.0
#12 0x00005d756bf7ccd8 in server_context_impl::update_slots() ()
#13 0x00005d756bfc4d6e in server_queue::start_loop(long) ()
#14 0x00005d756bedb810 in main ()
[Inferior 1 (process 15362) detached]
Aborted (core dumped)
`

[ggerganov]

@snapo This is already fixed on latest version - update

[jorismak]

On Intel Lunar Lake with the SYCL F16 backend:

bin/llama-cli -m ~/models/Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf --mmproj ~/models/Qwen3.5-35B-A3B-mmproj-F16.gguf --ctx-size 16384 -ctv f16 -ctk f16 -ngl 999  --temp 0.7 --top-p 0.8 --presence-penalty 1.5 --top-k 20 --min-p 0 --flash-attn on --jinja -p "Who are what are you, can you identify yourself?" -no-cnv -st

build      : b8157-2943210c1
model      : Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf
modalities : text, vision

> Who are what are you, can you identify yourself?

以上就是小编彼岸った.[New LWP 19597]
[New LWP 19595]
[New LWP 19592]
[New LWP 19588]
[New LWP 19587]
[New LWP 19551]
[New LWP 19550]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
⚠️ warning: File "/opt/intel/oneapi/compiler/2025.0/lib/libsycl.so.8.0.0-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /opt/intel/oneapi/compiler/2025.0/lib/libsycl.so.8.0.0-gdb.py
line to your configuration file "~/.config/gdb/gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "~/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007fa303cadf32 in ?? () from /usr/lib/libc.so.6
#0  0x00007fa303cadf32 in ?? () from /usr/lib/libc.so.6
#1  0x00007fa303ca239c in ?? () from /usr/lib/libc.so.6
#2  0x00007fa303ca23e4 in ?? () from /usr/lib/libc.so.6
#3  0x00007fa303d1267f in wait4 () from /usr/lib/libc.so.6
#4  0x00007fa304ee6e7a in ggml_print_backtrace () from ~/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007fa304efd4a6 in ggml_uncaught_exception() () from ~/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007fa303eb1eba in ?? () from /usr/lib/libstdc++.so.6
#7  0x00007fa303e975d9 in std::terminate() () from /usr/lib/libstdc++.so.6
#8  0x00007fa303eb2176 in __cxa_throw () from /usr/lib/libstdc++.so.6
#9  0x000000000055f5ab in common_chat_peg_parse(common_peg_arena const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, common_chat_parser_params const&) ()
#10 0x000000000055ebd6 in common_chat_parse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, common_chat_parser_params const&) ()
#11 0x0000000000441c38 in task_result_state::update_chat_msg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<common_chat_msg_diff, std::allocator<common_chat_msg_diff> >&) ()
#12 0x000000000046cb68 in server_task_result_cmpl_partial::update(task_result_state&) ()
#13 0x00000000004a3abf in server_response_reader::next(std::function<bool ()> const&) ()
#14 0x00000000004294b7 in cli_context::generate_completion[abi:cxx11](result_timings&) ()
#15 0x0000000000424c38 in main ()
[Inferior 1 (process 19535) detached]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to parse input at pos 0

Am I doing something wrong or is there stlil a similar bug in SYCL (or other) backends?

[snapo]

On Intel Lunar Lake with the SYCL F16 backend:

bin/llama-cli -m ~/models/Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf --mmproj ~/models/Qwen3.5-35B-A3B-mmproj-F16.gguf --ctx-size 16384 -ctv f16 -ctk f16 -ngl 999  --temp 0.7 --top-p 0.8 --presence-penalty 1.5 --top-k 20 --min-p 0 --flash-attn on --jinja -p "Who are what are you, can you identify yourself?" -no-cnv -st

build      : b8157-2943210c1
model      : Qwen3.5-35B-A3B-UD-Q5_K_XL.gguf
modalities : text, vision

> Who are what are you, can you identify yourself?

以上就是小编彼岸った.[New LWP 19597]
[New LWP 19595]
[New LWP 19592]
[New LWP 19588]
[New LWP 19587]
[New LWP 19551]
[New LWP 19550]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
⚠️ warning: File "/opt/intel/oneapi/compiler/2025.0/lib/libsycl.so.8.0.0-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /opt/intel/oneapi/compiler/2025.0/lib/libsycl.so.8.0.0-gdb.py
line to your configuration file "~/.config/gdb/gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "~/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
0x00007fa303cadf32 in ?? () from /usr/lib/libc.so.6
#0  0x00007fa303cadf32 in ?? () from /usr/lib/libc.so.6
#1  0x00007fa303ca239c in ?? () from /usr/lib/libc.so.6
#2  0x00007fa303ca23e4 in ?? () from /usr/lib/libc.so.6
#3  0x00007fa303d1267f in wait4 () from /usr/lib/libc.so.6
#4  0x00007fa304ee6e7a in ggml_print_backtrace () from ~/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007fa304efd4a6 in ggml_uncaught_exception() () from ~/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007fa303eb1eba in ?? () from /usr/lib/libstdc++.so.6
#7  0x00007fa303e975d9 in std::terminate() () from /usr/lib/libstdc++.so.6
#8  0x00007fa303eb2176 in __cxa_throw () from /usr/lib/libstdc++.so.6
#9  0x000000000055f5ab in common_chat_peg_parse(common_peg_arena const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, common_chat_parser_params const&) ()
#10 0x000000000055ebd6 in common_chat_parse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, common_chat_parser_params const&) ()
#11 0x0000000000441c38 in task_result_state::update_chat_msg(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::vector<common_chat_msg_diff, std::allocator<common_chat_msg_diff> >&) ()
#12 0x000000000046cb68 in server_task_result_cmpl_partial::update(task_result_state&) ()
#13 0x00000000004a3abf in server_response_reader::next(std::function<bool ()> const&) ()
#14 0x00000000004294b7 in cli_context::generate_completion[abi:cxx11](result_timings&) ()
#15 0x0000000000424c38 in main ()
[Inferior 1 (process 19535) detached]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to parse input at pos 0

Am I doing something wrong or is there stlil a similar bug in SYCL (or other) backends?

i just downloaded and compiled the latest version, all runs fine now... maybe you want to try that first too

[jorismak]

build : b8157-2943210c1

I have.

[Art4es]

I think I got the same issue with latest build b8157 on windows with cuda 12 and 13

start server with command:

.\llama\llama-server.exe ^
--model .\models\unsloth\Qwen3.5-35B-A3B\Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf ^
--alias "Qwen3.5-35B" ^
--jinja ^
--ctx-size 229376 ^
--parallel 1 ^
-fa on ^
-ub 2048 ^
-b 2048 ^
-ngl 99 ^
--temp 0.7 ^
--top-p 0.8 ^
--top-k 20 ^
--min-p 0.01 ^
--presence-penalty 1.5 ^
--repeat-penalty 1.0 ^
--fit off ^
-ot ".([3-9][0-9]).ffn_.*_exps.=CPU" ^
-Cb 0xFF ^
--cpu-strict 1 ^
--threads 16 ^
--cpu-range 0-15 ^
--no-mmap ^
--host 0.0.0.0 ^
--port 9090 ^
--verbose-prompt ^
--log-verbosity 4 ^
--chat-template-kwargs "{"enable_thinking": false}"

Call /init via claude. After some processing just silently crash.

...


res send: sending result for task id = 509
res send: task id = 509 pushed to result queue
slot release: id 0 | task 509 | stop processing: n_tokens = 485, truncated = 0
slot reset: id 0 | task 509 |
srv update_chat_: Parsing chat message: <is_displaying_contents>
false
</is_displaying_contents>

Parsing input with format peg-constructed: false Parsed message: {"role":"assistant","content":"\nfalse\n\n\n\n"} que pop_deferred: pop deferred task, id_task = 510 srv update_slots: run slots completed que start_loop: waiting for new tasks que start_loop: processing new tasks que start_loop: processing task, id = 510 srv operator(): http: streamed chunk: event: content_block_stop data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":26}}

event: message_stop
data: {"type":"message_stop"}

srv operator(): all results received, terminating stream
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = 35017089
srv get_availabl: updating prompt cache
srv operator(): http: stream ended
res remove_waiti: remove task 509 from waiting list. current waiting = 2 (before remove)
srv stop: all tasks already finished, no need to cancel
srv prompt_save: - saving prompt with length 485, total state size = 72.292 MiB
srv load: - looking for better prompt, base f_keep = 0.006, sim = 0.000
srv load: - found better prompt with f_keep = 1.000, sim = 0.996

this is the end of the terminal message

[pierreguillot]

I encountered the same problem. It seems to me that the model generates JSON tool call parameters such as True or False for booleans that are not supported by the parser, which only recognizes lowercase true or false. I solved the problem by “forcing” the model to use lowercase letters for booleans (but I use my own C++ integration of llama.cpp).