Eval bug: GGML_ASSERT(id >= 0 && id < n_expert) when running GPT-OSS 20B at any quantization and with any flags

[HyperWinX]

Name and Version

➜ build git:(master) bin/llama-server --version
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 580 Series (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
version: 7713 (4150da9)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 5 5600G + RX 580 8GB

Models

Unsloth GPT-OSS 20B Q4_K_XL

Problem description & steps to reproduce

Running mentioned model with any set of flags results in an error. On some configurations it can result in the same error at warmup step, on some - only when you run the model. Tested on Qwen3 4B Instruct - works just fine.

First Bad Commit

Commit 2038101bd9b1dcf45b5410b969fbc5206e25d993 (#18166), tag b7668 introduced that behavior.

Relevant log output

Logs

➜  build git:(master) bin/llama-server -ngl 12 -c 4096 -m ~/d/gpt-oss-20b-UD-Q4_K_XL.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 580 Series (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 7713 (4150da9a9) with GNU 15.2.1 for Linux x86_64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12

system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

init: using 11 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/home/hyperwin/d/gpt-oss-20b-UD-Q4_K_XL.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 5855 MiB of device memory vs. 7282 MiB of free device memory
llama_params_fit_impl: will leave 1426 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.60 seconds
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon RX 580 Series (RADV POLARIS10)) (0000:01:00.0) - 7282 MiB free
llama_model_loader: direct I/O is enabled, disabling mmap
llama_model_loader: loaded meta data with 37 key-value pairs and 459 tensors from /home/hyperwin/d/gpt-oss-20b-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gpt-oss
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gpt-Oss-20B
llama_model_loader: - kv   3:                           general.basename str              = Gpt-Oss-20B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 20B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   8:                               general.tags arr[str,2]       = ["vllm", "text-generation"]
llama_model_loader: - kv   9:                        gpt-oss.block_count u32              = 24
llama_model_loader: - kv  10:                     gpt-oss.context_length u32              = 131072
llama_model_loader: - kv  11:                   gpt-oss.embedding_length u32              = 2880
llama_model_loader: - kv  12:                gpt-oss.feed_forward_length u32              = 2880
llama_model_loader: - kv  13:               gpt-oss.attention.head_count u32              = 64
llama_model_loader: - kv  14:            gpt-oss.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                     gpt-oss.rope.freq_base f32              = 150000.000000
llama_model_loader: - kv  16:   gpt-oss.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                       gpt-oss.expert_count u32              = 32
llama_model_loader: - kv  18:                  gpt-oss.expert_used_count u32              = 4
llama_model_loader: - kv  19:               gpt-oss.attention.key_length u32              = 64
llama_model_loader: - kv  20:             gpt-oss.attention.value_length u32              = 64
llama_model_loader: - kv  21:           gpt-oss.attention.sliding_window u32              = 128
llama_model_loader: - kv  22:         gpt-oss.expert_feed_forward_length u32              = 2880
llama_model_loader: - kv  23:                  gpt-oss.rope.scaling.type str              = yarn
llama_model_loader: - kv  24:                gpt-oss.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  25: gpt-oss.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = gpt-4o
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,201088]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,201088]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,446189]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 199998
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 200002
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 200017
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {# Chat template fixes by Unsloth #}\n...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                          general.file_type u32              = 15
llama_model_loader: - type  f32:  289 tensors
llama_model_loader: - type q5_0:   60 tensors
llama_model_loader: - type q8_0:   14 tensors
llama_model_loader: - type q4_K:   14 tensors
llama_model_loader: - type q6_K:   10 tensors
llama_model_loader: - type mxfp4:   72 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 11.04 GiB (4.54 BPW) 
load: 0 unused tokens
load: printing all EOG tokens:
load:   - 199999 ('<|endoftext|>')
load:   - 200002 ('<|return|>')
load:   - 200007 ('<|end|>')
load:   - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>', or '<|calls|>' and '<|flush|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1.3332 MB
print_info: arch             = gpt-oss
print_info: vocab_only       = 0
print_info: no_alloc         = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2880
print_info: n_embd_inp       = 2880
print_info: n_layer          = 24
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 128
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 2880
print_info: n_expert         = 32
print_info: n_expert_used    = 4
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = yarn
print_info: freq_base_train  = 150000.0
print_info: freq_scale_train = 0.03125
print_info: freq_base_swa    = 150000.0
print_info: freq_scale_swa   = 0.03125
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_yarn_log_mul= 0.0000
print_info: rope_finetuned   = unknown
print_info: model type       = 20B
print_info: model params     = 20.91 B
print_info: general.name     = Gpt-Oss-20B
print_info: n_ff_exp         = 2880
print_info: vocab type       = BPE
print_info: n_vocab          = 201088
print_info: n_merges         = 446189
print_info: BOS token        = 199998 '<|startoftext|>'
print_info: EOS token        = 200002 '<|return|>'
print_info: EOT token        = 199999 '<|endoftext|>'
print_info: PAD token        = 200017 '<|reserved_200017|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 199999 '<|endoftext|>'
print_info: EOG token        = 200002 '<|return|>'
print_info: EOG token        = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 11 repeating layers to GPU
load_tensors: offloaded 12/25 layers to GPU
load_tensors:      Vulkan0 model buffer size =  5237.01 MiB
load_tensors:  Vulkan_Host model buffer size =  6073.18 MiB
..............................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|return|> logit bias = -inf
common_init_result: added <|call|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = true
llama_context: freq_base     = 150000.0
llama_context: freq_scale    = 0.03125
llama_context: n_ctx_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     3.07 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 4096 cells
llama_kv_cache:        CPU KV buffer size =    48.00 MiB
llama_kv_cache:    Vulkan0 KV buffer size =    48.00 MiB
llama_kv_cache: size =   96.00 MiB (  4096 cells,  12 layers,  4/1 seqs), K (f16):   48.00 MiB, V (f16):   48.00 MiB
llama_kv_cache_iswa: creating     SWA KV cache, size = 1024 cells
llama_kv_cache:        CPU KV buffer size =    14.00 MiB
llama_kv_cache:    Vulkan0 KV buffer size =    10.00 MiB
llama_kv_cache: size =   24.00 MiB (  1024 cells,  12 layers,  4/1 seqs), K (f16):   12.00 MiB, V (f16):   12.00 MiB
llama_context: layer 0 is assigned to device CPU but the Flash Attention tensor is assigned to device Vulkan0 (usually due to missing support)
llama_context: Flash Attention was auto, set to disabled
llama_context:    Vulkan0 compute buffer size =   560.63 MiB
llama_context: Vulkan_Host compute buffer size =    55.03 MiB
llama_context: graph nodes  = 1470
llama_context: graph splits = 301 (with bs=512), 2 (with bs=1)
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv    load_model: initializing slots, n_slots = 4
slot   load_model: id  0 | task -1 | new slot, n_ctx = 4096
slot   load_model: id  1 | task -1 | new slot, n_ctx = 4096
slot   load_model: id  2 | task -1 | new slot, n_ctx = 4096
slot   load_model: id  3 | task -1 | new slot, n_ctx = 4096
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv    load_model: thinking = 0
<chat template, prompt, tool definitions>
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv  update_slots: all slots are idle
srv  log_server_r: request: GET / 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /v1/models 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  params_from_: Chat format: GPT-OSS
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, task.n_tokens = 69
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 5, batch.n_tokens = 5, progress = 0.072464
slot update_slots: id  3 | task 0 | n_tokens = 5, memory_seq_rm [5, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 69, batch.n_tokens = 64, progress = 1.000000
slot update_slots: id  3 | task 0 | prompt done, n_tokens = 69, batch.n_tokens = 64
slot init_sampler: id  3 | task 0 | init sampler, took 0.02 ms, tokens: text = 69, total = 69
/home/hyperwin/llama.cpp/ggml/src/ggml-backend.cpp:1517: GGML_ASSERT(id >= 0 && id < n_expert) failed
[New LWP 261727]
[New LWP 261726]
[New LWP 261725]
[New LWP 261724]
[New LWP 261723]
[New LWP 259342]
[New LWP 259341]
[New LWP 259340]
[New LWP 259339]
[New LWP 259338]
[New LWP 259337]
[New LWP 259336]
[New LWP 259335]
[New LWP 259334]
[New LWP 259333]
[New LWP 259332]
[New LWP 259331]
[New LWP 259330]
[New LWP 259328]
[New LWP 259327]

This GDB supports auto-downloading debuginfo from the following URLs:
  <ima:enforcing>
  <https://debuginfod.fedoraproject.org/>
  <ima:ignore>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f6cd4fe4982 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007f6cd4fe4982 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007f6cd4fd8c3c in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007f6cd4fd8c84 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007f6cd5048b4f in wait4 () from /lib64/libc.so.6
#4  0x00007f6cd8ec49a3 in ggml_print_backtrace () from /home/hyperwin/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007f6cd8ec4aef in ggml_abort () from /home/hyperwin/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007f6cd8edf5f9 in ggml_backend_sched_graph_compute_async () from /home/hyperwin/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007f6cd8c48ab0 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/hyperwin/llama.cpp/build/bin/libllama.so.0
#8  0x00007f6cd8c4a7a5 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/hyperwin/llama.cpp/build/bin/libllama.so.0
#9  0x00007f6cd8c51639 in llama_context::decode(llama_batch const&) () from /home/hyperwin/llama.cpp/build/bin/libllama.so.0
#10 0x00007f6cd8c52dee in llama_decode () from /home/hyperwin/llama.cpp/build/bin/libllama.so.0
#11 0x00000000004fdaa9 in server_context_impl::update_slots() ()
#12 0x0000000000539a4f in server_queue::start_loop(long) ()
#13 0x000000000046b0d9 in main ()
[Inferior 1 (process 259326) detached]
[1]    259326 IOT instruction (core dumped)  bin/llama-server -ngl 12 -c 4096 -m ~/d/gpt-oss-20b-UD-Q4_K_XL.gguf

[jeffbolznv]

Is this a regression? Can you bisect when it started?

[HyperWinX]

Is this a regression? Can you bisect when it started?

I think it is. Running bisect at the moment. A few weeks ago i was running the same model with the same GPU, but on Ryzen 5 2600X.

[HyperWinX]

Is this a regression? Can you bisect when it started?

Updated the issue with the first bad commit.

[jeffbolznv]

Thanks. I don't get the assertion failure, but the model output is corrupt with -ngl 12. Goes away with --no-direct-io. CC @JTischbein.

[wolfsilver00]

After a full system update and with models that were running at least 2 weeks ago (last time I used them on this pc), im getting this now:

Error: 500 Internal Server Error: llama runner process has terminated: GGML_ASSERT(max_blocks_per_sm > 0) failed

With any model..

Im running it on arch using the package

Edit: reverting to 0.12.11 works to fix the issue

[jeffbolznv]

Error: 500 Internal Server Error: llama runner process has terminated: GGML_ASSERT(max_blocks_per_sm > 0) failed

I think this is a cuda error, unrelated to this bug.

[pwilkin]

Adding to watchlist for #18766

[jeffbolznv]

@JTischbein will you be able to debug this?

[JTischbein]

I get a different error on DGX Spark with the Vulkan Backend and the same command void ggml_compute_forward_soft_max_f32(const ggml_compute_params*, ggml_tensor*): Assertion sum > 0.0' failed.`.
Using this diff it is running now. I will dig deeper into the implications this change has. @jeffbolznv

diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index 94c47dc24..06e7de6c2 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -2725,7 +2725,7 @@ bool llama_model::load_tensors(llama_model_loader & ml) {
 
             // avoid using a host buffer when using mmap
             auto * buft_dev = ggml_backend_buft_get_device(buft);
-            if (ml.use_mmap && buft_dev && buft == ggml_backend_dev_host_buffer_type(buft_dev)) {
+            if (buft_dev && buft == ggml_backend_dev_host_buffer_type(buft_dev)) {
                 auto * cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU);
                 if (!cpu_dev) {
                     throw std::runtime_error("no CPU backend found");

[0cc4m]

I think the implication of that change is that you stop using device host buffers always, not just when using mmap. That is not gonna work, host buffers are needed for fast transfers to and from GPU.