Low t/s for Qwen3 on M3 ultra

[otarkhan]

Describe the bug

I'm getting super low t/s (< 5t/s for Qwen/Qwen3-30B-A3B) on mac studio with M3 ultra and 512GB unified memory. With llama.cpp, I'm getting over 50t/s. Also, running the model in gguf format isn't working: "called Result::unwrap() on an Err value: Unknown GGUF architecture qwen3moe"

Latest commit or version

380da23

[EricLBuehler]

Hi @otarkhan! We don't have GGUF support yet as I've been busy with some other models. Are you using ISQ?

[otarkhan]

Even with ISQ it's still much slower than llama.cpp

[EricLBuehler]

Hi @otarkhan just to make sure, are you compiling with the metal feature?

On my M3 max I'm getting 70+ T/s for that model.

[otarkhan]

I did compile it with metal yes.

Interestingly enough, I'm currently either getting 50+ t/s or 0.5 t/s. When I start getting the slow generation, I restart the Mac and it starts working perfectly again.

./target/release/mistralrs-server -i --isq 4 plain -m Qwen/Qwen3-30B-A3B

[otarkhan]

It's more obvious here (with server):

2025-06-02T18:46:26.582195Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.60, Prefix cache hitrate 0.00%, 0 running, 0 waiting
2025-06-02T18:46:36.592309Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 78.40, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:46:41.597367Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 88.40, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:46:46.602423Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 88.00, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:46:51.607484Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 86.80, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:46:56.612542Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 87.20, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:47:01.614824Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 86.40, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:47:06.618582Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 86.40, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:47:11.623125Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 85.20, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:47:16.628193Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 73.60, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:47:21.633250Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 54.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:47:26.643320Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 24.80, Prefix cache hitrate 0.00%, 0 running, 0 waiting
2025-06-02T18:48:11.726065Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 4.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:48:16.736125Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.60, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:48:21.746173Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:48:26.756222Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.60, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:48:31.766287Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T18:48:36.776348Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 4.80, Prefix cache hitrate 0.00%, 2 running, 0 waiting
2025-06-02T18:48:41.786424Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.80, Prefix cache hitrate 0.00%, 2 running, 0 waiting

[otarkhan]

I think it has something to do with memory assigned/used. I could reproduce it (on a different model) by assigning different --pa-gpu-mem sizes

╰─$ mistralrs-server --port 1235 --paged-attn --max-seqs 10 --pa-gpu-mem 50000 --isq 8 plain -m Qwen/Qwen3-0.6B
2025-06-02T22:24:03.924525Z  INFO mistralrs_server: avx: false, neon: true, simd128: false, f16c: false
2025-06-02T22:24:03.924563Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-06-02T22:24:03.924574Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-06-02T22:24:03.924605Z  INFO hf_hub: Using token file found "......."
2025-06-02T22:24:03.924681Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:03.924759Z  INFO mistralrs_core::pipeline::normal: Loading `config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:04.527149Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model.safetensors"]
2025-06-02T22:24:04.738523Z  INFO mistralrs_core::pipeline::normal: Loading `generation_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:05.174847Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:05.390158Z  INFO mistralrs_quant::utils::log: Automatic loader type determined to be `qwen3`
2025-06-02T22:24:05.390213Z  INFO mistralrs_core::pipeline::normal: Prompt chunk size is 1024.
2025-06-02T22:24:05.402161Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:05.408816Z  INFO mistralrs_core::pipeline::loaders: Using automatic device mapping parameters: text[max_seq_len: 4096, max_batch_size: 1].
2025-06-02T22:24:05.408892Z  INFO mistralrs_quant::utils::log: Model has 28 repeating layers.
2025-06-02T22:24:05.408906Z  INFO mistralrs_quant::utils::log: Loading model according to the following repeating layer mappings:
2025-06-02T22:24:05.408916Z  INFO mistralrs_quant::utils::log: Layers 0-27: metal[4294970248] (464 GB)
2025-06-02T22:24:05.412868Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:05.412928Z  INFO mistralrs_core::pipeline::normal: Model config: Config { vocab_size: 151936, hidden_size: 1024, intermediate_size: 3072, num_hidden_layers: 28, num_attention_heads: 16, num_key_value_heads: 8, hidden_act: Silu, max_position_embeddings: 40960, rms_norm_eps: 1e-6, rope_theta: 1000000.0, sliding_window: None, head_dim: Some(128), quantization_config: None, tie_word_embeddings: true, max_window_layers: 28, use_sliding_window: false }
2025-06-02T22:24:05.414466Z  INFO mistralrs_core::pipeline::normal: Applying ISQ to Some(AFQ8)
2025-06-02T22:24:05.414659Z  INFO mistralrs_core::utils::varbuilder_utils: Loading model using mmap strategy.
2025-06-02T22:24:05.796446Z  INFO mistralrs_core::paged_attention: Allocating 50000 MB for PagedAttention KV cache per GPU
2025-06-02T22:24:05.796459Z  INFO mistralrs_core::paged_attention: Using PagedAttention with block size 32 and 14285 GPU blocks: available context length is 457120 tokens
2025-06-02T22:24:06.117365Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<|endoftext|>", eos_toks = "<|im_end|>", "<|endoftext|>", unk_tok = `None`
2025-06-02T22:24:06.124463Z  INFO mistralrs_server: Model loaded.
2025-06-02T22:24:06.124501Z  INFO mistralrs_core: Beginning dummy run.
2025-06-02T22:24:06.125009Z  INFO mistralrs_core::prefix_cacher: PrefixCacherV2 is enabled. Expect higher multi-turn throughput for both text and multimodal.
2025-06-02T22:24:08.658520Z  INFO mistralrs_core: Dummy run completed in 2.534002333s.
2025-06-02T22:24:08.658766Z  INFO mistralrs_server: Serving on http://0.0.0.0:1235.
2025-06-02T22:24:11.135075Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 67.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:16.145133Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 174.60, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:21.155202Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 76.60, Prefix cache hitrate 0.00%, 0 running, 0 waiting
^C^C

╰─$ mistralrs-server --port 1235 --paged-attn --max-seqs 10 --pa-gpu-mem 100000 --isq 8 plain -m Qwen/Qwen3-0.6B
2025-06-02T22:24:36.529400Z  INFO mistralrs_server: avx: false, neon: true, simd128: false, f16c: false
2025-06-02T22:24:36.529442Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-06-02T22:24:36.529453Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-06-02T22:24:36.529484Z  INFO hf_hub: Using token file found "......."
2025-06-02T22:24:36.529558Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:36.529635Z  INFO mistralrs_core::pipeline::normal: Loading `config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:36.771620Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model.safetensors"]
2025-06-02T22:24:36.988904Z  INFO mistralrs_core::pipeline::normal: Loading `generation_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:37.417649Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:37.636436Z  INFO mistralrs_quant::utils::log: Automatic loader type determined to be `qwen3`
2025-06-02T22:24:37.636489Z  INFO mistralrs_core::pipeline::normal: Prompt chunk size is 1024.
2025-06-02T22:24:37.648185Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:37.655080Z  INFO mistralrs_core::pipeline::loaders: Using automatic device mapping parameters: text[max_seq_len: 4096, max_batch_size: 1].
2025-06-02T22:24:37.655151Z  INFO mistralrs_quant::utils::log: Model has 28 repeating layers.
2025-06-02T22:24:37.655163Z  INFO mistralrs_quant::utils::log: Loading model according to the following repeating layer mappings:
2025-06-02T22:24:37.655175Z  INFO mistralrs_quant::utils::log: Layers 0-27: metal[4294970248] (464 GB)
2025-06-02T22:24:37.659105Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:37.659173Z  INFO mistralrs_core::pipeline::normal: Model config: Config { vocab_size: 151936, hidden_size: 1024, intermediate_size: 3072, num_hidden_layers: 28, num_attention_heads: 16, num_key_value_heads: 8, hidden_act: Silu, max_position_embeddings: 40960, rms_norm_eps: 1e-6, rope_theta: 1000000.0, sliding_window: None, head_dim: Some(128), quantization_config: None, tie_word_embeddings: true, max_window_layers: 28, use_sliding_window: false }
2025-06-02T22:24:37.660753Z  INFO mistralrs_core::pipeline::normal: Applying ISQ to Some(AFQ8)
2025-06-02T22:24:37.660949Z  INFO mistralrs_core::utils::varbuilder_utils: Loading model using mmap strategy.
2025-06-02T22:24:38.036989Z  INFO mistralrs_core::paged_attention: Allocating 100000 MB for PagedAttention KV cache per GPU
2025-06-02T22:24:38.037001Z  INFO mistralrs_core::paged_attention: Using PagedAttention with block size 32 and 28571 GPU blocks: available context length is 914272 tokens
2025-06-02T22:24:38.359096Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<|endoftext|>", eos_toks = "<|im_end|>", "<|endoftext|>", unk_tok = `None`
2025-06-02T22:24:38.366171Z  INFO mistralrs_server: Model loaded.
2025-06-02T22:24:38.366210Z  INFO mistralrs_core: Beginning dummy run.
2025-06-02T22:24:38.366923Z  INFO mistralrs_core::prefix_cacher: PrefixCacherV2 is enabled. Expect higher multi-turn throughput for both text and multimodal.
2025-06-02T22:24:43.377006Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:44.766477Z  INFO mistralrs_core: Dummy run completed in 6.4002335s.
2025-06-02T22:24:44.766737Z  INFO mistralrs_server: Serving on http://0.0.0.0:1235.
2025-06-02T22:24:48.387084Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 4.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:53.397150Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:58.407244Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:03.417364Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:08.427454Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:13.437527Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:18.442288Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
^C^C^C

[e1732a364fed]

Yes I can confirm this. high --pa-gpu-mem will result in super slow qwen3 inference.

I was testing on the non-moe qwen3 ones. So it's not related to moe.

And actually I was testing on MLX. So it's not gguf related either.

Also, I need lower --pa-gpu-mem, 50000 is still too high. Also I find that even with no --pa-gpu-mem parameter, sometimes it's still super slow ( I checked and find that in this case, only one core cpu is used).

It will be ok if you don't give --paged-attn at all (so no paged-attention enabled).

[guoqingbao]

I think it has something to do with memory assigned/used. I could reproduce it (on a different model) by assigning different --pa-gpu-mem sizes

╰─$ mistralrs-server --port 1235 --paged-attn --max-seqs 10 --pa-gpu-mem 50000 --isq 8 plain -m Qwen/Qwen3-0.6B
2025-06-02T22:24:03.924525Z  INFO mistralrs_server: avx: false, neon: true, simd128: false, f16c: false
2025-06-02T22:24:03.924563Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-06-02T22:24:03.924574Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-06-02T22:24:03.924605Z  INFO hf_hub: Using token file found "......."
2025-06-02T22:24:03.924681Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:03.924759Z  INFO mistralrs_core::pipeline::normal: Loading `config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:04.527149Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model.safetensors"]
2025-06-02T22:24:04.738523Z  INFO mistralrs_core::pipeline::normal: Loading `generation_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:05.174847Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:05.390158Z  INFO mistralrs_quant::utils::log: Automatic loader type determined to be `qwen3`
2025-06-02T22:24:05.390213Z  INFO mistralrs_core::pipeline::normal: Prompt chunk size is 1024.
2025-06-02T22:24:05.402161Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:05.408816Z  INFO mistralrs_core::pipeline::loaders: Using automatic device mapping parameters: text[max_seq_len: 4096, max_batch_size: 1].
2025-06-02T22:24:05.408892Z  INFO mistralrs_quant::utils::log: Model has 28 repeating layers.
2025-06-02T22:24:05.408906Z  INFO mistralrs_quant::utils::log: Loading model according to the following repeating layer mappings:
2025-06-02T22:24:05.408916Z  INFO mistralrs_quant::utils::log: Layers 0-27: metal[4294970248] (464 GB)
2025-06-02T22:24:05.412868Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:05.412928Z  INFO mistralrs_core::pipeline::normal: Model config: Config { vocab_size: 151936, hidden_size: 1024, intermediate_size: 3072, num_hidden_layers: 28, num_attention_heads: 16, num_key_value_heads: 8, hidden_act: Silu, max_position_embeddings: 40960, rms_norm_eps: 1e-6, rope_theta: 1000000.0, sliding_window: None, head_dim: Some(128), quantization_config: None, tie_word_embeddings: true, max_window_layers: 28, use_sliding_window: false }
2025-06-02T22:24:05.414466Z  INFO mistralrs_core::pipeline::normal: Applying ISQ to Some(AFQ8)
2025-06-02T22:24:05.414659Z  INFO mistralrs_core::utils::varbuilder_utils: Loading model using mmap strategy.
2025-06-02T22:24:05.796446Z  INFO mistralrs_core::paged_attention: Allocating 50000 MB for PagedAttention KV cache per GPU
2025-06-02T22:24:05.796459Z  INFO mistralrs_core::paged_attention: Using PagedAttention with block size 32 and 14285 GPU blocks: available context length is 457120 tokens
2025-06-02T22:24:06.117365Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<|endoftext|>", eos_toks = "<|im_end|>", "<|endoftext|>", unk_tok = `None`
2025-06-02T22:24:06.124463Z  INFO mistralrs_server: Model loaded.
2025-06-02T22:24:06.124501Z  INFO mistralrs_core: Beginning dummy run.
2025-06-02T22:24:06.125009Z  INFO mistralrs_core::prefix_cacher: PrefixCacherV2 is enabled. Expect higher multi-turn throughput for both text and multimodal.
2025-06-02T22:24:08.658520Z  INFO mistralrs_core: Dummy run completed in 2.534002333s.
2025-06-02T22:24:08.658766Z  INFO mistralrs_server: Serving on http://0.0.0.0:1235.
2025-06-02T22:24:11.135075Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 67.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:16.145133Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 174.60, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:21.155202Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 76.60, Prefix cache hitrate 0.00%, 0 running, 0 waiting
^C^C

╰─$ mistralrs-server --port 1235 --paged-attn --max-seqs 10 --pa-gpu-mem 100000 --isq 8 plain -m Qwen/Qwen3-0.6B
2025-06-02T22:24:36.529400Z  INFO mistralrs_server: avx: false, neon: true, simd128: false, f16c: false
2025-06-02T22:24:36.529442Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-06-02T22:24:36.529453Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-06-02T22:24:36.529484Z  INFO hf_hub: Using token file found "......."
2025-06-02T22:24:36.529558Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:36.529635Z  INFO mistralrs_core::pipeline::normal: Loading `config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:36.771620Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model.safetensors"]
2025-06-02T22:24:36.988904Z  INFO mistralrs_core::pipeline::normal: Loading `generation_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:37.417649Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer_config.json` at `Qwen/Qwen3-0.6B`
2025-06-02T22:24:37.636436Z  INFO mistralrs_quant::utils::log: Automatic loader type determined to be `qwen3`
2025-06-02T22:24:37.636489Z  INFO mistralrs_core::pipeline::normal: Prompt chunk size is 1024.
2025-06-02T22:24:37.648185Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:37.655080Z  INFO mistralrs_core::pipeline::loaders: Using automatic device mapping parameters: text[max_seq_len: 4096, max_batch_size: 1].
2025-06-02T22:24:37.655151Z  INFO mistralrs_quant::utils::log: Model has 28 repeating layers.
2025-06-02T22:24:37.655163Z  INFO mistralrs_quant::utils::log: Loading model according to the following repeating layer mappings:
2025-06-02T22:24:37.655175Z  INFO mistralrs_quant::utils::log: Layers 0-27: metal[4294970248] (464 GB)
2025-06-02T22:24:37.659105Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-06-02T22:24:37.659173Z  INFO mistralrs_core::pipeline::normal: Model config: Config { vocab_size: 151936, hidden_size: 1024, intermediate_size: 3072, num_hidden_layers: 28, num_attention_heads: 16, num_key_value_heads: 8, hidden_act: Silu, max_position_embeddings: 40960, rms_norm_eps: 1e-6, rope_theta: 1000000.0, sliding_window: None, head_dim: Some(128), quantization_config: None, tie_word_embeddings: true, max_window_layers: 28, use_sliding_window: false }
2025-06-02T22:24:37.660753Z  INFO mistralrs_core::pipeline::normal: Applying ISQ to Some(AFQ8)
2025-06-02T22:24:37.660949Z  INFO mistralrs_core::utils::varbuilder_utils: Loading model using mmap strategy.
2025-06-02T22:24:38.036989Z  INFO mistralrs_core::paged_attention: Allocating 100000 MB for PagedAttention KV cache per GPU
2025-06-02T22:24:38.037001Z  INFO mistralrs_core::paged_attention: Using PagedAttention with block size 32 and 28571 GPU blocks: available context length is 914272 tokens
2025-06-02T22:24:38.359096Z  INFO mistralrs_core::pipeline::chat_template: bos_toks = "<|endoftext|>", eos_toks = "<|im_end|>", "<|endoftext|>", unk_tok = `None`
2025-06-02T22:24:38.366171Z  INFO mistralrs_server: Model loaded.
2025-06-02T22:24:38.366210Z  INFO mistralrs_core: Beginning dummy run.
2025-06-02T22:24:38.366923Z  INFO mistralrs_core::prefix_cacher: PrefixCacherV2 is enabled. Expect higher multi-turn throughput for both text and multimodal.
2025-06-02T22:24:43.377006Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:44.766477Z  INFO mistralrs_core: Dummy run completed in 6.4002335s.
2025-06-02T22:24:44.766737Z  INFO mistralrs_server: Serving on http://0.0.0.0:1235.
2025-06-02T22:24:48.387084Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 4.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:53.397150Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:24:58.407244Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:03.417364Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:08.427454Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:13.437527Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.40, Prefix cache hitrate 0.00%, 1 running, 0 waiting
2025-06-02T22:25:18.442288Z  INFO mistralrs_core::engine::logger: Throughput (T/s) 0.20, Prefix cache hitrate 0.00%, 1 running, 0 waiting
^C^C^C

A 50,000MB KV cache is significantly higher than the typical setting. Usually, we allocate around 5GB to 10GB, which is sufficient to cover the maximum model length for a single user in most cases. Setting the KV cache to 50GB can lead to frequent memory swapping, which degrades performance. On the other extreme, disabling the KV cache entirely (or not using paged attention) will also negatively impact performance, especially for long-context inference.

[otarkhan]

5-10GB isn't enough for parallel processing + large context.
Also, is this behavior normal? Does it make sense to drop that much because we doubled the KV cache?

[otarkhan]

@EricLBuehler Can you confirm that this is a normal behavior? It doesn't seem like

[guoqingbao]

5-10GB isn't enough for parallel processing + large context. Also, is this behavior normal? Does it make sense to drop that much because we doubled the KV cache?

For Qwen3, the actual maximum context length is 40,960 (refer to max_position_embeddings in config.json), not the max_model_len of 131,072 claimed in tokenizer_config.json (see issue #1361). For context lengths longer than 40,960 (which we don't support because it requires a sliding_window config provided by the model author — the same applies to vLLM, as noted here), it may result in unpredictable behavior.

If you set a 10GB KV cache, it will result in 2,275 GPU KV cache blocks with a block size of 32 (which corresponds to a maximum context length of 72,800 for single-user).

Here’s the running log from Candle-VLLM; I believe the logic is similar to that in Mistral.rs.

RUST_LOG=info cargo run --release --features cuda,nccl -- --kvcache-mem-gpu 10240 --log --port 2100 --dtype bf16 --weight-file /home/data/DeepSeek-R1-0528-Qwen3-8B-Q2_K.gguf qwen3 --quant gguf --temperature 0. --penalty 1.0

2025-06-09 17:07:51 INFO candle_vllm::openai::pipelines::pipeline PipelineConfig { max_model_len: 131072, default_max_tokens: 16384, penalty: 1.0, repeat_last_n: 64, temperature: Some(0.0), top_k: None, top_p: None, thinking: Some(false) }
2025-06-09 17:07:51 INFO candle_vllm::openai::pipelines::pipeline SpecificConfig { repeat_last_n: None, temperature: Some(0.0), top_k: None, top_p: None, penalty: Some(1.0), max_gen_tokens: None, quant: Some("gguf"), num_experts_offload_per_rank: None, thinking: false }
2025-06-09 17:07:51 WARN candle_vllm::openai::pipelines::pipeline stop_token_ids [151645]
2025-06-09 17:07:51 INFO candle_vllm parallel model: multithread!
2025-06-09 17:07:51 INFO candle_vllm Cache config CacheConfig { block_size: 32, num_gpu_blocks: Some(2275), num_cpu_blocks: Some(28), fully_init: true, dtype: BF16 }
2025-06-09 17:07:51 INFO candle_vllm Maximum Model Length (affected by `--kvcache-mem-gpu` and the number of ranks):
-> Batch 1: 72800
-> Batch 8: 9100
2025-06-09 17:07:51 WARN candle_vllm Server started at http://0.0.0.0:2100.

I'm not sure if M3 Ultra is typically used for serving multiple users. Normally, we use multi-rank GPUs to reduce per-rank KV cache usage. For example, if we have 8 GPUs, the total available KV cache will be 72,800 * 8 for a 10GB KV-cache configuration.

Shall we print the actual maximum model length in Mistral.rs? @EricLBuehler

[EricLBuehler]

@otarkhan yes, it is expected. If you increase the paged attention KV cache allocation dramatically, it'll cause macos to use non-GPU memory in my experience, which is a dramatic slowdown.

@guoqingbao we actually already have this:

mistral.rs/mistralrs-core/src/paged_attention/mod.rs

Line 145 in 408c888

info!("Using PagedAttention with block size {block_size} and {num_gpu_blocks} GPU blocks: available context length is {} tokens", num_gpu_blocks*block_size);