Creating a `uqff` for qwen2.5-vl-7b fails

[terhechte]

Describe the bug

Hey, I'm trying to create a uqff for the Qwen 2.5 vl 7b model (I also tried 32b with the same error):

target/release/mistralrs-server --isq Q4K -i vision-plain -m Qwen/Qwen2.5-VL-7B-Instruct --write-uqff qwen2.5-vl-7b-q4k.uqff -a qwen2_5vl

But when I do that, I get this error:

2025-03-27T09:00:24.204281Z  INFO mistralrs_core::pipeline::isq: Serializing 197 ISQ tensors to `qwen2.5-vl-7b-q4k.uqff`.
Error: `QuantizedSerde::serialize_with_bias` is not supported.

I'm on latest main branch.

[EricLBuehler]

@terhechte thanks for the issue. #1250 should fix this, can you please retry after git pull?

[terhechte]

Thanks for the fast response! This worked. I'm running into another issue now though:

target/release/mistralrs-server --isq Q4K -i vision-plain -m Qwen/Qwen2.5-VL-7B-Instruct --write-uqff qwen2.5-vl-7b-q4k.uqff -a qwen2_5vl                                                                                                   (base)
2025-03-29T15:57:23.007970Z  INFO mistralrs_server: avx: false, neon: true, simd128: false, f16c: false
2025-03-29T15:57:23.007992Z  INFO mistralrs_server: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-03-29T15:57:23.008003Z  INFO mistralrs_server: Model kind is: normal (no adapters)
2025-03-29T15:57:23.008043Z  INFO mistralrs_core::utils::tokens: Could not load token at "/Users/terhechte/.cache/huggingface/token", using no HF token.
2025-03-29T15:57:23.008100Z  INFO mistralrs_core::pipeline::vision: Loading `tokenizer.json` at `Qwen/Qwen2.5-VL-7B-Instruct`
2025-03-29T15:57:23.008292Z  INFO mistralrs_core::pipeline::vision: Loading `config.json` at `Qwen/Qwen2.5-VL-7B-Instruct`
2025-03-29T15:57:23.160409Z  INFO mistralrs_core::pipeline::paths: Found model weight filenames ["model-00001-of-00005.safetensors", "model-00002-of-00005.safetensors", "model-00003-of-00005.safetensors", "model-00004-of-00005.safetensors", "model-00005-of-00005.safetensors"]
2025-03-29T15:57:23.285362Z  INFO mistralrs_core::pipeline::vision: Loading `generation_config.json` at `Qwen/Qwen2.5-VL-7B-Instruct`
2025-03-29T15:57:23.407063Z  INFO mistralrs_core::pipeline::vision: Loading `preprocessor_config.json` at `Qwen/Qwen2.5-VL-7B-Instruct`
2025-03-29T15:57:23.529001Z  INFO mistralrs_core::pipeline::vision: Loading `tokenizer_config.json` at `Qwen/Qwen2.5-VL-7B-Instruct`
2025-03-29T15:57:23.656324Z  INFO mistralrs_core::pipeline::vision: Loading `chat_template.json` at `Qwen/Qwen2.5-VL-7B-Instruct`
2025-03-29T15:57:23.664700Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-03-29T15:57:23.676655Z  INFO mistralrs_core::pipeline::loaders: Using automatic device mapping parameters: vision[max_seq_len: 4096, max_batch_size: 1, max_image_shape: (1024, 1024), max_num_images: 1].
2025-03-29T15:57:23.676687Z  INFO mistralrs_core::pipeline::loaders: The following sub-models will not be device mapped and will be loaded on metal[4294968349]: vision
2025-03-29T15:57:23.676736Z  INFO mistralrs_core::utils::log: Model has 28 repeating layers.
2025-03-29T15:57:23.676754Z  INFO mistralrs_core::utils::log: Loading model according to the following repeating layer mappings:
2025-03-29T15:57:23.676756Z  INFO mistralrs_core::utils::log: Layers 0-27: metal[4294968349] (96 GB)
2025-03-29T15:57:23.678108Z  INFO mistralrs_core::utils::normal: DType selected is BF16.
2025-03-29T15:57:23.678126Z  INFO mistralrs_core::pipeline::vision: Model config: Config { vocab_size: 152064, hidden_size: 3584, intermediate_size: 18944, num_hidden_layers: 28, num_attention_heads: 28, num_key_value_heads: 4, hidden_act: Silu, max_position_embeddings: 128000, rms_norm_eps: 1e-6, tie_word_embeddings: false, rope_theta: 1000000.0, use_sliding_window: false, sliding_window: Some(32768), vision_config: VisionConfig { depth: 32, hidden_size: 1280, out_hidden_size: 3584, hidden_act: Silu, intermediate_size: 3420, num_heads: 16, in_chans: 3, patch_size: 14, spatial_merge_size: 2, temporal_patch_size: 2, window_size: 112, fullatt_block_indexes: [7, 15, 23, 31] }, rope_scaling: MRopeScaling { mrope_section: [16, 24, 24] }, quantization_config: None, image_token_id: 151655, video_token_id: 151656 }
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 97/97 [00:00<00:00, 4145.43it/s]100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 98/98 [00:00<00:00, 114.86it/s]100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 97/97 [00:00<00:00, 68.35it/s]100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 436/436 [00:00<00:00, 472.18it/s]100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 0.00it/s]
2025-03-29T15:57:25.818379Z  INFO mistralrs_core::pipeline::isq: Applying in-situ quantization into Some(Q4K) to 197 tensors.
2025-03-29T15:57:25.818416Z  INFO mistralrs_core::pipeline::isq: Applying ISQ on 16 threads.
[00:00:00] [###>------------------------------------] 17/197 (0s)                                                                                                                                                                                                                         

-[AGXG16XFamilyCommandBuffer tryCoalescingPreviousComputeCommandEncoderWithConfig:nextEncoderClass:]:1091: failed assertion `A command encoder is already encoding to this command buffer'
fish: Job 1, 'target/release/mistralrs-server…' terminated by signal SIGABRT (Abort)

A command encoder is already encoding to this command buffer in AGXG16XFamilyCommandBuffer tryCoalescingPreviousComputeCommandEncoderWithConfig:nextEncoderClass:]:109

[EricLBuehler]

@terhechte thanks for this. I merged #1252 - this should fix the issue, can you please try again after git pull?