[Bug]: Quantization method specified in the model config (fp8) does not match the quantization method specified in the `quantization` argument (gguf).

[Minami-su]

Your current environment

MODEL_PATH="/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf"
LOG_FILE="vllm.log"

export VLLM_USE_V1=0
SERVED_MODEL_NAME="DeepSeek-R1-0528"
export CUDA_VISIBLE_DEVICES=2,3,4,5

运行命令

nohup vllm serve
"$MODEL_PATH"
--hf-config-path /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--tokenizer /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--served-model-name "$SERVED_MODEL_NAME"
--trust-remote-code
--port 6011
--host 0.0.0.0
--dtype auto
--max-model-len 8192
--gpu_memory_utilization 0.98
--tensor_parallel_size 4
--enable-prefix-caching
>"$LOG_FILE" 2>&1 &

🐛 Describe the bug

INFO 06-03 11:53:19 [init.py:243] Automatically detected platform cuda.
INFO 06-03 11:53:22 [init.py:31] Available plugins for group vllm.general_plugins:
INFO 06-03 11:53:22 [init.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-03 11:53:22 [init.py:36] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 06-03 11:53:24 [api_server.py:1289] vLLM API server version 0.9.0.1
INFO 06-03 11:53:24 [cli_args.py:300] non-default args: {'host': '0.0.0.0', 'port': 6011, 'tokenizer': '/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', 'trust_remote_code': True, 'hf_config_path': '/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', 'max_model_len': 8192, 'served_model_name': ['DeepSeek-R1-0528'], 'tensor_parallel_size': 4, 'gpu_memory_utilization': 0.98, 'enable_prefix_caching': True}
INFO 06-03 11:53:24 [config.py:213] Replacing legacy 'type' key with 'rope_type'
INFO 06-03 11:53:32 [config.py:793] This model supports multiple tasks: {'reward', 'generate', 'embed', 'score', 'classify'}. Defaulting to 'generate'.
Traceback (most recent call last):
File "/data/jcxy/haolu/anaconda3/envs/haolu/bin/vllm", line 8, in
sys.exit(main())
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 56, in main
args.dispatch_function(args)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 42, in cmd
uvloop.run(run_server(args))
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1324, in run_server
async with build_async_engine_client(args) as engine_client:
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 153, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 173, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 983, in create_engine_config
model_config = self.create_model_config()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 875, in create_model_config
return ModelConfig(
File "", line 42, in init
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/config.py", line 606, in post_init
self._verify_quantization()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/config.py", line 893, in _verify_quantization
raise ValueError(
ValueError: Quantization method specified in the model config (fp8) does not match the quantization method specified in the quantization argument (gguf).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[lengrongfu]

Maybe we should in gguf.py add a override_quantization_method logic, @Isotr0py what do you think?, if you see ok, i will pull request.

[Isotr0py]

Maybe we should in gguf.py add a override_quantization_method logic

Hmmm, I think --hf-overrides should have same effect, because original Deepseek-R1 model repo is a FP8 model exactly, and its quantization config is in the config.json.

@Minami-su Perhaps try to add --hf-overrides '{"quantization_config": {}}'?

[Minami-su]

@Isotr0py INFO 06-04 10:16:13 [init.py:239] Automatically detected platform cuda.
INFO 06-04 10:16:18 [api_server.py:1043] vLLM API server version 0.8.5
INFO 06-04 10:16:18 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf', config='', host='0.0.0.0', port=6011, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf', task='auto', tokenizer='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', hf_config_path='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=8192, guided_decoding_backend='xgrammar', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=4, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.98, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=True, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides={'quantization_config': {}}, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=['DeepSeek-R1-0528'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f2a58c64940>)
INFO 06-04 10:16:18 [config.py:209] Replacing legacy 'type' key with 'rope_type'
INFO 06-04 10:16:18 [config.py:456] Overriding HF config with {'quantization_config': {}}
INFO 06-04 10:16:25 [config.py:717] This model supports multiple tasks: {'reward', 'classify', 'generate', 'score', 'embed'}. Defaulting to 'generate'.
Traceback (most recent call last):
File "/data/jcxy/haolu/anaconda3/envs/haolu/bin/vllm", line 8, in
sys.exit(main())
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1099, in create_engine_config
model_config = self.create_model_config()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 987, in create_model_config
return ModelConfig(
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/config.py", line 546, in init
self._verify_quantization()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/config.py", line 816, in _verify_quantization
raise ValueError(
ValueError: Quantization method specified in the model config () does not match the quantization method specified in the quantization argument (gguf).

[Minami-su]

MODEL_PATH="/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf"
LOG_FILE="vllm.log"
export VLLM_USE_V1=0
SERVED_MODEL_NAME="DeepSeek-R1-0528"
export CUDA_VISIBLE_DEVICES=2,3,4,5

运行命令

nohup vllm serve
"$MODEL_PATH"
--hf-config-path /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--tokenizer /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--hf-overrides '{"quantization_config": {}}'
--served-model-name "$SERVED_MODEL_NAME"
--trust-remote-code
--port 6011
--host 0.0.0.0
--dtype auto
--max-model-len 8192
--gpu_memory_utilization 0.98
--tensor_parallel_size 4
--enable-prefix-caching
>"$LOG_FILE" 2>&1 &

[lengrongfu]

--hf-overrides '{"quantization_config": {"gguf"}}' @Minami-su you should use this

[Minami-su]

@lengrongfu
MODEL_PATH="/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf"
LOG_FILE="vllm.log"
export VLLM_USE_V1=0
SERVED_MODEL_NAME="DeepSeek-R1-0528"
export CUDA_VISIBLE_DEVICES=2,3

运行命令

nohup vllm serve
"$MODEL_PATH"
--hf-config-path /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--tokenizer /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--hf-overrides '{"quantization_config": {"gguf"}}'
--served-model-name "$SERVED_MODEL_NAME"
--trust-remote-code
--port 6011
--host 0.0.0.0
--dtype auto
--max-model-len 8192
--gpu_memory_utilization 0.98
--tensor_parallel_size 2
--enable-prefix-caching
>"$LOG_FILE" 2>&1 &

---

INFO 06-04 11:48:50 [init.py:239] Automatically detected platform cuda.
usage: vllm serve [model_tag] [options]
vllm serve: error: argument --hf-overrides: invalid loads value: '{"quantization_config": {"gguf"}}'

[Isotr0py]

Hmmm, perhaps you can remove the quantization_config field in config.json directly. (https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/config.json#L37-L45)

[Minami-su]

@Isotr0py I removed the quantization_config field in config.json.

MODEL_PATH="/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf"
LOG_FILE="vllm.log"
export VLLM_USE_V1=0
SERVED_MODEL_NAME="DeepSeek-R1-0528"
export CUDA_VISIBLE_DEVICES=4,5

运行命令

nohup vllm serve
"$MODEL_PATH"
--hf-config-path /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--tokenizer /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS
--served-model-name "$SERVED_MODEL_NAME"
--trust-remote-code
--port 6011
--host 0.0.0.0
--dtype auto
--max-model-len 8192
--gpu_memory_utilization 0.98
--tensor_parallel_size 2
--enable-prefix-caching
>"$LOG_FILE" 2>&1 &

INFO 06-04 13:05:25 [init.py:239] Automatically detected platform cuda.
INFO 06-04 13:05:30 [api_server.py:1043] vLLM API server version 0.8.5
INFO 06-04 13:05:30 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf', config='', host='0.0.0.0', port=6011, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf', task='auto', tokenizer='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', hf_config_path='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=8192, guided_decoding_backend='xgrammar', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.98, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=True, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=['DeepSeek-R1-0528'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7fa460450940>)
INFO 06-04 13:05:30 [config.py:209] Replacing legacy 'type' key with 'rope_type'
INFO 06-04 13:05:38 [config.py:717] This model supports multiple tasks: {'classify', 'generate', 'embed', 'reward', 'score'}. Defaulting to 'generate'.
WARNING 06-04 13:05:38 [config.py:830] gguf quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 06-04 13:05:38 [config.py:1770] Defaulting to use mp for distributed inference
INFO 06-04 13:05:42 [init.py:239] Automatically detected platform cuda.
INFO 06-04 13:05:45 [api_server.py:246] Started engine process with PID 415797
INFO 06-04 13:05:45 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.5) with config: model='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf', speculative_config=None, tokenizer='/data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=gguf, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gguf, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=DeepSeek-R1-0528, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
WARNING 06-04 13:05:46 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 06-04 13:05:49 [init.py:239] Automatically detected platform cuda.
INFO 06-04 13:05:52 [cuda.py:188] Using Triton MLA backend.
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:52 [multiproc_worker_utils.py:225] Worker ready; awaiting tasks
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:52 [cuda.py:188] Using Triton MLA backend.
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:53 [utils.py:1055] Found nccl from library libnccl.so.2
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:53 [pynccl.py:69] vLLM is using nccl==2.21.5
INFO 06-04 13:05:53 [utils.py:1055] Found nccl from library libnccl.so.2
INFO 06-04 13:05:53 [pynccl.py:69] vLLM is using nccl==2.21.5
INFO 06-04 13:05:53 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /home/haolu/.cache/vllm/gpu_p2p_access_cache_for_4,5.json
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:53 [custom_all_reduce_utils.py:244] reading GPU P2P access cache from /home/haolu/.cache/vllm/gpu_p2p_access_cache_for_4,5.json
INFO 06-04 13:05:53 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_cde1f207'), local_subscribe_addr='ipc:///tmp/c6bc99ce-58bd-46f5-bda2-a955c43f1ecd', remote_subscribe_addr=None, remote_addr_ipv6=False)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:53 [parallel_state.py:1004] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1
INFO 06-04 13:05:53 [parallel_state.py:1004] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 06-04 13:05:53 [model_runner.py:1108] Starting to load model /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf...
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m INFO 06-04 13:05:53 [model_runner.py:1108] Starting to load model /data2/jcxy/llm_model/DeepSeek-R1-0528-GGUF-UD-IQ2_XXS/DeepSeek-R1-0528-UD-IQ2_XXS-00001-of-00005.gguf...
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m [2025-06-04 13:05:54,127] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-06-04 13:05:54,205] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m WARNING 06-04 13:06:21 [utils.py:168] The model class DeepseekV3ForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
WARNING 06-04 13:06:21 [utils.py:168] The model class DeepseekV3ForCausalLM has not defined packed_modules_mapping, this may lead to incorrect mapping of quantized or ignored modules
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m /data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/torch/nested/init.py:228: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. We recommend specifying layout=torch.jagged when constructing a nested tensor, as this layout receives active development, has better operator coverage, and works with torch.compile. (Triggered internally at /pytorch/aten/src/ATen/NestedTensorImpl.cpp:178.)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m return _nested.nested_tensor(
/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/torch/nested/init.py:228: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. We recommend specifying layout=torch.jagged when constructing a nested tensor, as this layout receives active development, has better operator coverage, and works with torch.compile. (Triggered internally at /pytorch/aten/src/ATen/NestedTensorImpl.cpp:178.)
return _nested.nested_tensor(
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m /data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/torch/nn/parameter.py:167: UserWarning: The use of x.T on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider x.mT to transpose batches of matrices or x.permute(*torch.arange(x.ndim - 1, -1, -1)) to reverse the dimensions of a tensor. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3725.)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m return super().torch_function(func, types, args, kwargs)
/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/torch/nn/parameter.py:167: UserWarning: The use of x.T on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider x.mT to transpose batches of matrices or x.permute(*torch.arange(x.ndim - 1, -1, -1)) to reverse the dimensions of a tensor. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3725.)
return super().torch_function(func, types, args, kwargs)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method load_model.
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] Traceback (most recent call last):
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/worker/worker.py", line 203, in load_model
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] self.model_runner.load_model()
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1111, in load_model
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] self.model = get_model(vllm_config=self.vllm_config)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] return loader.load_model(vllm_config=vllm_config)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1411, in load_model
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] _process_weights_after_loading(model, model_config, target_device)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 190, in _process_weights_after_loading
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] module.process_weights_after_loading(model_config.dtype)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/layer.py", line 255, in process_weights_after_loading
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] self.impl.process_weights_after_loading(act_dtype)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1194, in process_weights_after_loading
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1183, in get_and_maybe_dequant_weights
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] dequant_weights = layer.quant_method.apply(layer,
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 254, in apply
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] out = _fuse_mul_mat(x, qweight, qweight_type)
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 109, in _fuse_mul_mat
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] return x @ qweight.T
�[1;36m(VllmWorkerProcess pid=416050)�[0;0m ERROR 06-04 13:07:06 [multiproc_worker_utils.py:238] RuntimeError: size mismatch, got input (512), mat (512x512), vec (0)
ERROR 06-04 13:07:06 [engine.py:448] size mismatch, got input (512), mat (512x512), vec (0)
ERROR 06-04 13:07:06 [engine.py:448] Traceback (most recent call last):
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
ERROR 06-04 13:07:06 [engine.py:448] engine = MQLLMEngine.from_vllm_config(
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
ERROR 06-04 13:07:06 [engine.py:448] return cls(
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
ERROR 06-04 13:07:06 [engine.py:448] self.engine = LLMEngine(*args, **kwargs)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 275, in init
ERROR 06-04 13:07:06 [engine.py:448] self.model_executor = executor_class(vllm_config=vllm_config)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 286, in init
ERROR 06-04 13:07:06 [engine.py:448] super().init(*args, **kwargs)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 52, in init
ERROR 06-04 13:07:06 [engine.py:448] self._init_executor()
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 125, in _init_executor
ERROR 06-04 13:07:06 [engine.py:448] self._run_workers("load_model",
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
ERROR 06-04 13:07:06 [engine.py:448] driver_worker_output = run_method(self.driver_worker, sent_method,
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method
ERROR 06-04 13:07:06 [engine.py:448] return func(*args, **kwargs)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/worker/worker.py", line 203, in load_model
ERROR 06-04 13:07:06 [engine.py:448] self.model_runner.load_model()
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1111, in load_model
ERROR 06-04 13:07:06 [engine.py:448] self.model = get_model(vllm_config=self.vllm_config)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
ERROR 06-04 13:07:06 [engine.py:448] return loader.load_model(vllm_config=vllm_config)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1411, in load_model
ERROR 06-04 13:07:06 [engine.py:448] _process_weights_after_loading(model, model_config, target_device)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 190, in _process_weights_after_loading
ERROR 06-04 13:07:06 [engine.py:448] module.process_weights_after_loading(model_config.dtype)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/layer.py", line 255, in process_weights_after_loading
ERROR 06-04 13:07:06 [engine.py:448] self.impl.process_weights_after_loading(act_dtype)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1194, in process_weights_after_loading
ERROR 06-04 13:07:06 [engine.py:448] kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1183, in get_and_maybe_dequant_weights
ERROR 06-04 13:07:06 [engine.py:448] dequant_weights = layer.quant_method.apply(layer,
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 254, in apply
ERROR 06-04 13:07:06 [engine.py:448] out = _fuse_mul_mat(x, qweight, qweight_type)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 109, in _fuse_mul_mat
ERROR 06-04 13:07:06 [engine.py:448] return x @ qweight.T
ERROR 06-04 13:07:06 [engine.py:448] RuntimeError: size mismatch, got input (512), mat (512x512), vec (0)
Process SpawnProcess-1:
ERROR 06-04 13:07:06 [multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 416050 died, exit code: -15
INFO 06-04 13:07:06 [multiproc_worker_utils.py:124] Killing local vLLM worker processes
Traceback (most recent call last):
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 450, in run_mp_engine
raise e
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
engine = MQLLMEngine.from_vllm_config(
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
return cls(
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
self.engine = LLMEngine(*args, **kwargs)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 275, in init
self.model_executor = executor_class(vllm_config=vllm_config)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 286, in init
super().init(*args, **kwargs)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 125, in _init_executor
self._run_workers("load_model",
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
driver_worker_output = run_method(self.driver_worker, sent_method,
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method
return func(*args, **kwargs)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/worker/worker.py", line 203, in load_model
self.model_runner.load_model()
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1111, in load_model
self.model = get_model(vllm_config=self.vllm_config)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
return loader.load_model(vllm_config=vllm_config)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1411, in load_model
_process_weights_after_loading(model, model_config, target_device)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 190, in _process_weights_after_loading
module.process_weights_after_loading(model_config.dtype)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/layer.py", line 255, in process_weights_after_loading
self.impl.process_weights_after_loading(act_dtype)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1194, in process_weights_after_loading
kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1183, in get_and_maybe_dequant_weights
dequant_weights = layer.quant_method.apply(layer,
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 254, in apply
out = _fuse_mul_mat(x, qweight, qweight_type)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 109, in _fuse_mul_mat
return x @ qweight.T
RuntimeError: size mismatch, got input (512), mat (512x512), vec (0)
Traceback (most recent call last):
File "/data/jcxy/haolu/anaconda3/envs/haolu/bin/vllm", line 8, in
sys.exit(main())
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

[Isotr0py]

packages/vllm/attention/backends/mla/common.py", line 1194, in process_weights_after_loading
ERROR 06-04 13:07:06 [engine.py:448] kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1183, in get_and_maybe_dequant_weights
ERROR 06-04 13:07:06 [engine.py:448] dequant_weights = layer.quant_method.apply(layer,
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 254, in apply
ERROR 06-04 13:07:06 [engine.py:448] out = _fuse_mul_mat(x, qweight, qweight_type)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 109, in _fuse_mul_mat
ERROR 06-04 13:07:06 [engine.py:448] return x @ qweight.T
ERROR 06-04 13:07:06 [engine.py:448] RuntimeError: size mismatch, got input (512), mat (512x512), vec (0)

Hmmm, seems it's because the kv_b_proj_weight from new version DeepSeek-R1 has different names. (#13167 (comment))

[ChuanhongLi]

packages/vllm/attention/backends/mla/common.py", line 1194, in process_weights_after_loading
ERROR 06-04 13:07:06 [engine.py:448] kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/attention/backends/mla/common.py", line 1183, in get_and_maybe_dequant_weights
ERROR 06-04 13:07:06 [engine.py:448] dequant_weights = layer.quant_method.apply(layer,
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 254, in apply
ERROR 06-04 13:07:06 [engine.py:448] out = _fuse_mul_mat(x, qweight, qweight_type)
ERROR 06-04 13:07:06 [engine.py:448] File "/data/jcxy/haolu/anaconda3/envs/haolu/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/gguf.py", line 109, in _fuse_mul_mat
ERROR 06-04 13:07:06 [engine.py:448] return x @ qweight.T
ERROR 06-04 13:07:06 [engine.py:448] RuntimeError: size mismatch, got input (512), mat (512x512), vec (0)

Hmmm, seems it's because the kv_b_proj_weight from new version DeepSeek-R1 has different names. (#13167 (comment))

Hi, how can this problem be solved?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!