[Bug]: AttributeError: 'EmbeddingChatRequest' object has no attribute 'mm_processor_kwargs'

[hxyghostor]

Your current environment

when I deploy dse-qwen2-2b-mrl-v1 embedding model, assert [Bug]: AttributeError: 'EmbeddingChatRequest' object has no attribute 'mm_processor_kwargs'
File "/home/tiger/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/applications.py", line 112, in call await self.middleware_stack(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in call raise exc File "/home/tiger/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in call await self.app(scope, receive, _send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call await self.app(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/tiger/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/routing.py", line 714, in call await self.middleware_stack(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc
File "/home/tiger/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender) File "/home/tiger/.local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/vllm/entrypoints/utils.py", line 56, in wrapper
return handler_task.result()
^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 475, in create_embedding
generator = await handler.create_embedding(request, raw_request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/vllm/entrypoints/openai/serving_embedding.py", line 118, in create_embedding
) = await self._preprocess_chat(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/vllm/entrypoints/openai/serving_engine.py", line 454, in _preprocess_chat
if request.mm_processor_kwargs is not None:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.local/lib/python3.11/site-packages/pydantic/main.py", line 891, in getattr
raise AttributeError(f'{type(self).name!r} object has no attribute {item!r}')
AttributeError: 'EmbeddingChatRequest' object has no attribute 'mm_processor_kwargs'

🐛 Describe the bug

server code
vllm serve /dse-qwen2-2b-mrl-v1 --port $PORT0 --host 0.0.0.0 --max-model-len 4096 --task embed --chat-template /server_vllm/template_dse_qwen2_vl.jinja

client code
api_url = f"http://localhost:{vllm_port}/v1/embeddings"

    # image_path = "/server_vllm/image_a7ec7349586043edb90e8601daa39286.png"
    # with open(image_path, "rb") as f:
    #     encoded_image = base64.b64encode(f.read())
    # encoded_image_text = encoded_image.decode("utf-8")
    
    # base64_qwen = f"data:image;base64,{image}"

data = {
"model" : "/dse-qwen2-2b-mrl-v1",
"messages":[
{"role": "system", "content": "You are a helpful assistant."},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": base64_qwen,
"min_pixels": 42828,
"max_pixels": 2562828,
},
},
{"type": "text",
"text": "..."},
],
},
],
"encoding_format": "float",
}

    response = requests.post(api_url, json=data)
    response.raise_for_status()
    chat_response = response.json()
    print("Chat response:", chat_response)

I follow https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_embedding_client_for_multimodal.py and https://docs.vllm.ai/en/stable/serving/multimodal_inputs.html#multimodal-inputs
I have no idea what happened

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Can you show the full logs when you start up the server?

[DarkLight1337]

Please note, you should set --task embed when using embedding models.

[hxyghostor]

I set vllm serve /dse-qwen2-2b-mrl-v1 --port $PORT0 --host 0.0.0.0 --max-model-len 4096 --task embed --chat-template /server_vllm/template_dse_qwen2_vl.jinja
you can see my server code

[hxyghostor]

estion_classification/server_vllm/template_dse_qwen2_vl.jinja
INFO 03-19 00:09:16 init.py:207] Automatically detected platform cuda.
INFO 03-19 00:09:16 api_server.py:912] vLLM API server version 0.7.3
INFO 03-19 00:09:16 api_server.py:913] args: Namespace(subparser='serve', model_tag='/dse-qwen2-2b-mrl-v1', config='', host='0.0.0.0', port=10386, uvicorn_log_level='info', allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='/server_vllm/template_dse_qwen2_vl.jinja', chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/dse-qwen2-2b-mrl-v1', task='embed', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=4096, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7f5d2dda2d40>)
INFO 03-19 00:09:16 api_server.py:209] Started engine process with PID 83924
INFO 03-19 00:09:21 init.py:207] Automatically detected platform cuda.
INFO 03-19 00:09:28 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.3) with config: model='/dse-qwen2-2b-mrl-v1', speculative_config=None, tokenizer='/dse-qwen2-2b-mrl-v1', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/dse-qwen2-2b-mrl-v1, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=PoolerConfig(pooling_type=None, normalize=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
INFO 03-19 00:09:29 cuda.py:229] Using Flash Attention backend.
INFO 03-19 00:09:31 model_runner.py:1110] Starting to load model /dse-qwen2-2b-mrl-v1...
INFO 03-19 00:09:32 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
Loading pt checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:02<00:00, 2.81s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:02<00:00, 2.81s/it]

WARNING 03-19 00:09:35 utils.py:175] Unable to collect loaded parameters for module Qwen2ForEmbedding(
WARNING 03-19 00:09:35 utils.py:175] (model): Qwen2Model(
WARNING 03-19 00:09:35 utils.py:175] (embed_tokens): VocabParallelEmbedding(num_embeddings=151936, embedding_dim=1536, org_vocab_size=151936, num_embeddings_padded=151936, tp_size=1)
WARNING 03-19 00:09:35 utils.py:175] (layers): ModuleList(
WARNING 03-19 00:09:35 utils.py:175] (0-27): 28 x Qwen2DecoderLayer(
WARNING 03-19 00:09:35 utils.py:175] (self_attn): Qwen2Attention(
WARNING 03-19 00:09:35 utils.py:175] (qkv_proj): QKVParallelLinear(in_features=1536, output_features=2048, bias=True, tp_size=1, gather_output=False)
WARNING 03-19 00:09:35 utils.py:175] (o_proj): RowParallelLinear(input_features=1536, output_features=1536, bias=False, tp_size=1, reduce_results=True)
WARNING 03-19 00:09:35 utils.py:175] (rotary_emb): MRotaryEmbedding(head_size=128, rotary_dim=128, max_position_embeddings=131072, base=1000000.0, is_neox_style=True)
WARNING 03-19 00:09:35 utils.py:175] (attn): Attention(head_size=128, num_heads=12, num_kv_heads=2, scale=0.08838834764831845, backend=FlashAttentionImpl)
WARNING 03-19 00:09:35 utils.py:175] )
WARNING 03-19 00:09:35 utils.py:175] (mlp): Qwen2MLP(
WARNING 03-19 00:09:35 utils.py:175] (gate_up_proj): MergedColumnParallelLinear(in_features=1536, output_features=17920, bias=False, tp_size=1, gather_output=False)
WARNING 03-19 00:09:35 utils.py:175] (down_proj): RowParallelLinear(input_features=8960, output_features=1536, bias=False, tp_size=1, reduce_results=True)
WARNING 03-19 00:09:35 utils.py:175] (act_fn): SiluAndMul()
WARNING 03-19 00:09:35 utils.py:175] )
WARNING 03-19 00:09:35 utils.py:175] (input_layernorm): RMSNorm(hidden_size=1536, eps=1e-06)
WARNING 03-19 00:09:35 utils.py:175] (post_attention_layernorm): RMSNorm(hidden_size=1536, eps=1e-06)
WARNING 03-19 00:09:35 utils.py:175] )
WARNING 03-19 00:09:35 utils.py:175] )
WARNING 03-19 00:09:35 utils.py:175] (norm): RMSNorm(hidden_size=1536, eps=1e-06)
WARNING 03-19 00:09:35 utils.py:175] )
WARNING 03-19 00:09:35 utils.py:175] (sampler): Sampler()
WARNING 03-19 00:09:35 utils.py:175] (_pooler): LastPool(
WARNING 03-19 00:09:35 utils.py:175] (head): PoolerHead()
WARNING 03-19 00:09:35 utils.py:175] )
WARNING 03-19 00:09:35 utils.py:175] )
INFO 03-19 00:09:35 model_runner.py:1115] Loading model weights took 4.1512 GB
INFO 03-19 00:09:35 api_server.py:820] Using supplied chat template:
INFO 03-19 00:09:35 api_server.py:820] {% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}{% raw %}<|im_start|>system
INFO 03-19 00:09:35 api_server.py:820] You are a helpful assistant.<|im_end|>
INFO 03-19 00:09:35 api_server.py:820] {% endraw %}{% endif %}<|im_start|>{{ message['role'] }}{% raw %}
INFO 03-19 00:09:35 api_server.py:820] {% endraw %}{% if message['content'] is string %}{{ message['content'] }}<|im_end|>{% raw %}
INFO 03-19 00:09:35 api_server.py:820] {% endraw %}{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>{% raw %}
INFO 03-19 00:09:35 api_server.py:820] {% endraw %}{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant{% raw %}
INFO 03-19 00:09:35 api_server.py:820] {% endraw %}{% endif %}<|endoftext|>
INFO 03-19 00:09:35 api_server.py:958] Starting vLLM API server on http://0.0.0.0:10386
INFO 03-19 00:09:35 launcher.py:23] Available routes are:
INFO 03-19 00:09:35 launcher.py:31] Route: /openapi.json, Methods: GET, HEAD
INFO 03-19 00:09:35 launcher.py:31] Route: /docs, Methods: GET, HEAD
INFO 03-19 00:09:35 launcher.py:31] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 03-19 00:09:35 launcher.py:31] Route: /redoc, Methods: GET, HEAD
INFO 03-19 00:09:35 launcher.py:31] Route: /health, Methods: GET
INFO 03-19 00:09:35 launcher.py:31] Route: /ping, Methods: POST, GET
INFO 03-19 00:09:35 launcher.py:31] Route: /tokenize, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /detokenize, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/models, Methods: GET
INFO 03-19 00:09:35 launcher.py:31] Route: /version, Methods: GET
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/chat/completions, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/completions, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/embeddings, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /pooling, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /score, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/score, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/audio/transcriptions, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /rerank, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v1/rerank, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /v2/rerank, Methods: POST
INFO 03-19 00:09:35 launcher.py:31] Route: /invocations, Methods: POST
INFO: Started server process [83887]
INFO: Waiting for application startup.
INFO: Application startup complete.

[DarkLight1337]

This should be fixed by #13644 already, can you use a newer version of vLLM?

[hxyghostor]

seems pip3 install -U vllm not work

[DarkLight1337]

Can you try again? v0.8.0 just released

[hxyghostor]

Thanks, it works
but is the performance normal? It seems only one instance has been launched

QPS 4.462
avg latency 889 ms

[DarkLight1337]

The support for embedding models in vLLM is primarily for convenience, so it might not be that optimized.

[hxyghostor]

OK, thanks for your reply