The local vllm cannot start paddleocr-vl.

[dgzxx-2000]

🔎 Search before asking

• I have searched the PaddleOCR Docs and found no similar bug report.
• I have searched the PaddleOCR Issues and found no similar bug report.
• I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

(vllm) zengxiangxi@w550-pc:~$ python -m vllm.entrypoints.openai.api_server --model /home/zengxiangxi/.paddlex/official_models/PaddleOCR-VL --tensor-parallel-size 1 --gpu-memory-utilization 0.25 --max-model-len 4096 --trust-remote-code --host 0.0.0.0 --port 18006
INFO 10-20 11:18:00 [init.py:216] Automatically detected platform cuda.
ERROR 10-20 11:18:05 [init.py:54] Failed to load plugin register_paddlex_genai_models
ERROR 10-20 11:18:05 [init.py:54] Traceback (most recent call last):
ERROR 10-20 11:18:05 [init.py:54] File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/plugins/init.py", line 51, in load_plugins_by_group
ERROR 10-20 11:18:05 [init.py:54] func = plugin.load()
ERROR 10-20 11:18:05 [init.py:54] ^^^^^^^^^^^^^
ERROR 10-20 11:18:05 [init.py:54] File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/importlib/metadata/init.py", line 202, in load
ERROR 10-20 11:18:05 [init.py:54] module = import_module(match.group('module'))
ERROR 10-20 11:18:05 [init.py:54] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-20 11:18:05 [init.py:54] File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/importlib/init.py", line 126, in import_module
ERROR 10-20 11:18:05 [init.py:54] return _bootstrap._gcd_import(name[level:], package, level)
ERROR 10-20 11:18:05 [init.py:54] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 10-20 11:18:05 [init.py:54] File "", line 1204, in _gcd_import
ERROR 10-20 11:18:05 [init.py:54] File "", line 1176, in _find_and_load
ERROR 10-20 11:18:05 [init.py:54] File "", line 1126, in _find_and_load_unlocked
ERROR 10-20 11:18:05 [init.py:54] File "", line 241, in _call_with_frames_removed
ERROR 10-20 11:18:05 [init.py:54] File "", line 1204, in _gcd_import
ERROR 10-20 11:18:05 [init.py:54] File "", line 1176, in _find_and_load
ERROR 10-20 11:18:05 [init.py:54] File "", line 1126, in _find_and_load_unlocked
ERROR 10-20 11:18:05 [init.py:54] File "", line 241, in _call_with_frames_removed
ERROR 10-20 11:18:05 [init.py:54] File "", line 1204, in _gcd_import
ERROR 10-20 11:18:05 [init.py:54] File "", line 1176, in _find_and_load
ERROR 10-20 11:18:05 [init.py:54] File "", line 1147, in _find_and_load_unlocked
ERROR 10-20 11:18:05 [init.py:54] File "", line 690, in _load_unlocked
ERROR 10-20 11:18:05 [init.py:54] File "", line 940, in exec_module
ERROR 10-20 11:18:05 [init.py:54] File "", line 241, in _call_with_frames_removed
ERROR 10-20 11:18:05 [init.py:54] File "/home/zengxiangxi/.local/lib/python3.11/site-packages/paddlex/inference/genai/init.py", line 17, in
ERROR 10-20 11:18:05 [init.py:54] require_genai_engine_plugin()
ERROR 10-20 11:18:05 [init.py:54] File "/home/zengxiangxi/.local/lib/python3.11/site-packages/paddlex/utils/deps.py", line 290, in require_genai_engine_plugin
ERROR 10-20 11:18:05 [init.py:54] raise RuntimeError(f"{prefix} not available. Please install it properly.")
ERROR 10-20 11:18:05 [init.py:54] RuntimeError: The generative AI engine plugins are not available. Please install it properly.
(APIServer pid=64471) INFO 10-20 11:18:06 [api_server.py:1839] vLLM API server version 0.11.0
(APIServer pid=64471) INFO 10-20 11:18:06 [utils.py:233] non-default args: {'host': '0.0.0.0', 'port': 18006, 'model': '/home/zengxiangxi/.paddlex/official_models/PaddleOCR-VL', 'trust_remote_code': True, 'max_model_len': 4096, 'gpu_memory_utilization': 0.25}
(APIServer pid=64471) The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=64471) Traceback (most recent call last):
(APIServer pid=64471) File "", line 198, in _run_module_as_main
(APIServer pid=64471) File "", line 88, in _run_code
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1953, in
(APIServer pid=64471) uvloop.run(run_server(args))
(APIServer pid=64471) File "/home/zengxiangxi/.local/lib/python3.11/site-packages/uvloop/init.py", line 105, in run
(APIServer pid=64471) return runner.run(wrapper())
(APIServer pid=64471) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=64471) return self._loop.run_until_complete(task)
(APIServer pid=64471) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=64471) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=64471) File "/home/zengxiangxi/.local/lib/python3.11/site-packages/uvloop/init.py", line 61, in wrapper
(APIServer pid=64471) return await main
(APIServer pid=64471) ^^^^^^^^^^
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1884, in run_server
(APIServer pid=64471) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1902, in run_server_worker
(APIServer pid=64471) async with build_async_engine_client(
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/contextlib.py", line 210, in aenter
(APIServer pid=64471) return await anext(self.gen)
(APIServer pid=64471) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client
(APIServer pid=64471) async with build_async_engine_client_from_engine_args(
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/contextlib.py", line 210, in aenter
(APIServer pid=64471) return await anext(self.gen)
(APIServer pid=64471) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 206, in build_async_engine_client_from_engine_args
(APIServer pid=64471) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=64471) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 1142, in create_engine_config
(APIServer pid=64471) model_config = self.create_model_config()
(APIServer pid=64471) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=64471) File "/data/home/zengxiangxi/app/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm/engine/arg_utils.py", line 994, in create_model_config
(APIServer pid=64471) return ModelConfig(
(APIServer pid=64471) ^^^^^^^^^^^^
(APIServer pid=64471) File "/home/zengxiangxi/.local/lib/python3.11/site-packages/pydantic/_internal/_dataclasses.py", line 123, in init
(APIServer pid=64471) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=64471) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=64471) Value error, Invalid repository ID or local directory specified: '/home/zengxiangxi/.paddlex/official_models/PaddleOCR-VL'.
(APIServer pid=64471) Please verify the following requirements:
(APIServer pid=64471) 1. Provide a valid Hugging Face repository ID.
(APIServer pid=64471) 2. Specify a local directory that contains a recognized configuration file.
(APIServer pid=64471) - For Hugging Face models: ensure the presence of a 'config.json'.
(APIServer pid=64471) - For Mistral models: ensure the presence of a 'params.json'.
(APIServer pid=64471) 3. For GGUF: pass the local path of the GGUF checkpoint.
(APIServer pid=64471) Loading GGUF from a remote repo directly is not yet supported.
(APIServer pid=64471) [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]
(APIServer pid=64471) For further information visit https://errors.pydantic.dev/2.11/v/value_error

🏃‍♂️ Environment (运行环境)

python3.11 liunx

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

(vllm) zengxiangxi@w550-pc:~$ python -m vllm.entrypoints.openai.api_server --model /home/zengxiangxi/.paddlex/official_models/PaddleOCR-VL --tensor-parallel-size 1 --gpu-memory-utilization 0.25 --max-model-len 4096 --trust-remote-code --host 0.0.0.0 --port 18006

[Bobholamovic]

Since PaddleOCR-VL has not been integrated into vllm, currently the standard way is to start the vLLM server via PaddleOCR CLI. The started server is identical to the one you expected to start with vllm.entrypoints.openai.api_server, and can be requested through requests, openai, etc.

[dgzxx-2000]

由于 PaddleOCR-VL 尚未集成到 vllm，目前标准方式是通过 PaddleOCR CLI 启动 vLLM 服务器。启动的服务器与您预期的相同vllm.entrypoints.openai.api_server，可以通过 、 等方式进行requests请求openai。

paddlex_genai_server --model_name PaddleOCR-VL-0.9B --backend vllm --port 8118 ?

[Bobholamovic]

由于 PaddleOCR-VL 尚未集成到 vllm，目前标准方式是通过 PaddleOCR CLI 启动 vLLM 服务器。启动的服务器与您预期的相同vllm.entrypoints.openai.api_server，可以通过 、 等方式进行requests请求openai。

paddlex_genai_server --model_name PaddleOCR-VL-0.9B --backend vllm --port 8118 ?

是的，这是标准的启动方式。

[RanHuang]

vllm-project/vllm#27758