[New Model]: Google SigLip 2

[tranhoangnguyen03]

The model to consider.

Google SigLip 2

The closest model vllm already supports.

https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/siglip.py

"""Implementation of SiglipVisionModel intended to be only used
within a vision language model."""

However, SigLip 2 can be very useful for zero-shot image classification and image-text retrieval

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

The model has the same architecture as SigLIP v1. We just need to modify the code a bit so that it can be directly called in vLLM from the model worker (i.e. implementing the VllmModelForPooling and SupportsMultiModal interface).

[lengrongfu]

I can try to solve it, please assign it to me.

/assign

[lengrongfu]

@DarkLight1337 Which model should I use to test the support Google SigLip2 model function, can you help me? Give me a huggingface model address.

[DarkLight1337]

The first link in the OP should point you to the model collection.

[lengrongfu]

we exec this command, but get have error.
vllm serve google/siglip2-base-patch16-naflex --hf_overrides '{"architectures": ["LlavaForConditionalGeneration"]}'

INFO 02-25 00:03:42 parallel_state.py:948] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 02-25 00:03:42 model_runner.py:1110] Starting to load model google/siglip2-base-patch16-naflex...
ERROR 02-25 00:03:43 engine.py:400] 'Siglip2Config' object has no attribute 'projector_hidden_act'
ERROR 02-25 00:03:43 engine.py:400] Traceback (most recent call last):
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
ERROR 02-25 00:03:43 engine.py:400] engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
ERROR 02-25 00:03:43 engine.py:400] return cls(ipc_path=ipc_path,
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 76, in init
ERROR 02-25 00:03:43 engine.py:400] self.engine = LLMEngine(*args, **kwargs)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/engine/llm_engine.py", line 273, in init
ERROR 02-25 00:03:43 engine.py:400] self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/executor/executor_base.py", line 52, in init
ERROR 02-25 00:03:43 engine.py:400] self._init_executor()
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/executor/uniproc_executor.py", line 47, in _init_executor
ERROR 02-25 00:03:43 engine.py:400] self.collective_rpc("load_model")
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 02-25 00:03:43 engine.py:400] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/utils.py", line 2232, in run_method
ERROR 02-25 00:03:43 engine.py:400] return func(*args, **kwargs)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/worker/worker.py", line 183, in load_model
ERROR 02-25 00:03:43 engine.py:400] self.model_runner.load_model()
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/worker/model_runner.py", line 1113, in load_model
ERROR 02-25 00:03:43 engine.py:400] self.model = get_model(vllm_config=self.vllm_config)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/model_executor/model_loader/init.py", line 14, in get_model
ERROR 02-25 00:03:43 engine.py:400] return loader.load_model(vllm_config=vllm_config)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/model_executor/model_loader/loader.py", line 406, in load_model
ERROR 02-25 00:03:43 engine.py:400] model = _initialize_model(vllm_config=vllm_config)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/model_executor/model_loader/loader.py", line 125, in _initialize_model
ERROR 02-25 00:03:43 engine.py:400] return model_class(vllm_config=vllm_config, prefix=prefix)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/root/lengrongfu/vllm/vllm/model_executor/models/llava.py", line 518, in init
ERROR 02-25 00:03:43 engine.py:400] if (config.projector_hidden_act is None
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] File "/usr/local/anaconda3/envs/lrf-vllm-dev/lib/python3.12/site-packages/transformers/configuration_utils.py", line 214, in getattribute
ERROR 02-25 00:03:43 engine.py:400] return super().getattribute(key)
ERROR 02-25 00:03:43 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 00:03:43 engine.py:400] AttributeError: 'Siglip2Config' object has no attribute 'projector_hidden_act'
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/local/anaconda3/envs/lrf-vllm-dev/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/anaconda3/envs/lrf-vllm-dev/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 402, in run_mp_engine
raise e
File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
return cls(ipc_path=ipc_path,
^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/engine/multiprocessing/engine.py", line 76, in init
self.engine = LLMEngine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/engine/llm_engine.py", line 273, in init
self.model_executor = executor_class(vllm_config=vllm_config, )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/root/lengrongfu/vllm/vllm/executor/uniproc_executor.py", line 47, in _init_executor
self.collective_rpc("load_model")
File "/root/lengrongfu/vllm/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/utils.py", line 2232, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/worker/worker.py", line 183, in load_model
self.model_runner.load_model()
File "/root/lengrongfu/vllm/vllm/worker/model_runner.py", line 1113, in load_model
self.model = get_model(vllm_config=self.vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/model_executor/model_loader/init.py", line 14, in get_model
return loader.load_model(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/model_executor/model_loader/loader.py", line 406, in load_model
model = _initialize_model(vllm_config=vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/model_executor/model_loader/loader.py", line 125, in _initialize_model
return model_class(vllm_config=vllm_config, prefix=prefix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/lengrongfu/vllm/vllm/model_executor/models/llava.py", line 518, in init
if (config.projector_hidden_act is None
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/anaconda3/envs/lrf-vllm-dev/lib/python3.12/site-packages/transformers/configuration_utils.py", line 214, in getattribute
return super().getattribute(key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Siglip2Config' object has no attribute 'projector_hidden_act'

[DarkLight1337]

What this issue is asking for is to support SigLIP2 as a standalone image embedding model, rather than a generative model that includes SigLIP2 as a component.

[DarkLight1337]

From my understanding, you need transformers v4.49 to use SigLIP2.

[lengrongfu]

From my understanding, you need transformers v4.49 to use SigLIP2.

Yes, I use this version pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2.