[Models] Replace all `nn.Conv2d` with vLLM's Conv2dLayer #28842

Isotr0py · 2025-11-17T07:55:51Z

Purpose

Following PR for [Model][MM] Extract conv layer as CustomOp #28455
Update all nn.Conv2d usages with Conv2dLayer

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Isotr0py <[email protected]>

gemini-code-assist

Code Review

This pull request replaces nn.Conv2d with vLLM's Conv2dLayer across multiple model files. The refactoring is mostly correct, but there are several critical errors in vllm/model_executor/models/chameleon.py. In this file, torch.Conv2dLayer is used instead of Conv2dLayer, which will lead to an AttributeError as Conv2dLayer is not a member of the torch module. I've provided suggestions to fix these issues. Please apply the corrections to ensure the model works as expected.

vllm/model_executor/models/chameleon.py

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

vllm/model_executor/models/chameleon.py

chatgpt-codex-connector · 2025-11-17T07:59:16Z

vllm/model_executor/models/siglip.py

+        self.patch_embedding = Conv2dLayer(
            in_channels=config.num_channels,
            out_channels=self.embed_dim,
            kernel_size=self.patch_size,


Pass unsupported padding="valid" to Conv2dLayer in Siglip embeddings

Conv2dLayer stores the passed padding as-is and forwards it to torch.nn.functional.conv2d, which only accepts integers or tuples. The new code now hands it the string "valid", causing a runtime TypeError when embedding images. Compute the numeric padding or drop the argument to preserve the previous behaviour.

Useful? React with 👍 / 👎.

Good bot, thanks! :)

chatgpt-codex-connector · 2025-11-17T07:59:16Z

vllm/model_executor/models/siglip2navit.py

+            self.patch_embedding = Conv2dLayer(
                in_channels=config.num_channels,
                out_channels=self.embed_dim,
                kernel_size=self.patch_size,


Siglip2 NaViT embedding uses Conv2dLayer with string padding

Similar to the Siglip change, Conv2dLayer does not translate string paddings. Passing padding="valid" will be forwarded directly to F.conv2d, triggering a TypeError when the model processes images. Replace the string with an explicit numeric padding (likely 0) before constructing the layer.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2025-11-17T07:59:16Z

vllm/model_executor/models/idefics2_vision_model.py

+        self.patch_embedding = Conv2dLayer(
            in_channels=config.num_channels,
            out_channels=self.embed_dim,
            kernel_size=self.patch_size,


Idefics2 vision embeddings now call Conv2dLayer with padding string

The new Conv2dLayer wrapper forwards padding directly to F.conv2d and does not implement the "valid" shortcut that nn.Conv2d provided. Using the string here will cause a runtime failure when forward runs. Replace with the correct numeric padding (0) or add conversion logic.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2025-11-17T07:59:16Z

vllm/model_executor/models/keye.py

+        self.patch_embedding = Conv2dLayer(
            in_channels=config.num_channels,
            out_channels=self.embed_dim,
            kernel_size=self.patch_size,


Keye vision embeddings pass unsupported padding="valid"

Conv2dLayer’s constructor only handles integer/tuple padding values. Passing the string "valid" like nn.Conv2d previously allowed will lead to an exception in forward when the convolution executes. Replace the string with the equivalent numeric padding.

Useful? React with 👍 / 👎.

Can you update the type annotation to account for this?

chatgpt-codex-connector · 2025-11-17T07:59:16Z

vllm/model_executor/models/paddleocr_vl.py

+        self.patch_embedding = Conv2dLayer(
            in_channels=config.num_channels,
            out_channels=self.embed_dim,
            kernel_size=self.patch_size,


PaddleOCR vision embeddings forward padding string to Conv2dLayer

The padding argument is now the literal string "valid", but Conv2dLayer passes self.padding straight to F.conv2d, which expects integers or tuples. This will raise a TypeError when embeddings are computed. Compute the numeric padding instead of relying on the string shortcut.

Useful? React with 👍 / 👎.

Signed-off-by: Isotr0py <[email protected]>

DarkLight1337 · 2025-11-17T13:27:53Z

/gemini review

gemini-code-assist

Code Review

This pull request systematically replaces all instances of torch.nn.Conv2d with vllm.model_executor.layers.conv.Conv2dLayer across various model implementations. This is a consistent and necessary refactoring to leverage vLLM's custom convolution layer, which likely offers optimizations or specific functionalities tailored for the framework. The changes are applied uniformly, including updating import statements and isinstance checks where appropriate. No functional issues or bugs were identified in this mechanical replacement.

Signed-off-by: Isotr0py <[email protected]>

Isotr0py · 2025-11-17T15:09:23Z

/gemini review

gemini-code-assist

Code Review

This pull request systematically replaces all instances of torch.nn.Conv2d with vLLM's custom Conv2dLayer across various model implementations. This is a crucial step towards unifying convolution operations within the vLLM framework, likely enabling custom optimizations or distributed computing features. The changes in vllm/model_executor/layers/conv.py correctly extend the ConvLayerBase to handle string-based padding ("same", "valid") and include a necessary validation check for strided convolutions with "same" padding. This ensures correctness and prevents undefined behavior. The widespread adoption of Conv2dLayer across numerous models demonstrates a consistent application of this architectural change.

gemini-code-assist · 2025-11-17T15:19:04Z

vllm/model_executor/layers/conv.py

+        if padding == "same" and any(s != 1 for s in stride):
+            raise ValueError("padding='same' is not supported for strided convolutions")


The addition of this validation check is crucial for correctness. padding='same' behavior is not well-defined for strided convolutions in all frameworks, and explicitly disallowing it prevents potential silent miscalculations or unexpected output dimensions. This improves the robustness of the Conv2dLayer.

…t#28842) Signed-off-by: Isotr0py <[email protected]>

…t#28842) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Bhagyashri <[email protected]>

…t#28842) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Amir Samani <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: jiang1.li <[email protected]>

…t#28842) Signed-off-by: Isotr0py <[email protected]>

Due to the latest changes from upstream, gemma3 is failing to compile on HPU vllm-project/vllm#27772 vllm-project/vllm#28842 -replace unfold to view/reshape -replace text embedding to avoid dynamic shape -remove merge_multimodal replacement since masked_scatter issue is fixed -enable back gemma3 model test --------- Signed-off-by: Jimin Ha <[email protected]>

replace remain conv2d layers

3bd3e82

Signed-off-by: Isotr0py <[email protected]>

Isotr0py requested review from patrickvonplaten and sighingnow as code owners November 17, 2025 07:55

Isotr0py requested a review from DarkLight1337 November 17, 2025 07:56

mergify bot added the qwen Related to Qwen models label Nov 17, 2025

gemini-code-assist bot reviewed Nov 17, 2025

View reviewed changes

chatgpt-codex-connector bot reviewed Nov 17, 2025

View reviewed changes

oops

613c480

Signed-off-by: Isotr0py <[email protected]>

gemini-code-assist bot reviewed Nov 17, 2025

View reviewed changes

Isotr0py added 2 commits November 17, 2025 23:04

update padding

4b6bfda

Signed-off-by: Isotr0py <[email protected]>

update typing

2346a40

Signed-off-by: Isotr0py <[email protected]>

DarkLight1337 approved these changes Nov 17, 2025

View reviewed changes

gemini-code-assist bot reviewed Nov 17, 2025

View reviewed changes

Isotr0py enabled auto-merge (squash) November 17, 2025 15:22

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 17, 2025

Merge branch 'main' into replace-conv

257feaa

Isotr0py merged commit e4bb268 into vllm-project:main Nov 18, 2025
51 checks passed

Victor49152 pushed a commit to Victor49152/vllm that referenced this pull request Nov 20, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (vllm-projec…

9f05609

…t#28842) Signed-off-by: Isotr0py <[email protected]>

Isotr0py deleted the replace-conv branch November 20, 2025 07:10

bhagyashrigai pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Nov 20, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (vllm-projec…

bbbd331

…t#28842) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Bhagyashri <[email protected]>

Amir-19 pushed a commit to Amir-19/vllm that referenced this pull request Nov 21, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (vllm-projec…

6088945

…t#28842) Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Amir Samani <[email protected]>

bigPYJ1151 pushed a commit that referenced this pull request Nov 25, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (#28842)

7bc8e45

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: jiang1.li <[email protected]>

bringlein pushed a commit to bringlein/vllm that referenced this pull request Nov 26, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (vllm-projec…

0aaeab7

…t#28842) Signed-off-by: Isotr0py <[email protected]>

devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (vllm-projec…

b611a57

…t#28842) Signed-off-by: Isotr0py <[email protected]>

kitaekatt pushed a commit to kitaekatt/vllm that referenced this pull request Dec 1, 2025

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer (vllm-projec…

e804f3a

…t#28842) Signed-off-by: Isotr0py <[email protected]>

jiminha mentioned this pull request Dec 2, 2025

Fix compile error for Gemma3 multimodal inputs vllm-project/vllm-gaudi#671

Merged

		if padding == "same" and any(s != 1 for s in stride):
		raise ValueError("padding='same' is not supported for strided convolutions")

Uh oh!

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer #28842

[Models] Replace all nn.Conv2d with vLLM's Conv2dLayer #28842

Uh oh!

Conversation

Isotr0py commented Nov 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Isotr0py Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented Nov 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Isotr0py commented Nov 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[Models] Replace all `nn.Conv2d` with vLLM's Conv2dLayer #28842

[Models] Replace all `nn.Conv2d` with vLLM's Conv2dLayer #28842

Isotr0py commented Nov 17, 2025 •

edited by github-actions bot

Loading