Skip to content

[VLM] Post-layernorm override and quant config in vision encoder #9217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Oct 23, 2024
Merged

[VLM] Post-layernorm override and quant config in vision encoder #9217

merged 20 commits into from
Oct 23, 2024

Conversation

@DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Oct 10, 2024 •
edited
Loading

Also, QuantizationConfig and associated prefix argument is now passed to vision towers to maintain consistency. Nevertheless, since vision tower is not quantized by existing methods yet, we ignore it and add a code comment explaining this.

FIX #9186

@DarkLight1337 DarkLight1337 requested a review from ywang96 October 10, 2024 04:17
@github-actions
Copy link

github-actions bot commented Oct 10, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Oct 10, 2024

cc @litianjian

@litianjian
Copy link
Contributor

litianjian commented Oct 10, 2024

cc @litianjian

OK， I will update the llava models after this PR.

@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Oct 11, 2024

@ywang96 PTAL when you have time.

@DarkLight1337 DarkLight1337 changed the title [VLM] Enable overriding whether post layernorm is used in vision encoder [VLM] Enable overriding whether post layernorm is used in vision encoder + normalize quant args Oct 18, 2024
@DarkLight1337 DarkLight1337 changed the title [VLM] Enable overriding whether post layernorm is used in vision encoder + normalize quant args [VLM] Enable overriding whether post layernorm is used in vision encoder + fix quant args Oct 18, 2024
@litianjian
Copy link
Contributor

litianjian commented Oct 22, 2024

@DarkLight1337 Would you mind sharing the current progress？

@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Oct 22, 2024

It's ready for review, @ywang96 is away so I have asked @Isotr0py for a review.

@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Oct 22, 2024

Sorry for the delay, I have been focused on other stuff lately.

@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 23, 2024
Isotr0py
Isotr0py approved these changes Oct 23, 2024
View reviewed changes
Copy link
Member

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for enabling this!

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) October 23, 2024 11:02
@DarkLight1337 DarkLight1337 merged commit c18e1a3 into main Oct 23, 2024
63 checks passed
@DarkLight1337 DarkLight1337 deleted the require-post-norm branch October 23, 2024 11:28
@DarkLight1337 DarkLight1337 mentioned this pull request Oct 23, 2024
Merged
@mgoin mgoin mentioned this pull request Oct 23, 2024
Merged
@litianjian litianjian mentioned this pull request Oct 24, 2024
Merged
Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024
@DarkLight1337 @Isotr0py @Alvant
[VLM] Enable overriding whether post layernorm is used in vision enco…
45142e3 
…der + fix quant args (vllm-project#9217)

Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: Alvant <[email protected]>
MErkinSag pushed a commit to MErkinSag/vllm that referenced this pull request Oct 26, 2024
@DarkLight1337 @Isotr0py
[VLM] Enable overriding whether post layernorm is used in vision enco…
71d1761 
…der + fix quant args (vllm-project#9217)

Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: Erkin Sagiroglu <[email protected]>
gshtras
gshtras reviewed Oct 28, 2024
View reviewed changes
vllm/model_executor/models/mllama.py
MllamaVisionEncoderLayer(config, is_gated)
for _ in range(num_layers)
MllamaVisionEncoderLayer(config,
quant_config=quant_config,
Copy link
Collaborator

@gshtras gshtras Oct 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change breaks partially quantized models such as https://huggingface.co/amd/Llama-3.2-11B-Vision-Instruct-FP8-KV
Since the vision part is not quantized, now because of this config an on the fly quantization is being applied (https://github.com/ROCm/vllm/blob/main/vllm/model_executor/layers/quantization/utils/w8a8_utils.py#L126), which can't work with vision MLP due to it being 3-dimentional. So it eventually crashes at https://github.com/ROCm/vllm/blob/main/vllm/_custom_ops.py#L711

Copy link
Member Author

@DarkLight1337 DarkLight1337 Oct 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgoin can you help with this so we can handle both partially and fully quantized models?

Copy link
Collaborator

@gshtras gshtras Oct 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposed a fix in #9800

Copy link
Member

@mgoin mgoin Oct 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DarkLight1337 I am working on this for as many models as possible in #9772

FerdinandZhong pushed a commit to FerdinandZhong/vllm that referenced this pull request Oct 29, 2024
@DarkLight1337 @Isotr0py @FerdinandZhong
[VLM] Enable overriding whether post layernorm is used in vision enco…
b82e5c9 
…der + fix quant args (vllm-project#9217)

Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: qishuai <[email protected]>
@gshtras gshtras mentioned this pull request Oct 29, 2024
Closed
@DarkLight1337 DarkLight1337 mentioned this pull request Oct 29, 2024
Open
54 tasks
@DarkLight1337 DarkLight1337 changed the title [VLM] Enable overriding whether post layernorm is used in vision encoder + fix quant args [VLM] Enable overriding post layernorm usage + fix quant args Nov 1, 2024
@DarkLight1337 DarkLight1337 changed the title [VLM] Enable overriding post layernorm usage + fix quant args [VLM] Enable post layernorm override and quant config in vision encoder Nov 1, 2024
@DarkLight1337 DarkLight1337 changed the title [VLM] Enable post layernorm override and quant config in vision encoder [VLM] Post-layernorm override and quant config in vision encoder Nov 1, 2024
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
@DarkLight1337 @Isotr0py @sumitd2
[VLM] Enable overriding whether post layernorm is used in vision enco…
530b4ab 
…der + fix quant args (vllm-project#9217)

Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: Sumit Dubey <[email protected]>
LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025
@DarkLight1337 @Isotr0py @LeiWang1999
[VLM] Enable overriding whether post layernorm is used in vision enco…
cc73e6e 
…der + fix quant args (vllm-project#9217)

Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: LeiWang1999 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgoin mgoin mgoin left review comments

@gshtras gshtras gshtras left review comments

@Isotr0py Isotr0py Isotr0py approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Output state configuration of vision encoder In VLM

6 participants

@DarkLight1337 @litianjian @mgoin @Isotr0py @gshtras