Skip to content

[Bugfix] remove post_layernorm in siglip #8106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Sep 4, 2024
Merged

[Bugfix] remove post_layernorm in siglip #8106

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
4a65e17
[Bugfix] remove post_layernorm in siglip
wnma3mz Sep 3, 2024
229bdee
add post_layernorm when num_hidden_layers_override is None for siglip
wnma3mz Sep 3, 2024
98b19bc
update num_hidden_layers_override equals config.num_hidden_layers
wnma3mz Sep 3, 2024
2ec4f3e
format code
wnma3mz Sep 3, 2024
b93318f
fix load_weights
wnma3mz Sep 3, 2024
908d316
Merge branch 'main' into main
wnma3mz Sep 3, 2024
459974b
format code
wnma3mz Sep 3, 2024
e06c4de
format code
wnma3mz Sep 3, 2024
4da0638
format code
wnma3mz Sep 3, 2024
544ab65
move need_post_layernorm to SiglipVisionTransformer
wnma3mz Sep 4, 2024
5ea1163
make need_post_layernorm as a property
wnma3mz Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions vllm/model_executor/models/siglip.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,13 +456,14 @@ def forward(

encoder_outputs = self.encoder(inputs_embeds=hidden_states)

last_hidden_state = self.post_layernorm(encoder_outputs)
# When it is used as a visual encoder, post_layernorm is not required
# last_hidden_state = self.post_layernorm(encoder_outputs)

# TODO: add this back when pooled_output is used in inference
# if self.use_head:
# pooled_output = self.head(last_hidden_state)

return last_hidden_state
return encoder_outputs


class SiglipVisionModel(nn.Module):
Expand Down