Skip to content

Add Apertus #39381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 46 commits into from
Aug 28, 2025
Merged

Add Apertus #39381

merged 46 commits into from
Aug 28, 2025

Conversation

@EduardDurech
Copy link
Contributor

@EduardDurech EduardDurech commented Jul 12, 2025

Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

  • xIELU Activation
  • QK-norm

@ArthurZucker

@EduardDurech
Copy link
Contributor Author

EduardDurech commented Jul 12, 2025

@dhia680

@chiffa
Copy link

chiffa commented Jul 14, 2025

@Cyrilvallez - this is the part 1 of the PR from Swiss AI initiative

ArthurZucker
ArthurZucker reviewed Jul 15, 2025
View reviewed changes
Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice and very transformers like! Do you mind using modular to isolate the changes?

@EduardDurech
Copy link
Contributor Author

EduardDurech commented Jul 15, 2025

very nice and very transformers like! Do you mind using modular to isolate the changes?

Yep, I was planning to and Cyril suggested, I built from Alex's original implementation but I'll refactor

@ArthurZucker
Copy link
Collaborator

ArthurZucker commented Jul 15, 2025

It should not require too much changes don't worry, its already in an excellent state!

@AllenHaoHuang AllenHaoHuang mentioned this pull request Aug 13, 2025
Closed
dhia680 added 5 commits August 27, 2025 22:10
@dhia680
remove fx_compatible (deprecated)
e19d543
@dhia680
remove rotary_embedding_layer
69c46ed 
As the tests are written for a config without default scaling (which is not the case in Apertus) - besides, rope scaling is tested in other models so it's all safe.
@dhia680
fully removing Mask4DTestHard class
864c4dd 
Not needed (for now)
@dhia680
switch to dtype instead of torch_dtype
e7d03ad 
Following this:
huggingface#39782
@dhia680
remove unused imports
c394446
@github-actions
Copy link
Contributor

github-actions bot commented Aug 27, 2025

[For maintainers] Suggested jobs to run (before merge)

run-slow: apertus, auto

Cyrilvallez
Cyrilvallez approved these changes Aug 28, 2025
View reviewed changes
Copy link
Member

@Cyrilvallez Cyrilvallez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright! Very very nice, congrats! Super efficient work! Merging it now! 🤗

@Cyrilvallez Cyrilvallez merged commit d10603f into huggingface:main Aug 28, 2025
20 of 22 checks passed
@EduardDurech EduardDurech mentioned this pull request Aug 29, 2025
Merged
@malte-aws malte-aws mentioned this pull request Sep 1, 2025
Merged
@EduardDurech EduardDurech mentioned this pull request Sep 1, 2025
Merged
@theo77186 theo77186 mentioned this pull request Sep 2, 2025
Closed
4 tasks
@pwilkin pwilkin mentioned this pull request Sep 2, 2025
Closed
4 tasks
@abenmrad abenmrad mentioned this pull request Sep 2, 2025
Open
@ArthurZucker
Copy link
Collaborator

ArthurZucker commented Sep 12, 2025

Hey @EduardDurech do you think we can add integration tests now? 🤗

@EduardDurech
Copy link
Contributor Author

EduardDurech commented Sep 12, 2025

@ArthurZucker yea should be able, the models are hosted on HF now, @dhia680 would you be able to? I'm too busy with RL for a bit

@ArthurZucker
Copy link
Collaborator

ArthurZucker commented Sep 12, 2025

Nice! 🤗 we can also have a god if neither of you can!!

vermouth1992 pushed a commit to volcengine/verl that referenced this pull request Sep 13, 2025
@EduardDurech
[model] feat: Add Apertus (#3295)
3c9b884 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
wlf-darkmatter pushed a commit to wlf-darkmatter/verl that referenced this pull request Sep 13, 2025
@EduardDurech @wlf-darkmatter
[model] feat: Add Apertus (volcengine#3295)
8071ffd 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
@andresnowak andresnowak mentioned this pull request Sep 21, 2025
Open
@EduardDurech
Copy link
Contributor Author

EduardDurech commented Sep 21, 2025

@andresnowak has a draft PR for tests #41037 if you guys want to check, in the meantime there are xIELU CUDA parity issues (already known issue) I asked the group about, will see if that's fixed before and included

VocabVictor pushed a commit to VocabVictor/verl-plus that referenced this pull request Sep 24, 2025
@EduardDurech
[model] feat: Add Apertus (#3295)
f0a09a3 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
masoudhashemi pushed a commit to masoudhashemi/verl that referenced this pull request Oct 19, 2025
@EduardDurech @masoudhashemi
[model] feat: Add Apertus (volcengine#3295)
c4a41cd 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
techkang pushed a commit to techkang/verl that referenced this pull request Oct 31, 2025
@EduardDurech @techkang
[model] feat: Add Apertus (volcengine#3295)
419f820 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
mtian8 pushed a commit to mtian8/verl that referenced this pull request Nov 1, 2025
@EduardDurech
[model] feat: Add Apertus (volcengine#3295)
a03521b 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
chenjiaoAngel added a commit to chenjiaoAngel/verl that referenced this pull request Nov 14, 2025
@chenjiaoAngel
[model] feat: Add Apertus (volcengine#3295)
720c60a 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
NenoL2001 pushed a commit to NenoL2001/verl that referenced this pull request Nov 26, 2025
@EduardDurech
[model] feat: Add Apertus (volcengine#3295)
b8726a0 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
paolo328 added a commit to paolo328/Verl that referenced this pull request Nov 27, 2025
@paolo328
[model] feat: Add Apertus (#3295)
e980e99 
Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

- xIELU Activation
- QK-norm

Associated Transformers PR
huggingface/transformers#39381
Associated vLLM PR vllm-project/vllm#23068
Associated SGLang PR sgl-project/sglang#9774

GSM8K
<img width="430" height="262" alt="image"
src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed"
/>
<img width="436" height="266" alt="image"
src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08"
/>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ArthurZucker ArthurZucker ArthurZucker left review comments

@Cyrilvallez Cyrilvallez Cyrilvallez approved these changes

+1 more reviewer

@dhia680 dhia680 dhia680 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

New model

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@EduardDurech @chiffa @ArthurZucker @Cyrilvallez @dhia680 @haeggee