Feature Request: Aperture model support

[pwilkin]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add support for Aperture family of models: a Swiss fully open-source (Apache 2.0 weights AND full training data) initiative with models trained on strongly multilingual data.

As per Transformers PR: huggingface/transformers#39381:

Main modifications from Llama

• xIELU Activation
• QK-norm

I guess the xIELU activation is the hard part here.

Motivation

This is an official (developed by the Swiss AI Initiative: https://www.swiss-ai.org/) model with a feature that is strongly lacking from many current open models: comprehensive support for a very large language set.

Possible Implementation

No response

[pwilkin]

Sorry, didn't realize there was already one, didn't refresh before 😀