Dedicated tokenizer for byte level transformers

[apehex]

Feature request

There are alternative transformer architectures that handle bytes directly:

• byte latent transformers by Meta
• ByT5 by Google
• tokun by me 👾

Instead of tokenizing according to a vocabulary, the idea would be to get the raw encoding bytes.

Motivation

Combinations of bytes have more expressive power than flat vocabularies and avoid dimensions of 100k in the first and last layers.
A patch of 4 bytes can represent 4294967296 tokens of length 4.

Your contribution

I have a draft that I will PR shortly!

[Rocketknight1]

cc @ArthurZucker - I'm not sure about this, so asking a core maintainer! I think byte-level tokenization is simple enough that it doesn't need a dedicated class yet, but we may change that if we start seeing a lot of models using it (note that we haven't seen any actual released models using the Byte Latent Transformers architecture!)

[apehex]

Indeed, that's very niche! :)

Raw byte tokenization is easy to do manually, but the transformers tokenizers add a lot of quality of life: padding, templating, etc.

I've done a small PR here, just in case.

[hvaara]

What's the current status on blt support in transformers? Any help needed?

xref facebookresearch/blt#5
cc @NielsRogge

[NielsRogge]

Yes we are working on integrating the original checkpoints with facebookresearch/blt#98, however to make BLT Transformers compatible, you would have to follow this guide: https://huggingface.co/docs/transformers/custom_models.

[ArthurZucker]

BLT is coming in #38579