Popular repositories Loading

1. SaGe SaGe Public

   Code for SaGe subword tokenizer (EACL 2023)

   Python 27 6

2. tokenizers_intrinsic_benchmark tokenizers_intrinsic_benchmark Public

   Code for the paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods"

   Python 12 1

3. Splintering Splintering Public

   Splintering Nonconcatenative Languages for Better Tokenization

   Python 2 1

4. MeLeL-BiVERT MeLeL-BiVERT Public

   MT evaluation - Carinne Thesis

   Jupyter Notebook

5. Privacy-Preserving-Token-Manipulation Privacy-Preserving-Token-Manipulation Public

   Python

6. probing-subphonemes probing-subphonemes Public

   Python