Running AutoTokenizer.from_pretrained with Mistral V3 is actually loading LlamaTokenizer

[matheus-prandini]

System Info

• Ubuntu 22.04
• Python 3.10.12
• Transformers Version: from source

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I'm trying to load MistralTokenizer using AutoTokenizer from mistral model in the following code snippet:

from transformers import AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.3"
auto_tokenizer = AutoTokenizer.from_pretrained(model_id)

When I inspect auto_tokenizer variable, then I get LlamaTokenizerFast:

LlamaTokenizerFast(name_or_path='mistralai/Mistral-7B-Instruct-v0.3', vocab_size=32768, model_max_length=1000000000000000019884624838656, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	...
}

I don't know if I'm missing something, but it is loading a different tokenizer than I expected.

Expected behavior

IMHO it should instantiate a MistralTokenizer.v3() tokenizer as implemented in mistral-common. I checked the TOKENIZER_MAPPING object, and Mistral isn't even listed there.

[matheus-prandini]

I noticed from this transformers.AutoTokenizer documentation that the Mistral model is indeed using LlamaTokenizer. Why is that? I'm experiencing issues with control tokens that aren't encoded by MistralTokenizer but are encoded by AutoTokenizer (Llama). Perhaps this issue is not a bug. It could be a feature.

[amyeroberts]

@matheus-prandini It's not uncommon for models to use the same processing class e.g. mistral using the LlamaTokenizer if the logic is the same for both e.g. Phi3 uses the llama tokenizer too. In fact, there's no "mistral tokenizer" implemented in transformers as this would mean just copying all of the llama tokenizer code, which we'd like to avoid.

You can see the mappings defined here.

[matheus-prandini]

@amyeroberts Thank you very much for your response! My idea is to use an extension of MistralTokenizer to add new tokens. Since AutoTokenizer has the add_tokens method, my initial plan was to load the Mistral model in AutoTokenizer and add the new tokens through it. However, there is an issue because control tokens are not encoded by MistralTokenizer but are encoded by AutoTokenizer. I'll try to extend SentencePieceTokenizer from Mistral to accomplish this. If that doesn't work, I'll attempt to adjust AutoTokenizer with MistralTokenizer by adding control tokens and any other necessary elements...

[amyeroberts]

@matheus-prandini I'm not sure I completely followed what you're trying to achieve here, but the forums are a great place to ask the community about your project and to ask for guidance and help.

Just a few points of clarification / comments :

my initial plan was to load the Mistral model in AutoTokenizer

I'm guessing you meant tokenizer here rather than model.

However, there is an issue because control tokens are not encoded by MistralTokenizer but are encoded by AutoTokenizer

• There is no MistralTokenizer in the transformers library. Loading a tokenizer using AutoTokenizer and a mistral checkpoint will load a LlamaTokenizer
• If you instantiate a tokenizer and add tokens, this tokenizer class (llamatokenizer in this case) will encode those added tokens. You can then save this tokenizer out and load it with AutoTokenizer e.g.

from transformers import tokenizer 

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
tokenizer.add_tokens(["foo", "bar"])

tokenizer.save_pretrained("my_new_tokenizer")
tokenizer = AutoTokenizer.from_pretrained("my_new_tokenizer")

cc @ArthurZucker who knows more about the ins and outs of the tokenizers

[matheus-prandini]

@amyeroberts Sorry if I didn't make it clear before, but my goal is to expand the Mistral tokenizer's vocabulary with new special tokens.

I'm guessing you meant tokenizer here rather than model.

Yes, it's the tokenizer. Sorry!

If you instantiate a tokenizer and add tokens, this tokenizer class (llamatokenizer in this case) will encode those added tokens. You can then save this tokenizer out and load it with AutoTokenizer e.g.

This is the second approach I'm trying. The main issue is the encoding differences between AutoTokenizer and MistralTokenizer regarding Mistral's control tokens. For example, the control token [INST] should not be encoded using its token_id (considering that the token_id for [INST] is equal to 3) to avoid prompt injection. MistralTokenizer encodes it into three tokens ('[', 'INST', ']'), while AutoTokenizer encodes it into a single token ('[INST]') plus the begin_of_sentence token as shown in the example below. I want to add new control tokens and to adjust this behavior in AutoTokenizer.

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from transformers import AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.3"

mistral_tokenizer = MistralTokenizer.v3()
auto_tokenizer = AutoTokenizer.from_pretrained(model_id)

example = '[INST]'
mistral_result = mistral_tokenizer.instruct_tokenizer.tokenizer.encode(example, bos=False, eos=False)
print(f"Mistral Result: {mistral_result} - Len: {len(mistral_result)}")
auto_result = auto_tokenizer.encode(example)
print(f"Auto Result: {auto_result} - Len: {len(auto_result)}")

Results:

Mistral Result: [1501, 17057, 29561] - Len: 3 (['[', 'INST', ']'])
Auto Result: [1, 3] - Len: 2 (['<s>', '[INST]'])

[matheus-prandini]

@amyeroberts I managed to accomplish what I wanted. I had to study SentencePiece and modify the protobuf to add new tokens, and then load it into MistralTokenizer. This notebook was very helpful for that: https://github.com/google/sentencepiece/blob/master/python/add_new_vocab.ipynb. Thank you for your responses!

[ArthurZucker]

@matheus-prandini I think there is a missundertanding here, we specifically designed tokenizers to make it easy to add new tokens and expend the vocab. Simply calling tokenizer.add_tokens just solves this as long as the legacy flag is set to False. There is a big warning when you don't properly set this, and your mistral result show the token that is split is exactly because the token is either not added, or normalized!

[matheus-prandini]

@ArthurZucker Got it! Thank you for the explanation!

[mailvijayasingh]

@matheus-prandini - https://github.com/google/sentencepiece/blob/master/python/add_new_vocab.ipynb. Thank you for your responses!

I have the same issue, how did you use the attached notebook exactly to change the tokenizer's behaviors. Thanks!

[matheus-prandini]

@mailvijayasingh Hi! I've created a notebook to edit tokens directly in SentencePiece, as shown below. Then, I simply loaded the new protobuf into the Mistral tokenizer. Hope this helps you!

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding new special tokens to pre-trained sentencepiece model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load pre-trained sentencepiece model\n",
    "Pre-trained model is needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROTO_MODEL_PATH = '/workspace/neolang-datasets/neolang-tokenizers/mistral_instruct_tokenizer_240323.model.v3'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "587591"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sentencepiece.sentencepiece_model_pb2 as model\n",
    "m = model.ModelProto()\n",
    "m.ParseFromString(open(PROTO_MODEL_PATH, \"rb\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load tokens want to add\n",
    "Prepare the list of new tokens want to add"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_tokens = [\n",
    "    '[DT]',\n",
    "    '[/DT]'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add new tokens to sentencepiece model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "for token in special_tokens:\n",
    "    new_token = model.ModelProto().SentencePiece()\n",
    "    new_token.piece = token\n",
    "    new_token.score = 0\n",
    "    new_token.type = 4 # control token\n",
    "    m.pieces.append(new_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[piece: \"[DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED,\n",
       " piece: \"[/DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.pieces[32768:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save new sentencepiece model\n",
    "Load the new sentencepiece model to your NLP system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "NEW_PROTO_MODEL_PATH = '/workspace/neolang-datasets/neolang-tokenizers/mistral_instruct_tokenizer_240323_test.model.v3'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(NEW_PROTO_MODEL_PATH, 'wb') as f:\n",
    "    f.write(m.SerializeToString())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[piece: \"[DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED,\n",
       " piece: \"[/DT]\"\n",
       " score: 0.0\n",
       " type: USER_DEFINED]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sentencepiece.sentencepiece_model_pb2 as model\n",
    "m = model.ModelProto()\n",
    "m.ParseFromString(open(NEW_PROTO_MODEL_PATH, \"rb\").read())\n",
    "\n",
    "m.pieces[32768:]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}