how to replace the existing token in a tokenizer

[muziyongshixin]

Feature request

I have a tokenizer which have lots of preserved tokens like bellow:

 '<reserved_7>': 100,
 '<reserved_8>': 101,
 '<reserved_9>': 102,
 '<reserved_10>': 103,
 '<reserved_11>': 104,
 '<reserved_12>': 105,
 '<reserved_13>': 106,
 '<reserved_14>': 107,

I want to replace the '<reserved_7>' with '<|im_start|>' and replace '<reserved_8>' with '<|im_end|>'

what I want to get is a tokenizer which can act as below:
tokenizer.encode('<|im_start|>') => 100

Motivation

I want to replace the '<reserved_7>' with '<|im_start|>' and replace '<reserved_8>' with '<|im_end|>'

Your contribution

no

[amyeroberts]

Hi @muziyongshixin, thanks for raising an issue!

This is a question best placed in our forums. We try to reserve the github issues for feature requests and bug reports.

To learn about how to how to modify the tokenizers, you can check out the documentation, 1, 2. For example, you can add tokens to the tokenzers vocabulary by using the add_tokens method.

[muziyongshixin]

Hi @muziyongshixin, thanks for raising an issue!

This is a question best placed in our forums. We try to reserve the github issues for feature requests and bug reports.

To learn about how to how to modify the tokenizers, you can check out the documentation, 1, 2. For example, you can add tokens to the tokenzers vocabulary by using the add_tokens method.

thanks for your reply@amyeroberts

I know the add_tokens function, but I don't want to change the vocabulary size, this will introduce some inconvience in finetuning process(like resize the embedding layer and the lm_head).
So I am wondering whether there is a method to replace the existing tokens.

P.S. I am using a sentencepiece tokenizer which is released in this baichuan2 repo

[amyeroberts]

@ArthurZucker will be best placed to answer this. I believe add_tokens or add_special_tokens is the safe way to do this, as it correctly keeps track of the modifications vs. from the loaded checkpoint and ensures the correct preprocessing is applied before mapping to the token id. It should be possible to define your own vocab files to load a tokenizer but I don't know about the guarantees of the behaviour vs the original tokenizer.

[ArthurZucker]

hey! You should modify manually both the added_tokens_decoder field (saved in the tokenizer_config.json ) and the added_tokens field (saved in the tokenizer.json). We don't really support this manually, but that is the recommended way to do it! (If the reserved tokens were already part of the vocab, so not AddedTokens, then you have to overwrite the vocab as well, the vocab files, to make sure they are removed) that would be hard than if it's just the content of the added tokens that you are trying to modify 😉

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[MohamedAliRashad]

I know this is an old discussion but i think there should be a way to edit tokenizers vocab programmatically. It just feels intuitive for such a functionality to be present