[Tokenizer] Support adding special tokens to Qwen tokenizer

[DrownFish19]

PR types

New features

PR changes

APIs

Description

Support adding special tokens to Qwen tokenizer.

Note: Qwen tokenizer does not support adding normal tokens.

example：

from paddlenlp.transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("qwen/qwen-7b")

tokenizer.add_special_tokens({"cls_token": "<CLS>"})

tokenizer.add_special_tokens({"additional_special_tokens":["<img>"]})

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 70.58824% with 5 lines in your changes missing coverage. Please review.

Project coverage is 52.89%. Comparing base (483fff4) to head (f99fe98).
Report is 3 commits behind head on develop.

❗ Current head f99fe98 differs from pull request most recent head 0ea84ca

Please upload reports for the commit 0ea84ca to get more accurate results.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/qwen/tokenizer.py	70.58%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9344      +/-   ##
===========================================
- Coverage    52.91%   52.89%   -0.03%     
===========================================
  Files          673      673              
  Lines       107742   107711      -31     
===========================================
- Hits         57009    56970      -39     
- Misses       50733    50741       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ZHUI]

LGTM