Search results

<code>BaseTokenizer.normalize</code> method calls non-existent <code>_tokenizer.normalize</code>

Status: Open.

#1882 In huggingface/tokenizers;

· T90REAL opened on Oct 31, 2025

Provide byte-level offsets for effective alignment in Cross-Tokenizer On-Policy Distillation

Status: Open.

#1881 In huggingface/tokenizers;

· JqzChandler opened on Oct 30, 2025

rust tokenizer

Status: Open.

#1879 In huggingface/tokenizers;

· gogo2464 opened on Oct 24, 2025

encode bytes directly

Status: Open.

#1877 In huggingface/tokenizers;

· tsengalb99 opened on Oct 19, 2025

Add an API for tokens to ids

Status: Open.

#1876 In huggingface/tokenizers;

· winstxnhdw opened on Oct 15, 2025

Add Middle-Truncation Option to Tokenizer

Feature Request

Status: Open.

#1871 In huggingface/tokenizers;

· mxngjxa opened on Sep 30, 2025

How to set the cache_dir in the Rust implementation?

Status: Open.

#1868 In huggingface/tokenizers;

· sambaPython24 opened on Sep 24, 2025

WordPiece can't always avoid &lt;unk&gt; even with ByteLevel pretokenization.

Status: Open.

#1863 In huggingface/tokenizers;

· craigschmidt opened on Sep 11, 2025

Breaking change in 0.21.2

Status: Open.

#1855 In huggingface/tokenizers;

· phil-opp opened on Aug 27, 2025

Question: bert normalizer CJK handling

Status: Open.

#1854 In huggingface/tokenizers;

· xiaoshihou514 opened on Aug 26, 2025

SentencePieceBPE + Unicode NFD preprocessing leads to noise ?

Status: Open.

#1851 In huggingface/tokenizers;

· PonteIneptique opened on Aug 24, 2025

memory allocation of 21474836480 bytes failed

Status: Open.

#1847 In huggingface/tokenizers;

· lostmsu opened on Aug 16, 2025