Skip to content

Releases: tiktoken-go/tokenizer

Lazy loading maps

10 May 19:49
7f65fe7

Choose a tag to compare

Vocabulary maps are now lazy loaded when use for the first time. This should minimize the amount of memory consumed and start-up time.

v0.0.8: 🪄 clean up the bpe algorithm

09 Apr 21:48

Choose a tag to compare

Renamed variables struct to reflect what they do a little better.