Releases: tiktoken-go/tokenizer
Releases · tiktoken-go/tokenizer
Lazy loading maps
Vocabulary maps are now lazy loaded when use for the first time. This should minimize the amount of memory consumed and start-up time.
v0.0.8: 🪄 clean up the bpe algorithm
Renamed variables struct to reflect what they do a little better.