Author: davide_delbianco
Version: 0.0.1
Type: tool
Tokenizing text using various tokenizers from transformers.AutoTokenizer and Tiktoken. Returns the token count.
Note
Access to huggingface.co is required to download the tokenizer when using the transformers tokenizers.
Warning
Due to the plugin execution environment limitations of cloud.dify.ai, this plugin is only available for use in self-hosted environments.
You can choose the tokenizer by specifying the tokenizer parameter for transformers or the encoder parameter for tiktoken. The default is gpt2. For transformers tokenizers please check on https://huggingface.co/docs/transformers/v4.52.3/en/model_doc/auto#transformers.AutoTokenizer if the tokenizer you want to use is supported.
Please refer to the privacy policy for more information on how your data is handled.
Last updated: 2025-06-30