Tokenization for downstream tasks

[danigoju]

First of all thank you very much for your work.

I am working on the long text classification task, and given the spectacular results of MEGA for long sequence modelling I wanted to use it for this task. The only thing that I haven't figured out how to do is the tokenization of my text samples, so I was wondering if someone could help me out on how to tokenize my text with the dict that is obtained from a checkpoint like the LRA one from the text task.

Thank you very much for your time

[danigoju]

Also I have seen that you plan to release a checkpoint for machine translation (WMT'14), which could be useful for my task as the encoder is an english LM. Around what date do you target to release this checkpoint?
Many thanks