Sort unique_no_split_tokens to make it deterministic

[lhoestq]

The unique_no_split_tokens attribute of tokenizers is not deterministic, and it makes the hashing in the nlp lib return different hashes for the same tokenizer over different sessions.

To fix that I changed its type to a set instead of a list.

Fix #6460

[codecov]

Codecov Report

Merging #6461 into master will decrease coverage by 0.47%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #6461      +/-   ##
==========================================
- Coverage   80.09%   79.62%   -0.48%     
==========================================
  Files         153      153              
  Lines       28005    28005              
==========================================
- Hits        22430    22298     -132     
- Misses       5575     5707     +132     

Impacted Files	Coverage Δ
src/transformers/tokenization_utils.py	90.40% <100.00%> (ø)
src/transformers/modeling_tf_electra.py	25.13% <0.00%> (-70.95%)	⬇️
src/transformers/modeling_tf_utils.py	86.97% <0.00%> (-0.33%)	⬇️
src/transformers/file_utils.py	82.44% <0.00%> (+0.25%)	⬆️
src/transformers/modeling_tf_gpt2.py	94.72% <0.00%> (+22.87%)	⬆️
src/transformers/tokenization_albert.py	87.50% <0.00%> (+58.65%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d94aec...dfb7549. Read the comment docs.

[sgugger]

This looks good to me, however it might be breaking existing code if a user uses list properties on this attribute (probably long shot). Would a sort of the list solve the issue as well? This would allow us to keep the same exposed API. If not, no problem with using a set.

[thomwolf]

I've actually switched in the last version of transformers from a set to a list for the same reason (not deterministic for nlp). Are you sure this really solve the problem @lhoestq ?
Also regarding backward compatibility, I'm fine with changing this from a list to a set @sgugger

[thomwolf]

Maybe we should rather have a sorted list?

[lhoestq]

sorted should solves the issue.

I just tested and a set doesn't solve it actually. I'll change to sorted, thanks @thomwolf

[thomwolf]

This is such an important use-case (and potential source of regression) for us that we may want to add a test on that in nlp or transformers in a not too far future.

[lhoestq]

Yes definitely. Not sure how to test consistency across sessions in the CI though.
I guess we could have tests with hardcoded hashes for some tokenizers but I'm not sure that's ideal.

Or maybe there's a way to do two CI jobs in a row: one to generate the hashes in a first session, and one to verify that they're the same in another session.

[LysandreJik]

LGTM