Make OpenAIGPTTokenizer work with SpaCy 2.x and 3.x

[cody-moveworks]

SpaCy 3.x introduced an API change to creating the tokenizer that
breaks OpenAIGPTTokenizer. The old API for creating the tokenizer in
SpaCy 2.x no longer works under SpaCy 3.x, but the new API for creating
the tokenizer in SpaCy 3.x DOES work under SpaCy 2.x. Switching to the
new API should allow OpenAIGPTTokenizer to work under both SpaCy 2.x and
SpaCy 3.x versions.

Fixes #14449

[cody-moveworks]

I'm not able to test the changes locally. When I run pip install -e ".[dev]", I encounter a build error and the install fails. It doesn't look like pytest coverage is sufficient here either since SpaCy and ftfy are not required by transformers, and the code can execute correctly without them being installed.

[cody-moveworks]

@patrickvonplaten @LysandreJik Not sure why but I can't tag you as reviewers on GitHub, so tagging you in comments here.

[patrickvonplaten]

That looks good to me. @LysandreJik do you think it's worth adding a test the depends on a spacys install?

[LysandreJik]

Yes, why not! GPT-2 is among our most used models, so I think testing that the tokenization behaves correctly across all possibilities is important. Would you like to take a stab at it @cody-moveworks?

In order to do so, you could start by adding an is_spacy_available method in file_utils.py, analog to other methods such as is_vision_available here:

transformers/src/transformers/file_utils.py

Lines 507 to 508 in 2e9af29

	def is_vision_available():
	return importlib.util.find_spec("PIL") is not None

Then it would require defining a require_spacy unittest decorator in testing_utils.py, such as the require_vision here:

transformers/src/transformers/testing_utils.py

Lines 404 to 412 in 2e9af29

	def require_vision(test_case):
	"""
	Decorator marking a test that requires the vision dependencies. These tests are skipped when torchaudio isn't
	installed.
	"""
	if not is_vision_available():
	return unittest.skip("test requires vision")(test_case)
	else:
	return test_case

Thirdly, you can add a test in the tests/test_tokenization_gpt2.py file with the @require_spacy decorator, which will only run when SpaCy is installed.

And finally, we can modify this CircleCI run:

transformers/.circleci/config.yml

Lines 538 to 565 in 2e9af29

	run_tests_custom_tokenizers:
	working_directory: ~/transformers
	docker:
	- image: circleci/python:3.7
	environment:
	RUN_CUSTOM_TOKENIZERS: yes
	TRANSFORMERS_IS_CI: yes
	steps:
	- checkout
	- restore_cache:
	keys:
	- v0.4-custom_tokenizers-{{ checksum "setup.py" }}
	- v0.4-{{ checksum "setup.py" }}
	- run: pip install --upgrade pip
	- run: pip install .[ja,testing,sentencepiece,jieba]
	- run: python -m unidic download
	- save_cache:
	key: v0.4-custom_tokenizers-{{ checksum "setup.py" }}
	paths:
	- '~/.cache/pip'
	- run: |
	if [ -f test_list.txt ]; then
	python -m pytest -s --make-reports=tests_custom_tokenizers ./tests/test_tokenization_bert_japanese.py | tee tests_output.txt
	fi
	- store_artifacts:
	path: ~/transformers/tests_output.txt
	- store_artifacts:
	path: ~/transformers/reports

So that it:

1. Installs SpaCy
2. Runs the tokenization GPT-2 test file!

[cody-moveworks]

@LysandreJik Thanks for the detailed walkthrough of the changes needed to add testing. I'll take a stab at it.

[cody-moveworks]

@LysandreJik I made the changes and all checks are passing. Can you take a look?

[LysandreJik]

That's very nice! This looks good to me, @cody-moveworks, tested it locally.

Thank you, merging!