add openwebtext2

[richarddwang]

openwebtext2 is part of EleutherAI/The Pile, but AFAIK, The Pile dataset blend all sub datasets together thus we are not able to use just one of its sub dataset from The Pile data. So I create an independent dataset using The Pile preliminary components.

When I was creating dataset card. I found there is room for creating / editing dataset card. I've made it an issue. #2797

Also I am wondering whether the import of The Pile dataset is actively undertaken (because I may need it recently)? #1675

[richarddwang]

It seems we need to pip install jsonlines to pass the checks ?

[lhoestq]

Hi ! Do you really need jsonlines ? I think it simply uses json.loads under the hood.

Currently the test are failing because jsonlines is not part of the extra requirements TESTS_REQUIRE in setup.py

So either you can replace jsonlines with a simple for loop on the lines of the files and use json.loads, or you can add TESTS_REQUIRE to the test requirements (but in this case users will have to install it as well).

[richarddwang]

Thanks for your suggestion. I now know io and json lines format better and has changed jsonlines to just readlines.

[lhoestq]

Awesome thanks !

I'm also doing some minor modifications and we can merge :)

[lhoestq]

Suggested change

	- **Homepage:** https://openwebtext2.readthedocs.io/en/latest/
	- **Repository:** [Needs More Information]
	- **Paper:** https://arxiv.org/abs/2101.00027
	- **Leaderboard:** [Needs More Information]
	- **Point of Contact:** [Needs More Information]
	- **Homepage:** [openwebtext2](https://openwebtext2.readthedocs.io/en/latest/)
	- **Repository:** [Needs More Information]
	- **Paper:** [arXiv](https://arxiv.org/abs/2101.00027)
	- **Leaderboard:** [Needs More Information]
	- **Point of Contact:** [Needs More Information]