Skip to content

Conversation

@albertvillanova
Copy link
Member

@albertvillanova albertvillanova commented Jul 7, 2021

Use open in JSON dataset builder, so that it can be patched with xopen for streaming.

Close #2607.

Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thank you :)

@lhoestq lhoestq merged commit 3cbc28f into huggingface:master Jul 8, 2021
@albertvillanova albertvillanova added this to the 1.10 milestone Jul 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming local gzip compressed JSON line files is not working

2 participants