Webdataset dataset builder

[lhoestq]

Allow load_dataset to support the Webdataset format.

It allows users to download/stream data from local files or from the Hugging Face Hub.

Moreover it will enable the Dataset Viewer for Webdataset datasets on HF.

Implementation details

• I added a new Webdataset builder
• dataset with TAR files are now read using the Webdataset builder
• Basic decoding from webdataset is used by default, except unsafe ones like pickle
• HF authentication support is done with xopen

TODOS

• tests
• docs

[mariosasko]

Thanks for working on this!

A high-level review :)

[mariosasko]

Maybe we can make the module inference more robust by inspecting the contents of TAR archives (e.g., consecutive files with the same name (stem) but different extensions)

[lhoestq]

Some webdatasets may contain only one field (e.g. only images) so I'm not sure it would make sense.

Also I like the idea of keeping the TAR loading simple and only support webdataset for TAR

[mariosasko]

But this would break some existing repos on the Hub, no?

[lhoestq]

TAR has never been supported on the Hub, what would it break ?

[lhoestq]

Ah you mean that some TAR datasets not in webdataset format won't work properly.

Maybe I can add an error message if the first webdataset examples don't have the same type

[mariosasko]

Nevermind, I thought this could fail on

datasets/src/datasets/load.py

Lines 449 to 450 in 27d1fe5

	elif ext == ".zip":
	return infer_module_for_data_files_list_in_archives(data_files_list, download_config=download_config)

But this logic only inspects ZIP (and ignores TAR) archives :)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lhoestq]

I added an error message if the first examples don't appear to be in webdataset format

"The TAR archives of the dataset should be in Webdataset format, "
"but the files in the archive don't share the same prefix or the same types."

[lhoestq]

@mariosasko could you review this ? I think it's fine to have webdataset as an optional dependency for now, then depending on usage and user feedbacks see if it makes sense to have our own implementation or not

[mariosasko]

We should also do the same for the Audio feature, no?

[lhoestq]

Yes correct, that's for another PR :p

[mariosasko]

Ok, feel free to add a TO-DO.

[lhoestq]

I just removed the dependency on webdataset @mariosasko :)

[mariosasko]

A couple of nits, but looks good other than that

[lhoestq]

took your comments into account, lmk if you see anything else

[mariosasko]

Nice work, LGTM!