Support cloud storage in load_dataset

[lhoestq]

Would be nice to be able to do

data_files=["s3://..."]  # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)

The idea would be to use fsspec as in download_and_prepare and save_to_disk.

This has been requested several times already. Some users want to use their data from private cloud storage to train models

related:

#3490
#5244
forum

[alexjc]

Or for example an archive on GitHub releases! Before I added support for JXL (locally only, PR still pending) I was considering hosting my files on GitHub instead...

[iceboundflame]

+1 to this. I would like to use 'audiofolder' with a data_dir that's on S3, for example. I don't want to upload my dataset to the Hub, but I would find all the fingerprinting/caching features useful.

[Dref360]

Adding to the conversation, Dask also uses fsspec for this feature.

Dask: How to connect to remote data

Happy to help on this feature :D

[eballesteros]

+1 to this feature request since I think it also tackles my use-case. I am collaborating with a team, working with a loading script which takes some time to generate the dataset artifacts. It would be very handy to use this as a cloud cache to avoid duplicating the effort.

Currently we could use builder.download_and_prepare(path_to_cloud_storage, storage_options, ...) to cache the artifacts to cloud storage, but then builder.as_dataset() yields NotImplementedError: Loading a dataset cached in SomeCloudFileSystem is not supported

[lhoestq]

Makes sense ! If you want to load locally a dataset that you download_and_prepared on a cloud storage, you would use load_dataset(path_to_cloud_storage) indeed. It would download the data from the cloud storage, cache them locally, and return a Dataset.

[kyamagu]

It seems currently the cached_path function handles all URLs by get_from_cache that only supports ftp and http(s) here:

datasets/src/datasets/utils/file_utils.py

Line 181 in b5672a9

if is_remote_url(url_or_filename):

I guess one can add another condition that handles s3:// or gs:// URLs via fsspec here.

[dwyatte]

I could use this functionality, so I put together a PR using @kyamagu's suggestion to use fsspec in datasets.utils.file_utils

#5580

[lhoestq]

Thanks @dwyatte for adding support for fsspec urls

Let me just reopen this since the original issue is not resolved