Support remote cache_dir

[albertvillanova]

This PR implements complete support for remote cache_dir. Before, the support was just partial.

This is useful to create datasets using Apache Beam (parallel data processing) builder with cache_dir in a remote bucket, e.g., for Wikipedia dataset.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lhoestq]

Awesome ! Just one minor comment: you can use xjoin directly to support both URLs and local paths

[albertvillanova]

@lhoestq thanks for your review.

Please note that xjoin cannot be used in this context, as it always returns a POSIX path string and this is not suitable on Windows machines.

[lhoestq]

xjoin returns windows paths (not posix) on windows, since it just extendsos.path.join

Actually you are right.

datasets/src/datasets/utils/streaming_download_manager.py

Lines 104 to 105 in 08ec04c

	if is_local_path(a):
	a = Path(a, *p).as_posix()

Though this is not an issue because posix paths (as returned by Path().as_posix()) work on windows. That's why we can replace os.path.join with xjoin in streaming mode. They look like c:/Program Files/ or something (can't confirm right now, I don't have a windows with me)

[albertvillanova]

Until now, we have always replaced "/" in paths with os.path.join (os.sep,...) in order to support Windows paths (that contain r"\").

Now, you suggest ignoring this and work with POSIX strings (with "/").

As an example, when passing cache_dir=r"C:\Users\Username\.mycache":

• Until now, it results in self._cache_downloaded_dir = r"C:\Users\Username\.mycache\downloads"
• If we use xjoin, it will give self._cache_downloaded_dir = "C:/Users/Username/.mycache/downloads"

You say this is OK and we don't care if we work with POSIX strings on Windows machines.

I'm incorporating your suggested changes then...

[albertvillanova]

Also note that using xjoin, if we pass cache_dir="C:\\Users\\Username\\.mycache", we get:

• self._cache_dir_root = "C:\\Users\\Username\\.mycache"
• self._cache_downloaded_dir = "C:/Users/Username/.mycache/downloads"

[lhoestq]

It looks like it broke the CI on windows :/ maybe this was not a good idea, sorry

[lhoestq]

Thanks and sorry for the bad indications ;)