Datasets created with `push_to_hub` can't be accessed in offline mode

[TevenLeScao]

Describe the bug

In offline mode, one can still access previously-cached datasets. This fails with datasets created with push_to_hub.

Steps to reproduce the bug

in Python:

import datasets
mpwiki = datasets.load_dataset("teven/matched_passages_wikidata")

in bash:

export HF_DATASETS_OFFLINE=1

in Python:

import datasets
mpwiki = datasets.load_dataset("teven/matched_passages_wikidata")

Expected results

datasets should find the previously-cached dataset.

Actual results

ConnectionError: Couln't reach the Hugging Face Hub for dataset 'teven/matched_passages_wikidata': Offline mode is enabled

Environment info

• datasets version: 1.16.2.dev0
• Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
• Python version: 3.8.10
• PyArrow version: 3.0.0

[lhoestq]

Thanks for reporting. I think this can be fixed by improving the CachedDatasetModuleFactory and making it look into the parquet cache directory (datasets from push_to_hub are loaded with the parquet dataset builder). I'll look into it

[dangne]

Hi, I'm having the same issue. Is there any update on this?

[lhoestq]

We haven't had a chance to fix this yet. If someone would like to give it a try I'd be happy to give some guidance

[JohnGiorgi]

@lhoestq Do you have an idea of what changes need to be made to CachedDatasetModuleFactory? I would be willing to take a crack at it. Currently unable to train with datasets I have push_to_hub on a cluster whose compute nodes are not connected to the internet.

It looks like it might be this line:

datasets/src/datasets/load.py

Line 994 in 0c1d099

importable_directory_path = os.path.join(dynamic_modules_path, "datasets", self.name.replace("/", "--"))

Which wouldn't pick up the stuff saved under "datasets/allenai___parquet/*". Additionally, the datasets saved under "datasets/allenai___parquet/*" appear to have hashes in their name, e.g. "datasets/allenai___parquet/my_dataset-def9ee5552a1043e". This would not be detected by CachedDatasetModuleFactory, which currently looks for subdirectories

datasets/src/datasets/load.py

Lines 995 to 999 in 0c1d099

	hashes = (
	[h for h in os.listdir(importable_directory_path) if len(h) == 64]
	if os.path.isdir(importable_directory_path)
	else None
	)

[lhoestq]

importable_directory_path is used to find a dataset script that was previously downloaded and cached from the Hub

However in your case there's no dataset script on the Hub, only parquet files. So the logic must be extended for this case.

In particular I think you can add a new logic in the case where hashes is None (i.e. if there's no dataset script associated to the dataset in the cache).

In this case you can check directly in the in the datasets cache for a directory named <namespace>__parquet and a subdirectory named <config_id>. The config_id must match {self.name.replace("/", "--")}-*.

In your case those two directories correspond to allenai___parquet and then allenai--my_dataset-def9ee5552a1043e

Then you can find the most recent version of the dataset in subdirectories (e.g. sorting using the last modified time of the dataset_info.json file).

Finally, we will need return the module that is used to load the dataset from the cache. It is the same module than the one that would have been normally used if you had an internet connection.

At that point you can ping me, because we will need to pass all this:

• module_path = _PACKAGED_DATASETS_MODULES["parquet"][0]
• hash it corresponds the name of the directory that contains the .arrow file, inside <namespace>__parquet/<config_id>
• builder_kwargs = {"hash": hash, "repo_id": self.name, "config_id": config_id}
  and currently config_id is not a valid argument for a DatasetBuilder

I think in the future we want to change this caching logic completely, since I don't find it super easy to play with.

[zaccharieramzi]

Hi! Is there a workaround for the time being?
Like passing data_dir or something like that?

I would like to use this diffuser example on my cluster whose nodes are not connected to the internet. I have downloaded the dataset online form the login node.

[lhoestq]

Hi ! Yes you can save your dataset locally with my_dataset.save_to_disk("path/to/local") and reload it later with load_from_disk("path/to/local")

(removing myself from assignees since I'm currently not working on this right now)