load_from_disk and save_to_disk are not compatible with each other

[roholazandie]

Describe the bug

load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

Steps to reproduce the bug

from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")

Expected results

It is expected that these two functions be the reverse of each other without more manipulation

Actual results

FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

Environment info

• datasets version: 1.6.2
• Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.7.10
• PyTorch version (GPU?): 1.8.1+cu102 (True)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[mariosasko]

Hi,

load_dataset returns an instance of DatasetDict if split is not specified, so instead of Dataset.load_from_disk, use DatasetDict.load_from_disk to load the dataset from disk.

[roholazandie]

Thanks it worked!

[thomwolf]

Though I see a stream of issues open by people lost between datasets and datasets dicts so maybe there is here something that could be better in terms of UX. Could be better error handling or something else smarter to even avoid said errors but maybe we should think about this. Reopening to use this issue as a discussion place but feel free to open a new open if you prefer @lhoestq @albertvillanova

[lhoestq]

We should probably improve the error message indeed.

Also note that there exists a function load_from_disk that can load a Dataset or a DatasetDict. Under the hood it calls either Dataset.load_from_disk or DatasetDict.load_from_disk:

from datasets import load_from_disk

dataset_dict = load_from_disk("path/to/dataset/dict")
single_dataset = load_from_disk("path/to/single/dataset")

[lhoestq]

I just opened #2437 to improve the error message

[albertvillanova]

Superseded by #2462