Dataset order is not deterministic with ZIP archives and `iter_files`

[fxmarty]

Describe the bug

For the beans dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])

Expected behavior

The order of the images is the same on all machines.

Environment info

On the EC2 instance:

- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked

On my local laptop:

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1

On github actions:

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4

[lhoestq]

Thanks for reporting ! The issue doesn't come from shuffling, but from beans row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses dl_manager.iter_files on ZIP archives and the file order doesn't seen to be deterministic and changes across machines

[fxmarty]

Thank you for noticing indeed!

[lhoestq]

This is still a bug, so I'd keep this one open if you don't mind ;)

[mariosasko]

Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files here and here (e.g. fsspec's LocalFileSystem.glob relies on os.scandir, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to load_dataset similar to TFDS' shuffle_files or sort only if the number of data files is small?

[lhoestq]

We already return the result sorted at the end of _resolve_single_pattern_locally and _resolve_single_pattern_in_dataset_repository if I'm not mistaken

[mariosasko]

@lhoestq Oh, you are right. Feel free to ignore my comment.

[albertvillanova]

I think the corresponding PR is ready to be merged 🤗

[fxmarty]

@albertvillanova Thanks for the fix!