Unpack dl_manager.iter_files to allow parallization

[mariosasko]

Iterate over data files outside dl_manager.iter_files to allow parallelization in streaming mode.

(The issue reported here)

PS: Another option would be to override FilesIterable.__getitem__ to make it indexable and check for that type in _shard_kwargs and n_shards, but IMO this solution adds too much unnecessary complexity.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lhoestq]

Cool thanks ! Yup it sounds like the right solution.

It looks like _generate_tables needs to be updated as well to fix the CI

[lhoestq]

Sounds good, thanks !