Add concatenate_datasets for iterable datasets

[lhoestq]

concatenate_datasets currently only supports lists of datasets.Dataset, not lists of datasets.IterableDataset like interleave_datasets

Fix #2564

I also moved _interleave_map_style_datasets from combine.py to arrow_dataset.py, since the logic depends a lot on the Dataset object internals

And I moved concatenate_datasets from arrow_dataset.py to combine.py to have it with interleave_datasets (though it's also copied in arrow_dataset module for backward compatibility for now)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[mariosasko]

Looks good already!

There is a slight difference in concatenate_datasets between the version for map-style datasets and the one for iterable datasets:

• if axis=0, the map-style version checks the feature types (features can be None in iterable datasets, so it's ok not to have this check) of the shared columns, but doesn't require the equal set of column names among the input datasets, i.e., the following works:

   >>> from datasets import *
   >>> a = Dataset.from_dict({"a": [1, 2]})
   >>> b = Dataset.from_dict({"b": ["aa", "bb"]})
   >>> concatenate_datasets([a, b])

  We need to address this.
• if axis=1, the map-style version checks the length of input datasets (besides the column names check for duplicates) and throws an error if the lengths are not equal. Somewhat expected that this is ignored for iterable datasets, so it can stay as-is IMO.

Two nits in code:

[lhoestq]

Thanks ! I addressed your comments :)

There is a slight difference in concatenate_datasets between the version for map-style datasets and the one for iterable datasets

Indeed, here is what I did to fix this:

• axis 0: fill missing columns with None.
  (I first iterate over the input datasets to infer their columns from the first examples, then I set the features of the resulting dataset to be the merged features)
  This is consistent with non-streaming concatenation

• axis 1: fill the missing rows with None, for consistency with axis 0
  (but let me know what you think, I can still revert this behavior and raise an error when one of the dataset runs out of examples)
  We might have to align the non-streaming concatenation with this behavior though, for consistency. What do you think ?

[mariosasko]

Cool!

We might have to align the non-streaming concatenation with this behavior though, for consistency. What do you think ?

Yes, we can address that in a subsequent PR

[lhoestq]

Added more comments as suggested, and some typing

While factorizing _apply_features_types for both IterableDataset and TypedExamplesIterable, I fixed a missing token_per_repo_id that was not passed to TypedExamplesIteable

Let me know what you think now @mariosasko

[mariosasko]

Looks good! Thanks!