Convert polars DataFrame back to datasets

[ljw20180420]

Feature request

This returns error.

from datasets import Dataset

dsdf = Dataset.from_dict({"x": [[1, 2], [3, 4, 5]], "y": ["a", "b"]})
Dataset.from_polars(dsdf.to_polars())

ValueError: Arrow type large_list<item: int64> does not have a datasets dtype equivalent.

Motivation

When datasets contain Sequence data type, it will be converted to Arrow type large_list. However, the reverse (from large_list to Sequence) does not work.

Your contribution

No

[lhoestq]

Hi ! Thanks for reporting :)

We don't support large_list yet, though it should be added to Sequence IMO (maybe with a parameter large=True ?)