Skip to content

largelisttype not supported (.from_polars()) #6834

@Modexus

Description

@Modexus

Describe the bug

The following code fails because LargeListType is not supported.
This is especially a problem for .from_polars since polars uses LargeListType.

Steps to reproduce the bug

import datasets
import polars as pl

df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)

Expected behavior

Convert LargeListType to list.

Environment info

  • datasets version: 2.19.1.dev0
  • Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
  • Python version: 3.12.2
  • huggingface_hub version: 0.22.2
  • PyArrow version: 16.0.0
  • Pandas version: 2.1.4
  • fsspec version: 2024.3.1

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions