Skip to content

Conversation

@CloseChoice
Copy link
Contributor

When trying to upload nifti datasets lazily I got the error:

from pathlib import Path
from datasets import load_dataset

nifti_dir = Path("<local_path>")

dataset = load_dataset(
        "niftifolder",
        data_dir=str(nifti_dir.absolute()),
        streaming=True,
)

dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type

This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant