Skip to content

Ignore empty batch when writing #2698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lhoestq merged 2 commits into from
Jul 26, 2021
Merged

Ignore empty batch when writing #2698

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a24ebb
Ignore empty batch when writing.
pcuenca Jul 21, 2021
e7feaa2
minor
lhoestq Jul 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/datasets/arrow_writer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,10 +372,14 @@ def write_batch(
writer_batch_size: Optional[int] = None,
):
"""Write a batch of Example to file.
Ignores the batch if it appears to be empty,
preventing a potential schema update of unknown types.

Args:
example: the Example to add.
batch_examples: the batch of examples to add.
"""
if batch_examples and len(next(iter(batch_examples.values()))) == 0:
return
schema = None if self.pa_writer is None and self.update_features else self._schema
try_schema = self._schema if self.pa_writer is None and self.update_features else None
typed_sequence_examples = {}
Expand Down