Better cast error when generating dataset

[lhoestq]

I want to improve the error message for datasets like https://huggingface.co/datasets/m-a-p/COIG-CQIA

Cc @albertvillanova @severo is this new error ok ? Or should I use a dedicated error class ?

New:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1920, in _prepare_split_single
    writer.write_table(table)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/arrow_writer.py", line 574, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2322, in table_cast
    return cast_table_to_schema(table, schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2276, in cast_table_to_schema
    raise CastError(
datasets.table.CastError: Couldn't cast
instruction: string
other: string
index: string
domain: list<item: string>
  child 0, item: string
output: string
task_type: struct<major: list<item: string>, minor: list<item: string>>
  child 0, major: list<item: string>
      child 0, item: string
  child 1, minor: list<item: string>
      child 0, item: string
task_name_in_eng: string
input: string
to
{'answer_from': Value(dtype='string', id=None), 'instruction': Value(dtype='string', id=None), 'human_verified': Value(dtype='bool', id=None), 'domain': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'output': Value(dtype='string', id=None), 'task_type': {'major': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'minor': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}, 'copyright': Value(dtype='string', id=None), 'input': Value(dtype='string', id=None)}
because column names don't match

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/playground/ttest.py", line 74, in <module>
    load_dataset("m-a-p/COIG-CQIA")
  File "/Users/quentinlhoest/hf/datasets/src/datasets/load.py", line 2529, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 936, in download_and_prepare
    self._download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1031, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1791, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1922, in _prepare_split_single
    raise DatasetGenerationCastError.from_cast_error(
datasets.exceptions.DatasetGenerationCastError: An error occurred while generating the dataset

All the data files must have the same columns, but at some point there are 3 new columns (other, index, task_name_in_eng) and 3 missing columns (answer_from, copyright, human_verified).

This happened while the json dataset builder was generating data using

hf://datasets/m-a-p/COIG-CQIA/coig_pc/coig_pc_core_sample.json (at revision b7b7ecf290f6515036c7c04bd8537228ac2eb474)

Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)

Previously:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1931, in _prepare_split_single
    writer.write_table(table)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/arrow_writer.py", line 574, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2295, in table_cast
    return cast_table_to_schema(table, schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2253, in cast_table_to_schema
    raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nbecause column names don't match")
ValueError: Couldn't cast
task_type: struct<major: list<item: string>, minor: list<item: string>>
  child 0, major: list<item: string>
      child 0, item: string
  child 1, minor: list<item: string>
      child 0, item: string
other: string
instruction: string
task_name_in_eng: string
domain: list<item: string>
  child 0, item: string
index: string
output: string
input: string
to
{'human_verified': Value(dtype='bool', id=None), 'task_type': {'major': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'minor': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}, 'answer_from': Value(dtype='string', id=None), 'copyright': Value(dtype='string', id=None), 'instruction': Value(dtype='string', id=None), 'domain': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'output': Value(dtype='string', id=None), 'input': Value(dtype='string', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/playground/ttest.py", line 74, in <module>
    load_dataset("m-a-p/COIG-CQIA")
  File "/Users/quentinlhoest/hf/datasets/src/datasets/load.py", line 2529, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 949, in download_and_prepare
    self._download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1044, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1804, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1949, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[albertvillanova]

If a user wants to catch one of the errors, they will have to find out where the errors are defined: datasets.builder.CastErrorDuringDatasetGeneration and datasets.table.CastError, respectively.

I would suggest defining the errors in a single place: datasets.exceptions.

Additionally, I would suggest making them inherit from DatasetsError, so that a user can catch any error specifically generated by the datasets library.

[lhoestq]

I created DatatasetGenerationCastError in exceptions.py that inherits from DatasetGenerationError (for backward compatibility) that inherits from DatasetsError.

I also added a help message at the end of the error:

Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)

[albertvillanova]

Thanks.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.004991 / 0.011353 (-0.006361)	0.003362 / 0.011008 (-0.007646)	0.062093 / 0.038508 (0.023585)	0.051533 / 0.023109 (0.028424)	0.247508 / 0.275898 (-0.028390)	0.275593 / 0.323480 (-0.047886)	0.003828 / 0.007986 (-0.004158)	0.002573 / 0.004328 (-0.001755)	0.047727 / 0.004250 (0.043477)	0.037029 / 0.037052 (-0.000023)	0.250359 / 0.258489 (-0.008130)	0.282640 / 0.293841 (-0.011201)	0.027853 / 0.128546 (-0.100693)	0.010247 / 0.075646 (-0.065400)	0.206826 / 0.419271 (-0.212445)	0.035837 / 0.043533 (-0.007695)	0.251795 / 0.255139 (-0.003344)	0.275654 / 0.283200 (-0.007545)	0.017722 / 0.141683 (-0.123960)	1.120287 / 1.452155 (-0.331868)	1.203087 / 1.492716 (-0.289630)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092320 / 0.018006 (0.074314)	0.300079 / 0.000490 (0.299589)	0.000211 / 0.000200 (0.000011)	0.000044 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018193 / 0.037411 (-0.019218)	0.061310 / 0.014526 (0.046784)	0.072433 / 0.176557 (-0.104124)	0.119092 / 0.737135 (-0.618043)	0.074044 / 0.296338 (-0.222294)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.297184 / 0.215209 (0.081975)	2.805197 / 2.077655 (0.727543)	1.521326 / 1.504120 (0.017206)	1.374321 / 1.541195 (-0.166874)	1.388767 / 1.468490 (-0.079723)	0.571865 / 4.584777 (-4.012912)	2.385213 / 3.745712 (-1.360499)	2.726840 / 5.269862 (-2.543021)	1.725352 / 4.565676 (-2.840325)	0.063012 / 0.424275 (-0.361263)	0.004911 / 0.007607 (-0.002697)	0.336430 / 0.226044 (0.110385)	3.390616 / 2.268929 (1.121688)	1.846398 / 55.444624 (-53.598227)	1.576797 / 6.876477 (-5.299680)	1.579445 / 2.142072 (-0.562627)	0.652515 / 4.805227 (-4.152712)	0.118393 / 6.500664 (-6.382271)	0.042155 / 0.075469 (-0.033314)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.942269 / 1.841788 (-0.899518)	11.318258 / 8.074308 (3.243950)	10.299948 / 10.191392 (0.108556)	0.136088 / 0.680424 (-0.544336)	0.013682 / 0.534201 (-0.520519)	0.287549 / 0.579283 (-0.291734)	0.258346 / 0.434364 (-0.176018)	0.337146 / 0.540337 (-0.203191)	0.443922 / 1.386936 (-0.943014)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005302 / 0.011353 (-0.006051)	0.003234 / 0.011008 (-0.007774)	0.049159 / 0.038508 (0.010651)	0.050459 / 0.023109 (0.027350)	0.273718 / 0.275898 (-0.002180)	0.296997 / 0.323480 (-0.026483)	0.003948 / 0.007986 (-0.004038)	0.002590 / 0.004328 (-0.001739)	0.048129 / 0.004250 (0.043879)	0.039369 / 0.037052 (0.002317)	0.276469 / 0.258489 (0.017980)	0.306359 / 0.293841 (0.012519)	0.028864 / 0.128546 (-0.099682)	0.010253 / 0.075646 (-0.065394)	0.058264 / 0.419271 (-0.361008)	0.032451 / 0.043533 (-0.011082)	0.277336 / 0.255139 (0.022197)	0.296137 / 0.283200 (0.012937)	0.018094 / 0.141683 (-0.123589)	1.119539 / 1.452155 (-0.332615)	1.163116 / 1.492716 (-0.329600)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092578 / 0.018006 (0.074572)	0.300756 / 0.000490 (0.300267)	0.000222 / 0.000200 (0.000022)	0.000044 / 0.000054 (-0.000010)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022333 / 0.037411 (-0.015078)	0.076632 / 0.014526 (0.062107)	0.087829 / 0.176557 (-0.088727)	0.127686 / 0.737135 (-0.609449)	0.091314 / 0.296338 (-0.205024)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.297499 / 0.215209 (0.082290)	2.889775 / 2.077655 (0.812120)	1.598976 / 1.504120 (0.094856)	1.478805 / 1.541195 (-0.062389)	1.481818 / 1.468490 (0.013328)	0.557972 / 4.584777 (-4.026804)	2.453248 / 3.745712 (-1.292464)	2.771823 / 5.269862 (-2.498039)	1.721527 / 4.565676 (-2.844150)	0.062786 / 0.424275 (-0.361489)	0.005298 / 0.007607 (-0.002309)	0.346660 / 0.226044 (0.120615)	3.412262 / 2.268929 (1.143334)	1.940240 / 55.444624 (-53.504384)	1.654015 / 6.876477 (-5.222461)	1.652039 / 2.142072 (-0.490034)	0.636870 / 4.805227 (-4.168357)	0.116213 / 6.500664 (-6.384451)	0.040937 / 0.075469 (-0.034532)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.001605 / 1.841788 (-0.840183)	11.986592 / 8.074308 (3.912284)	10.231288 / 10.191392 (0.039896)	0.130242 / 0.680424 (-0.550182)	0.015764 / 0.534201 (-0.518437)	0.289257 / 0.579283 (-0.290026)	0.275996 / 0.434364 (-0.158368)	0.323089 / 0.540337 (-0.217248)	0.556383 / 1.386936 (-0.830553)