Raise FileNotFoundError when passing data_files that don't exist #6155

lhoestq · 2023-08-17T09:49:48Z

e.g. when running load_dataset("parquet", data_files="doesnt_exist.parquet")

HuggingFaceDocBuilderDev · 2023-08-17T09:57:19Z

The documentation is not available anymore as the PR was closed or merged.

github-actions · 2023-08-17T14:10:18Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009288 / 0.011353 (-0.002065)	0.005950 / 0.011008 (-0.005058)	0.122376 / 0.038508 (0.083868)	0.093177 / 0.023109 (0.070068)	0.448517 / 0.275898 (0.172619)	0.474999 / 0.323480 (0.151520)	0.005133 / 0.007986 (-0.002853)	0.005123 / 0.004328 (0.000795)	0.085479 / 0.004250 (0.081229)	0.065613 / 0.037052 (0.028561)	0.451179 / 0.258489 (0.192690)	0.516876 / 0.293841 (0.223036)	0.047536 / 0.128546 (-0.081010)	0.013894 / 0.075646 (-0.061752)	0.382149 / 0.419271 (-0.037122)	0.067380 / 0.043533 (0.023848)	0.419282 / 0.255139 (0.164143)	0.482042 / 0.283200 (0.198842)	0.041230 / 0.141683 (-0.100452)	1.818127 / 1.452155 (0.365972)	1.938123 / 1.492716 (0.445406)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.271824 / 0.018006 (0.253817)	0.604933 / 0.000490 (0.604443)	0.004953 / 0.000200 (0.004753)	0.000173 / 0.000054 (0.000119)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.036682 / 0.037411 (-0.000729)	0.095604 / 0.014526 (0.081078)	0.116862 / 0.176557 (-0.059695)	0.191335 / 0.737135 (-0.545800)	0.116620 / 0.296338 (-0.179718)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.620735 / 0.215209 (0.405526)	6.157119 / 2.077655 (4.079465)	2.848548 / 1.504120 (1.344428)	2.493731 / 1.541195 (0.952536)	2.505801 / 1.468490 (1.037311)	0.837315 / 4.584777 (-3.747462)	5.360653 / 3.745712 (1.614941)	4.908863 / 5.269862 (-0.360999)	3.184672 / 4.565676 (-1.381004)	0.105687 / 0.424275 (-0.318588)	0.011350 / 0.007607 (0.003743)	0.745729 / 0.226044 (0.519684)	7.431584 / 2.268929 (5.162655)	3.644670 / 55.444624 (-51.799954)	2.910159 / 6.876477 (-3.966317)	3.257137 / 2.142072 (1.115065)	1.041377 / 4.805227 (-3.763851)	0.213289 / 6.500664 (-6.287375)	0.089208 / 0.075469 (0.013739)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.727274 / 1.841788 (-0.114513)	25.448436 / 8.074308 (17.374128)	23.016108 / 10.191392 (12.824716)	0.219454 / 0.680424 (-0.460970)	0.028531 / 0.534201 (-0.505670)	0.500231 / 0.579283 (-0.079052)	0.614631 / 0.434364 (0.180267)	0.557926 / 0.540337 (0.017588)	0.786261 / 1.386936 (-0.600675)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008608 / 0.011353 (-0.002745)	0.006185 / 0.011008 (-0.004823)	0.089258 / 0.038508 (0.050750)	0.090109 / 0.023109 (0.067000)	0.522200 / 0.275898 (0.246302)	0.559218 / 0.323480 (0.235738)	0.008983 / 0.007986 (0.000997)	0.004488 / 0.004328 (0.000159)	0.083658 / 0.004250 (0.079408)	0.064962 / 0.037052 (0.027909)	0.519477 / 0.258489 (0.260988)	0.573842 / 0.293841 (0.280001)	0.053984 / 0.128546 (-0.074562)	0.014665 / 0.075646 (-0.060982)	0.089438 / 0.419271 (-0.329834)	0.065756 / 0.043533 (0.022223)	0.525131 / 0.255139 (0.269992)	0.568934 / 0.283200 (0.285734)	0.037308 / 0.141683 (-0.104375)	1.928790 / 1.452155 (0.476635)	2.027926 / 1.492716 (0.535209)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.309595 / 0.018006 (0.291588)	0.615675 / 0.000490 (0.615186)	0.004869 / 0.000200 (0.004669)	0.000116 / 0.000054 (0.000061)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.033306 / 0.037411 (-0.004105)	0.104429 / 0.014526 (0.089904)	0.116989 / 0.176557 (-0.059568)	0.183638 / 0.737135 (-0.553497)	0.132624 / 0.296338 (-0.163714)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.644511 / 0.215209 (0.429302)	6.425544 / 2.077655 (4.347889)	3.079071 / 1.504120 (1.574951)	2.720963 / 1.541195 (1.179769)	2.835607 / 1.468490 (1.367117)	0.863561 / 4.584777 (-3.721216)	5.333462 / 3.745712 (1.587750)	4.843183 / 5.269862 (-0.426678)	3.106858 / 4.565676 (-1.458819)	0.106790 / 0.424275 (-0.317485)	0.008829 / 0.007607 (0.001222)	0.759003 / 0.226044 (0.532958)	7.771247 / 2.268929 (5.502318)	3.896844 / 55.444624 (-51.547780)	3.246671 / 6.876477 (-3.629806)	3.486167 / 2.142072 (1.344094)	1.071290 / 4.805227 (-3.733937)	0.217972 / 6.500664 (-6.282692)	0.089848 / 0.075469 (0.014379)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.816048 / 1.841788 (-0.025739)	25.625084 / 8.074308 (17.550776)	24.490882 / 10.191392 (14.299490)	0.242356 / 0.680424 (-0.438067)	0.027886 / 0.534201 (-0.506315)	0.496997 / 0.579283 (-0.082286)	0.613815 / 0.434364 (0.179451)	0.607132 / 0.540337 (0.066795)	0.833051 / 1.386936 (-0.553885)

github-actions · 2023-08-17T14:36:22Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.011580 / 0.011353 (0.000227)	0.004199 / 0.011008 (-0.006809)	0.084055 / 0.038508 (0.045547)	0.096824 / 0.023109 (0.073715)	0.308755 / 0.275898 (0.032857)	0.341717 / 0.323480 (0.018237)	0.006018 / 0.007986 (-0.001968)	0.003597 / 0.004328 (-0.000731)	0.064953 / 0.004250 (0.060702)	0.059577 / 0.037052 (0.022525)	0.316292 / 0.258489 (0.057803)	0.358991 / 0.293841 (0.065150)	0.033925 / 0.128546 (-0.094621)	0.008828 / 0.075646 (-0.066818)	0.288673 / 0.419271 (-0.130599)	0.055494 / 0.043533 (0.011961)	0.311181 / 0.255139 (0.056042)	0.345220 / 0.283200 (0.062021)	0.024033 / 0.141683 (-0.117649)	1.504709 / 1.452155 (0.052554)	1.587920 / 1.492716 (0.095204)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.301099 / 0.018006 (0.283093)	0.594497 / 0.000490 (0.594007)	0.006244 / 0.000200 (0.006044)	0.000228 / 0.000054 (0.000174)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.027663 / 0.037411 (-0.009748)	0.081767 / 0.014526 (0.067241)	0.097342 / 0.176557 (-0.079215)	0.153200 / 0.737135 (-0.583935)	0.097474 / 0.296338 (-0.198864)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.405929 / 0.215209 (0.190719)	4.045398 / 2.077655 (1.967743)	2.044669 / 1.504120 (0.540549)	1.872889 / 1.541195 (0.331694)	1.911901 / 1.468490 (0.443411)	0.480939 / 4.584777 (-4.103838)	3.652833 / 3.745712 (-0.092879)	3.281659 / 5.269862 (-1.988202)	2.038023 / 4.565676 (-2.527654)	0.056501 / 0.424275 (-0.367775)	0.007571 / 0.007607 (-0.000036)	0.481053 / 0.226044 (0.255009)	4.802048 / 2.268929 (2.533119)	2.560479 / 55.444624 (-52.884145)	2.164852 / 6.876477 (-4.711625)	2.374595 / 2.142072 (0.232523)	0.576309 / 4.805227 (-4.228918)	0.134831 / 6.500664 (-6.365833)	0.060649 / 0.075469 (-0.014820)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.254210 / 1.841788 (-0.587578)	19.826143 / 8.074308 (11.751835)	14.446391 / 10.191392 (4.254999)	0.165707 / 0.680424 (-0.514717)	0.018221 / 0.534201 (-0.515980)	0.395996 / 0.579283 (-0.183287)	0.424567 / 0.434364 (-0.009796)	0.459836 / 0.540337 (-0.080501)	0.635969 / 1.386936 (-0.750967)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006696 / 0.011353 (-0.004657)	0.004131 / 0.011008 (-0.006877)	0.064587 / 0.038508 (0.026079)	0.079189 / 0.023109 (0.056080)	0.359977 / 0.275898 (0.084079)	0.389331 / 0.323480 (0.065851)	0.005502 / 0.007986 (-0.002483)	0.003492 / 0.004328 (-0.000837)	0.064967 / 0.004250 (0.060716)	0.055953 / 0.037052 (0.018901)	0.363997 / 0.258489 (0.105508)	0.398405 / 0.293841 (0.104564)	0.031292 / 0.128546 (-0.097254)	0.008693 / 0.075646 (-0.066953)	0.070451 / 0.419271 (-0.348820)	0.048965 / 0.043533 (0.005432)	0.358288 / 0.255139 (0.103149)	0.379136 / 0.283200 (0.095936)	0.024364 / 0.141683 (-0.117319)	1.478998 / 1.452155 (0.026843)	1.547282 / 1.492716 (0.054566)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.328188 / 0.018006 (0.310182)	0.525968 / 0.000490 (0.525478)	0.003782 / 0.000200 (0.003582)	0.000089 / 0.000054 (0.000034)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032528 / 0.037411 (-0.004883)	0.087685 / 0.014526 (0.073159)	0.100684 / 0.176557 (-0.075872)	0.155944 / 0.737135 (-0.581192)	0.101949 / 0.296338 (-0.194389)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.418591 / 0.215209 (0.203382)	4.199235 / 2.077655 (2.121580)	2.183880 / 1.504120 (0.679760)	2.024502 / 1.541195 (0.483307)	2.017435 / 1.468490 (0.548945)	0.488881 / 4.584777 (-4.095896)	3.635002 / 3.745712 (-0.110710)	3.359992 / 5.269862 (-1.909870)	2.089686 / 4.565676 (-2.475991)	0.057813 / 0.424275 (-0.366462)	0.007349 / 0.007607 (-0.000258)	0.490719 / 0.226044 (0.264674)	4.859950 / 2.268929 (2.591022)	2.616711 / 55.444624 (-52.827914)	2.238671 / 6.876477 (-4.637806)	2.442262 / 2.142072 (0.300190)	0.598368 / 4.805227 (-4.206859)	0.135281 / 6.500664 (-6.365383)	0.063072 / 0.075469 (-0.012397)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.356396 / 1.841788 (-0.485392)	20.075123 / 8.074308 (12.000815)	14.191317 / 10.191392 (3.999925)	0.167691 / 0.680424 (-0.512732)	0.018290 / 0.534201 (-0.515911)	0.392881 / 0.579283 (-0.186402)	0.413665 / 0.434364 (-0.020699)	0.480766 / 0.540337 (-0.059571)	0.655625 / 1.386936 (-0.731311)

mariosasko

Nice!

mariosasko · 2023-08-17T21:03:30Z

src/datasets/data_files.py

                )
            except FileNotFoundError:
-                pass
+                if "*" not in pattern and "[" not in pattern and "?" not in pattern:


The glob module has the has_magic method (public, but not documented IIRC; e.g., fsspec uses it) to check this:

Suggested change

if "*" not in pattern and "[" not in pattern and "?" not in pattern:

if has_magic(pattern):

But feel free to decide which one you prefer.

github-actions · 2023-08-18T10:34:45Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007834 / 0.011353 (-0.003519)	0.004744 / 0.011008 (-0.006264)	0.102061 / 0.038508 (0.063553)	0.089246 / 0.023109 (0.066137)	0.399936 / 0.275898 (0.124038)	0.436974 / 0.323480 (0.113494)	0.004791 / 0.007986 (-0.003195)	0.005976 / 0.004328 (0.001647)	0.079336 / 0.004250 (0.075086)	0.065947 / 0.037052 (0.028894)	0.403747 / 0.258489 (0.145258)	0.460249 / 0.293841 (0.166408)	0.038065 / 0.128546 (-0.090482)	0.010179 / 0.075646 (-0.065467)	0.403620 / 0.419271 (-0.015652)	0.066439 / 0.043533 (0.022906)	0.412123 / 0.255139 (0.156984)	0.452121 / 0.283200 (0.168921)	0.033533 / 0.141683 (-0.108150)	1.858650 / 1.452155 (0.406495)	1.916248 / 1.492716 (0.423532)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.237180 / 0.018006 (0.219174)	0.526844 / 0.000490 (0.526354)	0.004220 / 0.000200 (0.004020)	0.000123 / 0.000054 (0.000069)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.033860 / 0.037411 (-0.003552)	0.105054 / 0.014526 (0.090528)	0.116494 / 0.176557 (-0.060063)	0.185990 / 0.737135 (-0.551145)	0.119072 / 0.296338 (-0.177266)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.488549 / 0.215209 (0.273340)	4.884950 / 2.077655 (2.807295)	2.521819 / 1.504120 (1.017699)	2.329382 / 1.541195 (0.788188)	2.413710 / 1.468490 (0.945220)	0.568325 / 4.584777 (-4.016452)	4.243505 / 3.745712 (0.497793)	3.785983 / 5.269862 (-1.483879)	2.387146 / 4.565676 (-2.178531)	0.067176 / 0.424275 (-0.357099)	0.009145 / 0.007607 (0.001538)	0.571482 / 0.226044 (0.345437)	5.688822 / 2.268929 (3.419894)	3.067346 / 55.444624 (-52.377278)	2.688723 / 6.876477 (-4.187754)	2.883785 / 2.142072 (0.741713)	0.679326 / 4.805227 (-4.125901)	0.156018 / 6.500664 (-6.344646)	0.070947 / 0.075469 (-0.004522)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.556611 / 1.841788 (-0.285177)	23.545074 / 8.074308 (15.470766)	17.125108 / 10.191392 (6.933716)	0.180180 / 0.680424 (-0.500244)	0.021420 / 0.534201 (-0.512781)	0.466888 / 0.579283 (-0.112395)	0.485746 / 0.434364 (0.051383)	0.606181 / 0.540337 (0.065843)	0.776691 / 1.386936 (-0.610245)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007820 / 0.011353 (-0.003533)	0.004531 / 0.011008 (-0.006478)	0.076142 / 0.038508 (0.037634)	0.086367 / 0.023109 (0.063258)	0.456150 / 0.275898 (0.180252)	0.499712 / 0.323480 (0.176232)	0.006545 / 0.007986 (-0.001441)	0.003760 / 0.004328 (-0.000568)	0.076400 / 0.004250 (0.072150)	0.069689 / 0.037052 (0.032637)	0.459732 / 0.258489 (0.201243)	0.504217 / 0.293841 (0.210376)	0.037838 / 0.128546 (-0.090709)	0.009804 / 0.075646 (-0.065843)	0.084654 / 0.419271 (-0.334617)	0.060301 / 0.043533 (0.016768)	0.452984 / 0.255139 (0.197845)	0.479956 / 0.283200 (0.196757)	0.029674 / 0.141683 (-0.112009)	1.814059 / 1.452155 (0.361904)	1.878886 / 1.492716 (0.386170)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.326174 / 0.018006 (0.308168)	0.539722 / 0.000490 (0.539232)	0.025637 / 0.000200 (0.025437)	0.000209 / 0.000054 (0.000154)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.036328 / 0.037411 (-0.001084)	0.106369 / 0.014526 (0.091843)	0.118598 / 0.176557 (-0.057958)	0.182760 / 0.737135 (-0.554376)	0.120013 / 0.296338 (-0.176326)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.507328 / 0.215209 (0.292119)	5.092689 / 2.077655 (3.015034)	2.962334 / 1.504120 (1.458214)	2.507699 / 1.541195 (0.966504)	2.612245 / 1.468490 (1.143755)	0.568625 / 4.584777 (-4.016152)	4.296484 / 3.745712 (0.550772)	4.037788 / 5.269862 (-1.232073)	2.579826 / 4.565676 (-1.985850)	0.068558 / 0.424275 (-0.355717)	0.008916 / 0.007607 (0.001309)	0.601054 / 0.226044 (0.375010)	6.016061 / 2.268929 (3.747133)	3.311880 / 55.444624 (-52.132744)	2.912926 / 6.876477 (-3.963551)	3.101465 / 2.142072 (0.959393)	0.686848 / 4.805227 (-4.118380)	0.160243 / 6.500664 (-6.340421)	0.074084 / 0.075469 (-0.001385)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.754343 / 1.841788 (-0.087444)	24.215302 / 8.074308 (16.140994)	17.211007 / 10.191392 (7.019615)	0.188370 / 0.680424 (-0.492054)	0.028157 / 0.534201 (-0.506044)	0.490879 / 0.579283 (-0.088404)	0.501508 / 0.434364 (0.067144)	0.599719 / 0.540337 (0.059381)	0.852438 / 1.386936 (-0.534498)

github-actions · 2023-08-18T13:45:57Z

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009736 / 0.011353 (-0.001617)	0.004761 / 0.011008 (-0.006247)	0.100069 / 0.038508 (0.061561)	0.077944 / 0.023109 (0.054835)	0.419944 / 0.275898 (0.144046)	0.459803 / 0.323480 (0.136323)	0.006296 / 0.007986 (-0.001689)	0.005375 / 0.004328 (0.001047)	0.089457 / 0.004250 (0.085207)	0.060585 / 0.037052 (0.023532)	0.437988 / 0.258489 (0.179499)	0.482676 / 0.293841 (0.188835)	0.049126 / 0.128546 (-0.079420)	0.015043 / 0.075646 (-0.060603)	0.342500 / 0.419271 (-0.076771)	0.067088 / 0.043533 (0.023555)	0.418364 / 0.255139 (0.163225)	0.458259 / 0.283200 (0.175059)	0.034091 / 0.141683 (-0.107592)	1.721589 / 1.452155 (0.269434)	1.823142 / 1.492716 (0.330426)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.212110 / 0.018006 (0.194103)	0.530957 / 0.000490 (0.530467)	0.003581 / 0.000200 (0.003382)	0.000112 / 0.000054 (0.000058)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030202 / 0.037411 (-0.007210)	0.100552 / 0.014526 (0.086026)	0.108150 / 0.176557 (-0.068407)	0.173203 / 0.737135 (-0.563932)	0.108624 / 0.296338 (-0.187715)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.577340 / 0.215209 (0.362131)	5.794197 / 2.077655 (3.716543)	2.396285 / 1.504120 (0.892165)	2.151972 / 1.541195 (0.610777)	2.109485 / 1.468490 (0.640995)	0.873906 / 4.584777 (-3.710871)	5.083302 / 3.745712 (1.337589)	4.600756 / 5.269862 (-0.669105)	2.891731 / 4.565676 (-1.673945)	0.096293 / 0.424275 (-0.327982)	0.008651 / 0.007607 (0.001044)	0.719095 / 0.226044 (0.493051)	7.193225 / 2.268929 (4.924297)	3.220145 / 55.444624 (-52.224479)	2.496715 / 6.876477 (-4.379762)	2.672972 / 2.142072 (0.530900)	1.031656 / 4.805227 (-3.773571)	0.207854 / 6.500664 (-6.292810)	0.074507 / 0.075469 (-0.000962)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.552821 / 1.841788 (-0.288967)	22.573015 / 8.074308 (14.498707)	21.074321 / 10.191392 (10.882929)	0.231911 / 0.680424 (-0.448513)	0.027761 / 0.534201 (-0.506440)	0.474644 / 0.579283 (-0.104639)	0.563780 / 0.434364 (0.129416)	0.527593 / 0.540337 (-0.012745)	0.732299 / 1.386936 (-0.654637)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.008675 / 0.011353 (-0.002678)	0.005268 / 0.011008 (-0.005741)	0.079078 / 0.038508 (0.040570)	0.073505 / 0.023109 (0.050395)	0.453982 / 0.275898 (0.178083)	0.487839 / 0.323480 (0.164359)	0.005950 / 0.007986 (-0.002035)	0.003848 / 0.004328 (-0.000481)	0.076004 / 0.004250 (0.071754)	0.058410 / 0.037052 (0.021358)	0.460099 / 0.258489 (0.201610)	0.514860 / 0.293841 (0.221019)	0.048843 / 0.128546 (-0.079703)	0.014275 / 0.075646 (-0.061371)	0.090243 / 0.419271 (-0.329029)	0.060092 / 0.043533 (0.016559)	0.455669 / 0.255139 (0.200530)	0.484738 / 0.283200 (0.201538)	0.033012 / 0.141683 (-0.108671)	1.738854 / 1.452155 (0.286699)	1.852552 / 1.492716 (0.359835)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.245453 / 0.018006 (0.227447)	0.519929 / 0.000490 (0.519439)	0.007262 / 0.000200 (0.007062)	0.000108 / 0.000054 (0.000054)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031446 / 0.037411 (-0.005965)	0.094236 / 0.014526 (0.079710)	0.114457 / 0.176557 (-0.062100)	0.167448 / 0.737135 (-0.569687)	0.108791 / 0.296338 (-0.187548)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.603331 / 0.215209 (0.388122)	6.051556 / 2.077655 (3.973902)	2.797110 / 1.504120 (1.292990)	2.500517 / 1.541195 (0.959322)	2.531421 / 1.468490 (1.062931)	0.852075 / 4.584777 (-3.732702)	5.034140 / 3.745712 (1.288427)	4.576573 / 5.269862 (-0.693289)	2.973541 / 4.565676 (-1.592135)	0.101303 / 0.424275 (-0.322972)	0.008467 / 0.007607 (0.000860)	0.707143 / 0.226044 (0.481098)	7.262803 / 2.268929 (4.993874)	3.548841 / 55.444624 (-51.895783)	2.895975 / 6.876477 (-3.980502)	3.063521 / 2.142072 (0.921449)	1.014961 / 4.805227 (-3.790266)	0.208527 / 6.500664 (-6.292137)	0.074939 / 0.075469 (-0.000530)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.670708 / 1.841788 (-0.171080)	22.685227 / 8.074308 (14.610919)	20.393017 / 10.191392 (10.201625)	0.239303 / 0.680424 (-0.441121)	0.027742 / 0.534201 (-0.506459)	0.467230 / 0.579283 (-0.112053)	0.564169 / 0.434364 (0.129805)	0.554859 / 0.540337 (0.014522)	0.767471 / 1.386936 (-0.619465)

* raise filenotfound * only raise if no magic char in pattern * add test * use has_magic

raise filenotfound

b9a2b30

only raise if no magic char in pattern

0adfa9a

add test

a46ca9c

lhoestq marked this pull request as ready for review August 17, 2023 20:53

lhoestq requested a review from mariosasko August 17, 2023 20:53

mariosasko approved these changes Aug 17, 2023

View reviewed changes

use has_magic

d84cd1d

lhoestq merged commit 72a5735 into main Aug 18, 2023

lhoestq deleted the raise-filenotfound branch August 18, 2023 13:35

mariosasko mentioned this pull request Aug 29, 2023

Load dataset with non-existent file #6183

Closed

albertvillanova pushed a commit that referenced this pull request Oct 24, 2023

Raise FileNotFoundError when passing data_files that don't exist (#6155)

f34dba9

* raise filenotfound * only raise if no magic char in pattern * add test * use has_magic

	if "*" not in pattern and "[" not in pattern and "?" not in pattern:
	if has_magic(pattern):

Raise FileNotFoundError when passing data_files that don't exist #6155

Raise FileNotFoundError when passing data_files that don't exist #6155

Uh oh!

Conversation

lhoestq commented Aug 17, 2023

Uh oh!

HuggingFaceDocBuilderDev commented Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 17, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Uh oh!

github-actions bot commented Aug 17, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Uh oh!

mariosasko left a comment

Choose a reason for hiding this comment

Uh oh!

mariosasko Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 18, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Uh oh!

github-actions bot commented Aug 18, 2023

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Benchmark: benchmark_array_xd.json

Benchmark: benchmark_getitem_100B.json

Benchmark: benchmark_indices_mapping.json

Benchmark: benchmark_iterating.json

Benchmark: benchmark_map_filter.json

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HuggingFaceDocBuilderDev commented Aug 17, 2023 •

edited

Loading

mariosasko Aug 17, 2023 •

edited

Loading