Use auth to get parquet export

[lhoestq]

added token to the _datasets_server functions

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005076 / 0.011353 (-0.006277)	0.003510 / 0.011008 (-0.007499)	0.062939 / 0.038508 (0.024431)	0.049191 / 0.023109 (0.026082)	0.259088 / 0.275898 (-0.016810)	0.273523 / 0.323480 (-0.049957)	0.003902 / 0.007986 (-0.004083)	0.002699 / 0.004328 (-0.001630)	0.049077 / 0.004250 (0.044827)	0.037174 / 0.037052 (0.000121)	0.256467 / 0.258489 (-0.002022)	0.291235 / 0.293841 (-0.002606)	0.028119 / 0.128546 (-0.100427)	0.010404 / 0.075646 (-0.065243)	0.205825 / 0.419271 (-0.213446)	0.035741 / 0.043533 (-0.007792)	0.253219 / 0.255139 (-0.001920)	0.274986 / 0.283200 (-0.008214)	0.018379 / 0.141683 (-0.123304)	1.131139 / 1.452155 (-0.321016)	1.175875 / 1.492716 (-0.316841)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.090717 / 0.018006 (0.072710)	0.299285 / 0.000490 (0.298796)	0.000217 / 0.000200 (0.000017)	0.000050 / 0.000054 (-0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018678 / 0.037411 (-0.018733)	0.060558 / 0.014526 (0.046032)	0.073828 / 0.176557 (-0.102728)	0.119302 / 0.737135 (-0.617833)	0.075261 / 0.296338 (-0.221078)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.277018 / 0.215209 (0.061809)	2.713255 / 2.077655 (0.635601)	1.427512 / 1.504120 (-0.076608)	1.311374 / 1.541195 (-0.229821)	1.348756 / 1.468490 (-0.119734)	0.561777 / 4.584777 (-4.023000)	2.393578 / 3.745712 (-1.352134)	2.798109 / 5.269862 (-2.471753)	1.754808 / 4.565676 (-2.810869)	0.062302 / 0.424275 (-0.361973)	0.004948 / 0.007607 (-0.002659)	0.328468 / 0.226044 (0.102423)	3.246558 / 2.268929 (0.977629)	1.786816 / 55.444624 (-53.657808)	1.482937 / 6.876477 (-5.393540)	1.516109 / 2.142072 (-0.625963)	0.634457 / 4.805227 (-4.170770)	0.116505 / 6.500664 (-6.384159)	0.042162 / 0.075469 (-0.033308)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.935312 / 1.841788 (-0.906476)	11.540599 / 8.074308 (3.466291)	10.512593 / 10.191392 (0.321201)	0.129638 / 0.680424 (-0.550786)	0.013994 / 0.534201 (-0.520207)	0.291490 / 0.579283 (-0.287793)	0.263641 / 0.434364 (-0.170722)	0.328718 / 0.540337 (-0.211619)	0.437598 / 1.386936 (-0.949338)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005192 / 0.011353 (-0.006161)	0.003454 / 0.011008 (-0.007554)	0.049448 / 0.038508 (0.010940)	0.050968 / 0.023109 (0.027859)	0.273702 / 0.275898 (-0.002196)	0.296934 / 0.323480 (-0.026545)	0.004066 / 0.007986 (-0.003920)	0.002611 / 0.004328 (-0.001718)	0.048284 / 0.004250 (0.044034)	0.041399 / 0.037052 (0.004346)	0.283000 / 0.258489 (0.024511)	0.302553 / 0.293841 (0.008712)	0.029086 / 0.128546 (-0.099460)	0.010510 / 0.075646 (-0.065137)	0.058097 / 0.419271 (-0.361175)	0.032992 / 0.043533 (-0.010541)	0.271752 / 0.255139 (0.016613)	0.293535 / 0.283200 (0.010335)	0.016958 / 0.141683 (-0.124725)	1.130126 / 1.452155 (-0.322028)	1.187228 / 1.492716 (-0.305488)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092321 / 0.018006 (0.074315)	0.302599 / 0.000490 (0.302109)	0.000215 / 0.000200 (0.000015)	0.000051 / 0.000054 (-0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.021837 / 0.037411 (-0.015574)	0.071148 / 0.014526 (0.056622)	0.082448 / 0.176557 (-0.094108)	0.128083 / 0.737135 (-0.609053)	0.090864 / 0.296338 (-0.205474)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.296248 / 0.215209 (0.081039)	2.881130 / 2.077655 (0.803476)	1.580360 / 1.504120 (0.076240)	1.454642 / 1.541195 (-0.086553)	1.461453 / 1.468490 (-0.007037)	0.567500 / 4.584777 (-4.017277)	2.493708 / 3.745712 (-1.252004)	2.756623 / 5.269862 (-2.513239)	1.771319 / 4.565676 (-2.794358)	0.062287 / 0.424275 (-0.361988)	0.004917 / 0.007607 (-0.002691)	0.348034 / 0.226044 (0.121990)	3.426938 / 2.268929 (1.158010)	1.954190 / 55.444624 (-53.490435)	1.660870 / 6.876477 (-5.215607)	1.675118 / 2.142072 (-0.466955)	0.636843 / 4.805227 (-4.168384)	0.115028 / 6.500664 (-6.385636)	0.040702 / 0.075469 (-0.034767)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.988076 / 1.841788 (-0.853711)	11.890867 / 8.074308 (3.816559)	10.621169 / 10.191392 (0.429777)	0.131568 / 0.680424 (-0.548856)	0.014994 / 0.534201 (-0.519207)	0.288900 / 0.579283 (-0.290384)	0.272092 / 0.434364 (-0.162272)	0.329397 / 0.540337 (-0.210940)	0.569337 / 1.386936 (-0.817599)