Use HF_HUB_OFFLINE instead of HF_DATASETS_OFFLINE

[Wauplin]

To use datasets offline, one can use the HF_DATASETS_OFFLINE environment variable. This PR makes HF_HUB_OFFLINE the recommended environment variable for offline training. Goal is to be more consistent with the rest of HF ecosystem and have a single config value to set.

The changes are backward-compatible meaning that:

• HF_DATASETS_OFFLINE environment is still taken into account, though not documented
• datasets.config.HF_DATASETS_OFFLINE still exists, though it is not used anymore (in favor of datasets.config.HF_HUB_OFFLINE)

Note: it might break things in downstream libraries if they were monkeypatching datasets.config.HF_DATASETS_OFFLINE in their CI tests (for instance). Not much of a problem IMO.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

Nice ! Feel free to fix the code style :)

btw we're doing a release today, but no need to include this PR in it IMO, we can merge this one tomorrow

Could you also update the mentions of HF_DATASETS_OFFLINE in the docs ?

[Wauplin]

Oops, sorry for the style issue. Fixed in a4e2b28.

Regarding docs, I can't find mentions of HF_DATASETS_OFFLINE anywhere else in datasets/hub-docs. Once this is merged and released, I'm planning to update some transformers docs that briefly mention it.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005173 / 0.011353 (-0.006180)	0.003485 / 0.011008 (-0.007524)	0.063867 / 0.038508 (0.025359)	0.031338 / 0.023109 (0.008229)	0.242093 / 0.275898 (-0.033805)	0.266606 / 0.323480 (-0.056874)	0.003069 / 0.007986 (-0.004916)	0.003307 / 0.004328 (-0.001022)	0.051059 / 0.004250 (0.046808)	0.044396 / 0.037052 (0.007344)	0.254896 / 0.258489 (-0.003593)	0.282835 / 0.293841 (-0.011006)	0.027548 / 0.128546 (-0.100998)	0.010520 / 0.075646 (-0.065126)	0.201701 / 0.419271 (-0.217570)	0.035613 / 0.043533 (-0.007920)	0.240955 / 0.255139 (-0.014184)	0.271902 / 0.283200 (-0.011298)	0.019826 / 0.141683 (-0.121857)	1.116994 / 1.452155 (-0.335161)	1.162886 / 1.492716 (-0.329831)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.093683 / 0.018006 (0.075677)	0.297970 / 0.000490 (0.297480)	0.000211 / 0.000200 (0.000011)	0.000043 / 0.000054 (-0.000011)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.018952 / 0.037411 (-0.018459)	0.062710 / 0.014526 (0.048184)	0.073641 / 0.176557 (-0.102916)	0.121200 / 0.737135 (-0.615935)	0.075723 / 0.296338 (-0.220616)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.286056 / 0.215209 (0.070847)	2.811424 / 2.077655 (0.733770)	1.448045 / 1.504120 (-0.056075)	1.338309 / 1.541195 (-0.202885)	1.328371 / 1.468490 (-0.140119)	0.557282 / 4.584777 (-4.027495)	2.362235 / 3.745712 (-1.383477)	2.732108 / 5.269862 (-2.537754)	1.730911 / 4.565676 (-2.834765)	0.061689 / 0.424275 (-0.362586)	0.004947 / 0.007607 (-0.002660)	0.346700 / 0.226044 (0.120656)	3.355989 / 2.268929 (1.087060)	1.828078 / 55.444624 (-53.616546)	1.511531 / 6.876477 (-5.364946)	1.535897 / 2.142072 (-0.606175)	0.630276 / 4.805227 (-4.174951)	0.115808 / 6.500664 (-6.384857)	0.042199 / 0.075469 (-0.033270)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	0.969203 / 1.841788 (-0.872584)	11.282997 / 8.074308 (3.208689)	9.538914 / 10.191392 (-0.652478)	0.140072 / 0.680424 (-0.540352)	0.014021 / 0.534201 (-0.520180)	0.283784 / 0.579283 (-0.295499)	0.255973 / 0.434364 (-0.178391)	0.320284 / 0.540337 (-0.220053)	0.412689 / 1.386936 (-0.974247)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.005201 / 0.011353 (-0.006152)	0.003312 / 0.011008 (-0.007697)	0.050044 / 0.038508 (0.011536)	0.033610 / 0.023109 (0.010501)	0.266429 / 0.275898 (-0.009469)	0.287782 / 0.323480 (-0.035698)	0.004316 / 0.007986 (-0.003670)	0.002696 / 0.004328 (-0.001633)	0.049667 / 0.004250 (0.045417)	0.040244 / 0.037052 (0.003192)	0.278870 / 0.258489 (0.020381)	0.311415 / 0.293841 (0.017574)	0.029150 / 0.128546 (-0.099396)	0.010046 / 0.075646 (-0.065600)	0.058527 / 0.419271 (-0.360744)	0.032871 / 0.043533 (-0.010662)	0.266582 / 0.255139 (0.011443)	0.286157 / 0.283200 (0.002957)	0.017197 / 0.141683 (-0.124486)	1.120944 / 1.452155 (-0.331211)	1.161111 / 1.492716 (-0.331606)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.092679 / 0.018006 (0.074672)	0.299195 / 0.000490 (0.298705)	0.000204 / 0.000200 (0.000004)	0.000048 / 0.000054 (-0.000007)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022212 / 0.037411 (-0.015199)	0.076734 / 0.014526 (0.062208)	0.088326 / 0.176557 (-0.088230)	0.128209 / 0.737135 (-0.608926)	0.088807 / 0.296338 (-0.207531)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.291782 / 0.215209 (0.076573)	2.882990 / 2.077655 (0.805335)	1.601638 / 1.504120 (0.097518)	1.457560 / 1.541195 (-0.083635)	1.470517 / 1.468490 (0.002027)	0.565738 / 4.584777 (-4.019039)	0.949235 / 3.745712 (-2.796478)	2.661927 / 5.269862 (-2.607934)	1.722178 / 4.565676 (-2.843498)	0.063680 / 0.424275 (-0.360595)	0.005339 / 0.007607 (-0.002268)	0.344280 / 0.226044 (0.118235)	3.432998 / 2.268929 (1.164070)	1.985516 / 55.444624 (-53.459108)	1.651826 / 6.876477 (-5.224651)	1.764541 / 2.142072 (-0.377531)	0.640219 / 4.805227 (-4.165008)	0.116541 / 6.500664 (-6.384124)	0.041237 / 0.075469 (-0.034232)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.013927 / 1.841788 (-0.827861)	11.876661 / 8.074308 (3.802353)	10.264144 / 10.191392 (0.072752)	0.131151 / 0.680424 (-0.549273)	0.015774 / 0.534201 (-0.518427)	0.284948 / 0.579283 (-0.294335)	0.125924 / 0.434364 (-0.308439)	0.319845 / 0.540337 (-0.220493)	0.431978 / 1.386936 (-0.954958)