`load_dataset` requires a network connection for local download?

[MiroFurtado]

Describe the bug

Hi all - I see that in the past a network dependency has been mistakenly introduced into load_dataset even for local loads. Is it possible this has happened again?

Steps to reproduce the bug

>>> import datasets
>>> datasets.load_dataset("hh-rlhf")
Repo card metadata block was not found. Setting CardData to empty.
*hangs bc i'm firewalled*

stack trace from ctrl-c:

^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
    output_path = get_from_cache(                                                                                                                              [0/122]
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 532, in get_from_cache
    response = http_head(
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 419, in http_head
    response = _request_with_retry(
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 304, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
KeyboardInterrupt

Expected behavior

loads the dataset

Environment info

> pip show datasets
Name: datasets
Version: 2.18.0

Python 3.10.2

[Dref360]

Are you using HF_DATASETS_OFFLINE=1 ?

[izhx]

Are you using HF_DATASETS_OFFLINE=1 ?

This doesn't work for me. datasets=2.18.0

test.py:

import datasets

datasets.utils.logging.set_verbosity_info()

ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')

print(ds)

run python test.py

Generating dataset afqmc (/home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4)
Downloading and preparing dataset afqmc/default to /home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4...
Dataset not on Hf google storage. Downloading and preparing it from source
hf://datasets/C-MTEB/AFQMC@b44c3b011063adb25877c13823db83bb193913c4/data/validation-00000-of-00001-b8fc393b5ddedac7.parquet not found in cache or force_download set to True, downloading to /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b.incomplete
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 240k/240k [00:01<00:00, 178kB/s]
storing hf://datasets/C-MTEB/AFQMC@b44c3b011063adb25877c13823db83bb193913c4/data/validation-00000-of-00001-b8fc393b5ddedac7.parquet in cache at /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b
creating metadata file for /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b
Downloading took 0.0 min
Checksum Computation took 0.0 min
Generating test split
Generating test split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3861/3861 [00:00<00:00, 3972.00 examples/s]
Generating train split
Generating train split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34334/34334 [00:00<00:00, 34355.50 examples/s]
Generating validation split
Generating validation split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4316/4316 [00:00<00:00, 4477.00 examples/s]
All the splits matched successfully.
Dataset afqmc downloaded and prepared to /home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4. Subsequent calls will reuse this data.
DatasetDict({
    test: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 3861
    })
    train: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 34334
    })
    validation: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 4316
    })
})

Then run HF_DATASETS_OFFLINE=1 python test.py

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'C-MTEB/AFQMC': Offline mode is enabled.

[freckletonj]

I was having similar inexplicable issues.

Doing this I think helped, but, datasets still clearly does not want to respect the cache:

pip install --upgrade datasets  # now it is 2.18.0
HF_DATASETS_OFFLINE="1" python blah.py

Or similarly, I must spacify that env var to resuse the cache, IE, no arg to load_dataset helps it reuse the cache:

import os
os.environ["HF_DATASETS_OFFLINE"] = "1"

import logging
logging.basicConfig(level=logging.DEBUG)

import datasets
# >>> datasets.__version__
# '2.18.0'

datasets.utils.logging.set_verbosity_info()
data = datasets.load_dataset("c-s-ale/dolly-15k-instruction-alpaca-format")