Downloading “reddit” dataset keeps timing out.

[smeyerhot]

Describe the bug

A clear and concise description of what the bug is.
Everytime I try and download the reddit dataset it times out before finishing and I have to try again.

There is some timeout error that I will post once it happens again.

Steps to reproduce the bug

from datasets import load_dataset

dataset = load_dataset("reddit", ignore_verifications=True, cache_dir="/Volumes/My Passport for Mac/og-chat-data")

Expected results

A clear and concise description of the expected results.

I would expect the download to finish, or at least provide a parameter to extend the read timeout window.

Actual results

Specify the actual results or traceback.

Shown below in error message.

Environment info

• datasets version: 1.11.0
• Platform: macOS
• Python version: 3.9.6 (conda env)
• PyArrow version: N/A

[smeyerhot]

Using custom data configuration default
Downloading and preparing dataset reddit/default (download: 2.93 GiB, generated: 17.64 GiB, post-processed: Unknown size, total: 20.57 GiB) to /Volumes/My Passport for Mac/og-chat-data/reddit/default/1.0.0/98ba5abea674d3178f7588aa6518a5510dc0c6fa8176d9653a3546d5afcb3969...
Downloading: 13%
403M/3.14G [44:39<2:27:09, 310kB/s]
---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in _error_catcher(self)
    437             try:
--> 438                 yield
    439 

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content)
    518                 cache_content = False
--> 519                 data = self._fp.read(amt) if not fp_closed else b""
    520                 if (

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/http/client.py in read(self, amt)
    458             b = bytearray(amt)
--> 459             n = self.readinto(b)
    460             return memoryview(b)[:n].tobytes()

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/http/client.py in readinto(self, b)
    502         # (for example, reading in 1k chunks)
--> 503         n = self.fp.readinto(b)
    504         if not n and b:

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/socket.py in readinto(self, b)
    703             try:
--> 704                 return self._sock.recv_into(b)
    705             except timeout:

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/ssl.py in recv_into(self, buffer, nbytes, flags)
   1240                   self.__class__)
-> 1241             return self.read(nbytes, buffer)
   1242         else:

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/ssl.py in read(self, len, buffer)
   1098             if buffer is not None:
-> 1099                 return self._sslobj.read(len, buffer)
   1100             else:

timeout: The read operation timed out

During handling of the above exception, another exception occurred:

ReadTimeoutError                          Traceback (most recent call last)
/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/requests/models.py in generate()
    757                 try:
--> 758                     for chunk in self.raw.stream(chunk_size, decode_content=True):
    759                         yield chunk

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in stream(self, amt, decode_content)
    575             while not is_fp_closed(self._fp):
--> 576                 data = self.read(amt=amt, decode_content=decode_content)
    577 

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content)
    540                         # Content-Length are caught.
--> 541                         raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
    542 

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/contextlib.py in __exit__(self, type, value, traceback)
    134             try:
--> 135                 self.gen.throw(type, value, traceback)
    136             except StopIteration as exc:

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/urllib3/response.py in _error_catcher(self)
    442                 # there is yet no clean way to get at it from this context.
--> 443                 raise ReadTimeoutError(self._pool, None, "Read timed out.")
    444 

ReadTimeoutError: HTTPSConnectionPool(host='zenodo.org', port=443): Read timed out.

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
/var/folders/3f/md0t9sgj6rz8xy01fskttqdc0000gn/T/ipykernel_89016/1133441872.py in <module>
      1 from datasets import load_dataset
      2 
----> 3 dataset = load_dataset("reddit", ignore_verifications=True, cache_dir="/Volumes/My Passport for Mac/og-chat-data")

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
    845 
    846     # Download and prepare data
--> 847     builder_instance.download_and_prepare(
    848         download_config=download_config,
    849         download_mode=download_mode,

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    613                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    614                     if not downloaded_from_gcs:
--> 615                         self._download_and_prepare(
    616                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    617                         )

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    669         split_dict = SplitDict(dataset_name=self.name)
    670         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 671         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    672 
    673         # Checksums verification

~/.cache/huggingface/modules/datasets_modules/datasets/reddit/98ba5abea674d3178f7588aa6518a5510dc0c6fa8176d9653a3546d5afcb3969/reddit.py in _split_generators(self, dl_manager)
     73     def _split_generators(self, dl_manager):
     74         """Returns SplitGenerators."""
---> 75         dl_path = dl_manager.download_and_extract(_URL)
     76         return [
     77             datasets.SplitGenerator(

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/download_manager.py in download_and_extract(self, url_or_urls)
    287             extracted_path(s): `str`, extracted paths of given URL(s).
    288         """
--> 289         return self.extract(self.download(url_or_urls))
    290 
    291     def get_recorded_sizes_checksums(self):

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/download_manager.py in download(self, url_or_urls)
    195 
    196         start_time = datetime.now()
--> 197         downloaded_path_or_paths = map_nested(
    198             download_func,
    199             url_or_urls,

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)
    194     # Singleton
    195     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 196         return function(data_struct)
    197 
    198     disable_tqdm = bool(logger.getEffectiveLevel() > logging.INFO) or not utils.is_progress_bar_enabled()

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/download_manager.py in _download(self, url_or_filename, download_config)
    218             # append the relative path to the base_path
    219             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
--> 220         return cached_path(url_or_filename, download_config=download_config)
    221 
    222     def iter_archive(self, path):

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    286     if is_remote_url(url_or_filename):
    287         # URL, so get it from the cache (downloading if necessary)
--> 288         output_path = get_from_cache(
    289             url_or_filename,
    290             cache_dir=cache_dir,

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    643                 ftp_get(url, temp_file)
    644             else:
--> 645                 http_get(
    646                     url,
    647                     temp_file,

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/datasets/utils/file_utils.py in http_get(url, temp_file, proxies, resume_size, headers, cookies, timeout, max_retries)
    451         disable=bool(logging.get_verbosity() == logging.NOTSET),
    452     )
--> 453     for chunk in response.iter_content(chunk_size=1024):
    454         if chunk:  # filter out keep-alive new chunks
    455             progress.update(len(chunk))

/usr/local/anaconda3/envs/og-data-env/lib/python3.9/site-packages/requests/models.py in generate()
    763                     raise ContentDecodingError(e)
    764                 except ReadTimeoutError as e:
--> 765                     raise ConnectionError(e)
    766             else:
    767                 # Standard file-like object.

ConnectionError: HTTPSConnectionPool(host='zenodo.org', port=443): Read timed out.

[TejasKarkera10]

Hey @lhoestq should I try to fix this issue ?

[smeyerhot]

It also doesn't seem to be "smart caching" and I received an error about a file not being found...

[smeyerhot]

To be clear, the error I get when I try to "re-instantiate" the download after failure is:

OSError: Cannot find data file. 
Original error:
[Errno 20] Not a directory: <HOME>/.cache/huggingface/datasets/downloads/1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c/corpus-webis-tldr-17.json'

[smeyerhot]

Here is a new error:

ConnectionError: Couldn't reach https://zenodo.org/record/1043504/files/corpus-webis-tldr-17.zip?download=1

[lhoestq]

Hi ! Since #2803 we've changed the time out from 10sec to 100sec.
This should prevent the ReadTimeoutError. Feel free to try it out by installing datasets from source

pip install git+https://github.com/huggingface/datasets.git

When re-running your code you said you get a OSError, could you try deleting the file at the path returned by the error ? (the one after [Errno 20] Not a directory:). Ideally when a download fails you should be able to re-run it without error; there might be an issue here.

Finally not sure what we can do about ConnectionError, this must be an issue from zenodo. If it happens you simply need to try again

[smeyerhot]

@lhoestq thanks for the update. The directory specified by the OSError ie.

1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c/corpus-webis-tldr-17.json 

was not actually in that directory so I can't delete it.

[lhoestq]

Oh, then could you try deleting the parent directory 1ec12301abba4daa60eb3a90e53529b5b173296b22dc3bef3186e205c75e594c instead ?
This way the download manager will know that it has to uncompress the data again

[smeyerhot]

It seems to have worked. It only took like 20min! I think the extra timeout length did the trick! One thing is that it downloaded a total of 41gb instead of 20gb but at least it finished.

[lhoestq]

Great ! The timeout change will be available in the next release of datasets :)