add stack exchange

datasets/stack_exchange/README.md

@@ -0,0 +1,166 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- found
+languages:
+- en
+licenses:
+- mit
+multilinguality:
+- monolingual
+pretty_name: Stack Exchange
+size_categories:
+- 1M<n<10M
+source_datasets:
+- original
+task_categories:
+- sequence-modeling
+task_ids:
+- language-modeling
+---
+
+# Dataset Card for Stack Exchange
+
+## Table of Contents
+- [Dataset Card for Stack Exchange](#dataset-card-for-stack-exchange)
+  - [Table of Contents](#table-of-contents)
+  - [Dataset Description](#dataset-description)
+    - [Dataset Summary](#dataset-summary)
+    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+    - [Languages](#languages)
+  - [Dataset Structure](#dataset-structure)
+    - [Data Instances](#data-instances)
+    - [Data Fields](#data-fields)
+    - [Data Splits](#data-splits)
+  - [|split|num examples|](#splitnum-examples)
+  - [Dataset Creation](#dataset-creation)
+    - [Curation Rationale](#curation-rationale)
+    - [Source Data](#source-data)
+      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
+      - [Who are the source language producers?](#who-are-the-source-language-producers)
+    - [Annotations](#annotations)
+      - [Annotation process](#annotation-process)
+      - [Who are the annotators?](#who-are-the-annotators)
+    - [Personal and Sensitive Information](#personal-and-sensitive-information)
+  - [Considerations for Using the Data](#considerations-for-using-the-data)
+    - [Social Impact of Dataset](#social-impact-of-dataset)
+    - [Discussion of Biases](#discussion-of-biases)
+    - [Other Known Limitations](#other-known-limitations)
+  - [Additional Information](#additional-information)
+    - [Dataset Curators](#dataset-curators)
+    - [Licensing Information](#licensing-information)
+    - [Citation Information](#citation-information)
+    - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [GitHub](https://github.com/EleutherAI/stackexchange-dataset)
+- **Repository:** [Needs More Information]
+- **Paper:** [arXiv](https://arxiv.org/abs/2101.00027)
+- **Leaderboard:** [Needs More Information]
+- **Point of Contact:** [Needs More Information]
+
+### Dataset Summary
+
+This dataset is part of EleutherAI/The Pile dataset and is a dataset for Language Models from processing stackexchange data dump, which is an anonymized dump of all user-contributed content on the Stack Exchange network.
+
+|download_size|34.28 Gib|
+|dataset_size|10.3 Gib|
+
+### Supported Tasks and Leaderboards
+
+The dataset is used for Language Modeling.
+
+### Languages
+
+The dataset is in English.
+
+## Dataset Structure
+
+### Data Instances
+
+```
+{'domain': 'chemistry',
+'text':"\nQ:                                                                                                                                            \n                                                                                                                                              \nReviving old questions or asking a new one?                                                                                                   \n                                                                                                                                              \nI'm relatively new to the Chemistry SE community, and sometimes when I go to ask a question, I notice that the same (or similar) question has \nalready been asked. However, the previous question doesn't have a good answer (or is unanswered). In this case, is it better to ask the questi\non again in a new post (which might be marked as duplicate) or comment on the old post (which might be several years old)? In other words, wha\nt are the customs of this site in regards to reviving old questions/discussions?\n\nA:\n\nAs Martin commented, it really depends on the type of question. In any case, you always have the following possibilities:\n\nAsk a new question\nEdit the question to bump it to the first page\nAdd a bounty\nBring it to the attention of people in chat\n\nConsider the following cases:\n\nI have exactly the same question as asked and unanswered before!\n\nIf you ask a new question which turns out to be the same question, it may be closed as a dupe (depending on whether users remember the old que\nstion). Not the ideal option.\nIf you can find something substantial to edit and bump the question, do so. Maybe add a comment that you would really love an answer.\nIf you can spare some rep for a bounty (50 is usually enough), do so.\nYou can always bring it to the attention of people in chat.\n",}
+```
+
+### Data Fields
+
+- `domain`: Stack Exchange domain of the sample
+- `text`: Text content containing both the question and the answer
+
+### Data Splits
+
+|split|num examples|
+--------------------------------
+|train|5096117|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[Needs More Information]
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[Needs More Information]
+
+#### Who are the source language producers?
+
+[Needs More Information]
+
+### Annotations
+
+#### Annotation process
+
+[Needs More Information]
+
+#### Who are the annotators?
+
+[Needs More Information]
+
+### Personal and Sensitive Information
+
+[Needs More Information]
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[Needs More Information]
+
+### Discussion of Biases
+
+[Needs More Information]
+
+### Other Known Limitations
+
+[Needs More Information]
+
+## Additional Information
+
+### Dataset Curators
+
+[Needs More Information]
+
+### Licensing Information
+
+[Needs More Information]
+
+### Citation Information
+
+```
+@article{pile,
+    title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},
+    author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},
+    journal={arXiv preprint arXiv:2101.00027},
+    year={2020}
+}
+```
+
+### Contributions
+Thanks to [sdtblck](https://github.com/sdtblck) for creating the dataset.
+Thanks to [richarddwang](https://github.com/richarddwang) for adding the dataset.

datasets/stack_exchange/dataset_infos.json

@@ -0,0 +1 @@
+{"plain_text": {"description": "This dataset is part of EleutherAI/The Pile dataset and is a dataset for Language Models from processing stackexchange data dump, which is an anonymized dump of all user-contributed content on the Stack Exchange network.\n", "citation": "@article{pile,\n    title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\n    author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\n    journal={arXiv preprint arXiv:2101.00027},\n    year={2020}\n}\n", "homepage": "https://github.com/EleutherAI/stackexchange-dataset", "license": "", "features": {"domain": {"dtype": "string", "id": null, "_type": "Value"}, "text": {"dtype": "string", "id": null, "_type": "Value"}}, "post_processed": null, "supervised_keys": null, "task_templates": null, "builder_name": "stack_exchange", "config_name": "plain_text", "version": {"version_str": "1.0.0", "description": null, "major": 1, "minor": 0, "patch": 0}, "splits": {"train": {"name": "train", "num_bytes": 11075434609, "num_examples": 5096117, "dataset_name": "stack_exchange"}}, "download_checksums": {"https://the-eye.eu/public/AI/pile_preliminary_components/stackexchange_dataset.tar": {"num_bytes": 36802959360, "checksum": "f64f31d20db8d8692c1a019314a14974b4911a34ffef126feaf42da88860c666"}}, "download_size": 36802959360, "post_processing_size": null, "dataset_size": 11075434609, "size_in_bytes": 47878393969}}

datasets/stack_exchange/stack_exchange.py

@@ -0,0 +1,80 @@
+# coding=utf-8
+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""The Stack Exchange Corpus"""
+
+import os
+from pathlib import Path
+
+import datasets
+
+
+_CITATION = """\
+@article{pile,
+    title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},
+    author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},
+    journal={arXiv preprint arXiv:2101.00027},
+    year={2020}
+}
+"""
+
+_DESCRIPTION = """\
+This dataset is part of EleutherAI/The Pile dataset and is a dataset for Language Models from processing stackexchange data dump, \
+which is an anonymized dump of all user-contributed content on the Stack Exchange network.
+"""
+
+_URL = "https://the-eye.eu/public/AI/pile_preliminary_components/stackexchange_dataset.tar"
+
+
+class StackExchange(datasets.GeneratorBasedBuilder):
+    """The StackExchange dataset."""
+
+    BUILDER_CONFIGS = [
+        datasets.BuilderConfig(
+            name="plain_text",
+            description="Plain text",
+            version=datasets.Version("1.0.0"),
+        )
+    ]
+
+    def _info(self):
+        return datasets.DatasetInfo(
+            description=_DESCRIPTION,
+            features=datasets.Features({"domain": datasets.Value("string"), "text": datasets.Value("string")}),
+            homepage="https://github.com/EleutherAI/stackexchange-dataset",
+            citation=_CITATION,
+        )
+
+    def _split_generators(self, dl_manager):
+        dl_dir = dl_manager.download_and_extract(_URL)
+        zips = [str(f) for f in (Path(dl_dir) / "out").iterdir()]
+        extracted = dl_manager.extract(zips, num_proc=os.cpu_count())
+        # non-dir extracteds are zero-size unknown things
+        dirs = [path for path in extracted if os.path.isdir(path)]
+        return [
+            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"dirs": dirs}),
+        ]
+
+    def _generate_examples(self, dirs):
+        """Yields examples."""
+        _id = 0
+        for dir in sorted(dirs):
+            txt_files = sorted(Path(dir).glob("**/*.txt"))
+            for txt_file in txt_files:
+                # PosiPath(/home/user/.cache/huggingface/datasets/downloads/extracted/3923d60abeeb876021dc55a897ac2f260b181556f8ca56a7c61e3b8b80afec77/academia.stackexchange_0000000001.txt)
+                domain = txt_file.name.split(".")[0]
+                with txt_file.open(mode="r", encoding="utf-8") as f:
+                    document = f.read()
+                yield _id, {"domain": domain, "text": document}
+                _id += 1

src/datasets/utils/file_utils.py

@@ -424,7 +424,7 @@ def ftp_get(url, temp_file, timeout=10.0):
         raise ConnectionError(e)
 
 
-def http_get(url, temp_file, proxies=None, resume_size=0, headers=None, cookies=None, timeout=10.0, max_retries=0):
+def http_get(url, temp_file, proxies=None, resume_size=0, headers=None, cookies=None, timeout=100.0, max_retries=0):
     headers = copy.deepcopy(headers) or {}
     headers["user-agent"] = get_datasets_user_agent(user_agent=headers.get("user-agent"))
     if resume_size > 0: