Webdataset dataset builder

docs/source/image_dataset.mdx

@@ -148,6 +148,46 @@ Upload your dataset with [`~datasets.DatasetDict.push_to_hub`]:
 >>> dataset.push_to_hub("stevhliu/my-image-captioning-dataset")
 ```
 
+## WebDataset
+
+The [WebDataset](https://github.com/webdataset/webdataset) format is based on TAR archives and is suitable for big image datasets.
+Indeed you can group your images in TAR archives (e.g. 1GB of images per TAR archive) and have thousands of TAR archives:
+
+```
+folder/train/00000.tar
+folder/train/00001.tar
+folder/train/00002.tar
+...
+```
+
+In the archives, each example is made of files sharing the same prefix:
+
+```
+e39871fd9fd74f55.jpg
+e39871fd9fd74f55.json
+f18b91585c4d3f3e.jpg
+f18b91585c4d3f3e.json
+ede6e66b2fb59aab.jpg
+ede6e66b2fb59aab.json
+ed600d57fcee4f94.jpg
+ed600d57fcee4f94.json
+...
+```
+
+You can put your images labels/captions/bounding boxes using JSON or text files for example.
+
+For more details on the WebDataset format and the python library, please check the [WebDataset documentation](https://webdataset.github.io/webdataset).
+
+Load your WebDataset and it will create on column per file suffix (here "jpg" and "json"):
+
+```python
+>>> from datasets import load_dataset
+
+>>> dataset = load_dataset("webdataset", data_dir="/path/to/folder", split="train")
+>>> dataset[0]["json"]
+{"bbox": [[302.0, 109.0, 73.0, 52.0]], "categories": [0]}
+```
+
 ## Loading script
 
 Write a dataset loading script to share a dataset. It defines a dataset's splits and configurations, and handles downloading and generating a dataset. The script is located in the same folder or repository as the dataset and should have the same name.

docs/source/image_load.mdx

@@ -1,14 +1,14 @@
 # Load image data
 
-Image datasets are loaded from the `image` column, which contains a PIL object. 
+Image datasets have [`Image`] type columns, which contain PIL objects. 
 
 <Tip>
 
 To work with image datasets, you need to have the `vision` dependency installed. Check out the [installation](./installation#vision) guide to learn how to install it.
 
 </Tip>
 
-When you load an image dataset and call the `image` column, the [`Image`] feature automatically decodes the PIL object into an image:
+When you load an image dataset and call the image column, the images are decoded as PIL Images:
 
 ```py
 >>> from datasets import load_dataset, Image
@@ -93,3 +93,16 @@ To ignore the information in the metadata file, set `drop_labels=False` in [`loa
 For more information about creating your own `ImageFolder` dataset, take a look at the [Create an image dataset](./image_dataset) guide.
 
 </Tip>
+
+## WebDataset
+
+The [WebDataset](https://github.com/webdataset/webdataset) format is based on a folder of TAR archives and is suitable for big image datasets.
+Because of their size, WebDatasets are generally loaded in streaming mode (using `streaming=True`).
+
+You can load a WebDataset like this:
+
+```python
+>>> from datasets import load_dataset
+
+>>> dataset = load_dataset("webdataset", data_dir="/path/to/folder", streaming=True)
+```

docs/source/loading.mdx

@@ -224,6 +224,31 @@ For more details, check out the [how to load tabular datasets from SQL databases
 
 </Tip>
 
+### WebDataset
+
+The [WebDataset](https://github.com/webdataset/webdataset) format is based on TAR archives and is suitable for big image datasets.
+Because of their size, WebDatasets are generally loaded in streaming mode (using `streaming=True`).
+
+You can load a WebDataset like this:
+
+```python
+>>> from datasets import load_dataset
+>>>
+>>> path = "path/to/train/*.tar"
+>>> dataset = load_dataset("webdataset", data_files={"train": path}, split="train", streaming=True)
+```
+
+To load remote WebDatasets via HTTP, pass the URLs instead:
+
+```python
+>>> from datasets import load_dataset
+>>>
+>>> base_url = "https://huggingface.co/datasets/lhoestq/small-publaynet-wds/resolve/main/publaynet-train-{i:06d}.tar"
+>>> urls = [base_url.format(i=i) for i in range(4)]
+>>> dataset = load_dataset("webdataset", data_files={"train": urls}, split="train", streaming=True)
+```
+
+
 ## Multiprocessing
 
 When a dataset is made of several files (that we call "shards"), it is possible to significantly speed up the dataset downloading and preparation step.

docs/source/package_reference/loading_methods.mdx

@@ -96,3 +96,7 @@ load_dataset("csv", data_dir="path/to/data/dir", sep="\t")
 [[autodoc]] datasets.packaged_modules.audiofolder.AudioFolderConfig
 
 [[autodoc]] datasets.packaged_modules.audiofolder.AudioFolder
+
+### WebDataset
+
+[[autodoc]] datasets.packaged_modules.webdataset.WebDataset

src/datasets/config.py

@@ -141,7 +141,6 @@
 LZ4_AVAILABLE = importlib.util.find_spec("lz4") is not None
 PY7ZR_AVAILABLE = importlib.util.find_spec("py7zr") is not None
 
-
 # Cache location
 DEFAULT_XDG_CACHE_HOME = "~/.cache"
 XDG_CACHE_HOME = os.getenv("XDG_CACHE_HOME", DEFAULT_XDG_CACHE_HOME)

src/datasets/packaged_modules/__init__.py

@@ -12,6 +12,7 @@
 from .parquet import parquet
 from .sql import sql  # noqa F401
 from .text import text
+from .webdataset import webdataset
 
 
 def _hash_python_lines(lines: List[str]) -> str:
@@ -37,6 +38,7 @@ def _hash_python_lines(lines: List[str]) -> str:
     "text": (text.__name__, _hash_python_lines(inspect.getsource(text).splitlines())),
     "imagefolder": (imagefolder.__name__, _hash_python_lines(inspect.getsource(imagefolder).splitlines())),
     "audiofolder": (audiofolder.__name__, _hash_python_lines(inspect.getsource(audiofolder).splitlines())),
+    "webdataset": (webdataset.__name__, _hash_python_lines(inspect.getsource(webdataset).splitlines())),
 }
 
 # Used to infer the module to use based on the data files extensions
@@ -48,6 +50,7 @@ def _hash_python_lines(lines: List[str]) -> str:
     ".parquet": ("parquet", {}),
     ".arrow": ("arrow", {}),
     ".txt": ("text", {}),
+    ".tar": ("webdataset", {}),
 }
 _EXTENSION_TO_MODULE.update({ext: ("imagefolder", {}) for ext in imagefolder.ImageFolder.EXTENSIONS})
 _EXTENSION_TO_MODULE.update({ext.upper(): ("imagefolder", {}) for ext in imagefolder.ImageFolder.EXTENSIONS})