Doc readme improvements

docs/README.md

@@ -50,20 +50,33 @@ You can adapt the `--build_dir` to set any temporary folder that you prefer. Thi
 the MDX files that will be rendered as the documentation on the main website. You can inspect them in your favorite
 Markdown editor.
 
----
-**NOTE**
+## Previewing the documentation
+
+To preview the docs, first install the `watchdog` module with:
+
+```bash
+pip install watchdog
+```
 
-It's not possible to see locally how the final documentation will look like for now. Once you have opened a PR, you
-will see a bot add a comment to a link where the documentation with your changes lives.
+Then run the following command:
+
+```bash
+doc-builder preview datasets docs/source/
+```
+
+The docs will be viewable at [http://localhost:3000](http://localhost:3000). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives.
 
 ---
+**NOTE**
+
+The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again).
 
 ## Adding a new element to the navigation bar
 
 Accepted files are Markdown (.md or .mdx).
 
 Create a file with its extension and put it in the source directory. You can then link it to the toc-tree by putting
-the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/transformers/blob/master/docs/source/_toctree.yml) file.
+the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/datasets/blob/main/docs/source/_toctree.yml) file.
 
 ## Renaming section headers and moving sections
 
@@ -88,12 +101,12 @@ Sections that were moved:
 
 Use the relative style to link to the new file so that the versioned docs continue to work.
 
-For an example of a rich moved sections set please see the very end of [the Trainer doc](https://github.com/huggingface/transformers/blob/master/docs/source/main_classes/trainer.mdx).
+For an example of a rich moved sections set please see the very end of [the transformers Trainer doc](https://github.com/huggingface/transformers/blob/main/docs/source/en/main_classes/trainer.md).
 
 
 ## Writing Documentation - Specification
 
-The `huggingface/transformers` documentation follows the
+The `huggingface/datasets` documentation follows the
 [Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style for docstrings,
 although we can write them directly in Markdown.
 
@@ -104,63 +117,7 @@ Adding a new tutorial or section is done in two steps:
 - Add a new file under `./source`. This file can either be ReStructuredText (.rst) or Markdown (.md).
 - Link that file in `./source/_toctree.yml` on the correct toc-tree.
 
-Make sure to put your new file under the proper section. It's unlikely to go in the first section (*Get Started*), so
-depending on the intended targets (beginners, more advanced users or researchers) it should go into section two, three or
-four.
-
-### Adding a new model
-
-When adding a new model:
-
-- Create a file `xxx.mdx` or under `./source/model_doc` (don't hesitate to copy an existing file as template).
-- Link that file in `./source/_toctree.yml`.
-- Write a short overview of the model:
-    - Overview with paper & authors
-    - Paper abstract
-    - Tips and tricks and how to use it best
-- Add the classes that should be linked in the model. This generally includes the configuration, the tokenizer, and
-  every model of that class (the base model, alongside models with additional heads), both in PyTorch and TensorFlow.
-  The order is generally:
-    - Configuration,
-    - Tokenizer
-    - PyTorch base model
-    - PyTorch head models
-    - TensorFlow base model
-    - TensorFlow head models
-    - Flax base model
-    - Flax head models
-
-These classes should be added using our Markdown syntax. Usually as follows:
-
-```
-## XXXConfig
-
-[[autodoc]] XXXConfig
-```
-
-This will include every public method of the configuration that is documented. If for some reason you wish for a method
-not to be displayed in the documentation, you can do so by specifying which methods should be in the docs:
-
-```
-## XXXTokenizer
-
-[[autodoc]] XXXTokenizer
-    - build_inputs_with_special_tokens
-    - get_special_tokens_mask
-    - create_token_type_ids_from_sequences
-    - save_vocabulary
-```
-
-If you just want to add a method that is not documented (for instance magic method like `__call__` is not documented
-by default) you can put the list of methods to add in a list that contains `all`:
-
-```
-## XXXTokenizer
-
-[[autodoc]] XXXTokenizer
-    - all
-    - __call__
-```
+Make sure to put your new file under the proper section. If you have a doubt, feel free to ask in a Github Issue or PR.
 
 ### Writing source documentation
 
@@ -172,9 +129,9 @@ adds a link to its documentation with this syntax: \[\`XXXClass\`\] or \[\`funct
 function to be in the main package.
 
 If you want to create a link to some internal class or function, you need to
-provide its path. For instance: \[\`file_utils.ModelOutput\`\]. This will be converted into a link with
-`file_utils.ModelOutput` in the description. To get rid of the path and only keep the name of the object you are
-linking to in the description, add a ~: \[\`~file_utils.ModelOutput\`\] will generate a link with `ModelOutput` in the description.
+provide its path. For instance: \[\`table.InMemoryTable\`\]. This will be converted into a link with
+`table.InMemoryTable` in the description. To get rid of the path and only keep the name of the object you are
+linking to in the description, add a ~: \[\`~table.InMemoryTable\`\] will generate a link with `InMemoryTable` in the description.
 
 The same works for methods so you can either use \[\`XXXClass.method\`\] or \[~\`XXXClass.method\`\].
 
@@ -239,9 +196,6 @@ Multi-line code blocks can be useful for displaying examples. They are done betw
 ```
 ````
 
-We follow the [doctest](https://docs.python.org/3/library/doctest.html) syntax for the examples to automatically test
-the results stay consistent with the library.
-
 #### Writing a return block
 
 The return block should be introduced with the `Returns:` prefix, followed by a line return and an indentation.
@@ -274,12 +228,33 @@ them by URL. We recommend putting them in the following dataset: [huggingface/do
 If an external contribution, feel free to add the images to your PR and ask a Hugging Face member to migrate your images
 to this dataset.
 
-## Styling the docstring
+## Writing documentation examples
 
-We have an automatic script running with the `make style` comment that will make sure that:
-- the docstrings fully take advantage of the line width
-- all code examples are formatted using black, like the code of the Transformers library
+The syntax for Example docstrings can look as follows:
+
+```
+    Example:
+
+    ```py
+    >>> from datasets import load_dataset
+    >>> ds = load_dataset("rotten_tomatoes", split="validation")
+    >>> def add_prefix(example):
+    ...     example["text"] = "Review: " + example["text"]
+    ...     return example
+    >>> ds = ds.map(add_prefix)
+    >>> ds[0:3]["text"]
+    ['Review: compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .',
+        'Review: the soundtrack alone is worth the price of admission .',
+        'Review: rodriguez does a splendid job of racial profiling hollywood style--casting excellent latin actors of all ages--a trend long overdue .']
+
+    # process a batch of examples
+    >>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
+    # set number of processors
+    >>> ds = ds.map(add_prefix, num_proc=4)
+    ```
+```
 
-This script may have some weird failures if you make a syntax mistake or if you uncover a bug. Therefore, it's
-recommended to commit your changes before running `make style`, so you can revert the changes done by that script
-easily.
+The docstring should give a minimal, clear example of how the respective class or function is to be used in practice and also include the expected (ideally sensible) output.
+Often, readers will try out the example before even going through the function 
+or class definitions. Therefore, it is of utmost importance that the example 
+works as expected.