doc: switch to Markdown (MyST parser) (#324)

* doc: switch to Markdown (MyST parser)

* Fix mypy error

* Fix typos and missing requirement for rtd

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: araffin

docs/README.md

@@ -1,4 +1,4 @@
-## Stable Baselines3 Contrib Documentation
+# Stable Baselines3 Contrib Documentation
 
 This folder contains documentation for the RL baselines contribution repository.
 

docs/common/torch_layers.rst docs/common/torch_layers.mddocs/common/torch_layers.rst renamed to docs/common/torch_layers.md

@@ -1,7 +1,8 @@
-.. _th_layers:
+(th-layers)=
 
-Torch Layers
-============
+# Torch Layers
 
+```{eval-rst}
 .. automodule:: sb3_contrib.common.torch_layers
   :members:
+```

docs/common/utils.rst docs/common/utils.mddocs/common/utils.rst renamed to docs/common/utils.md

@@ -1,7 +1,8 @@
-.. _utils:
+(utils)=
 
-Utils
-=====
+# Utils
 
+```{eval-rst}
 .. automodule:: sb3_contrib.common.utils
   :members:
+```

docs/common/wrappers.md

@@ -0,0 +1,17 @@
+(wrappers)=
+
+# Gym Wrappers
+
+Additional [Gymnasium Wrappers](https://gymnasium.farama.org/api/wrappers/) to enhance Gymnasium environments.
+
+```{eval-rst}
+.. automodule:: sb3_contrib.common.wrappers
+
+```
+
+## TimeFeatureWrapper
+
+```{eval-rst}
+.. autoclass:: TimeFeatureWrapper
+  :members:
+```

docs/common/wrappers.rst

@@ -18,3 +18,4 @@ dependencies:
     - sphinx>=5,<8
     - sphinx_rtd_theme>=1.3.0
     - sphinx_copybutton
+    - myst-parser>=4,<6

docs/conda_env.yml

@@ -70,6 +70,7 @@
     "sphinx.ext.viewcode",
     # 'sphinx.ext.intersphinx',
     # 'sphinx.ext.doctest'
+    "myst_parser",
 ]
 
 autodoc_typehints = "description"
@@ -86,8 +87,7 @@
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-# source_suffix = ['.rst', '.md']
-source_suffix = ".rst"
+source_suffix = [".rst", ".md"]
 
 # The master toctree document.
 master_doc = "index"
@@ -102,7 +102,7 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path .
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "README.md"]
 
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = "sphinx"
@@ -200,6 +200,25 @@ def setup(app):
 
 # -- Extension configuration -------------------------------------------------
 
+myst_heading_anchors = 4
+# See: https://myst-parser.readthedocs.io/en/latest/syntax/optional.html
+myst_enable_extensions = [
+    # "amsmath",
+    "attrs_inline",
+    "colon_fence",
+    "deflist",
+    "dollarmath",
+    "fieldlist",
+    # "html_admonition",
+    "html_image",
+    # "linkify",
+    # "replacements",
+    # "smartquotes",
+    # "strikethrough",
+    "substitution",
+    # "tasklist",
+]
+
 # Example configuration for intersphinx: refer to the Python standard library.
 # intersphinx_mapping = {
 #     'python': ('https://docs.python.org/3/', None),

docs/conf.py

@@ -0,0 +1,28 @@
+# RL Algorithms
+
+This table displays the rl algorithms that are implemented in the Stable Baselines3 contrib project,
+along with some useful characteristics: support for discrete/continuous actions, multiprocessing.
+
+| Name         | `Box` | `Discrete` | `MultiDiscrete` | `MultiBinary` | Multi Processing |
+| ------------ | ----- | ---------- | --------------- | ------------- | ---------------- |
+| ARS          | ✔️    | ❌️         | ❌              | ❌            | ✔️               |
+| MaskablePPO  | ❌    | ✔️         | ✔️              | ✔️            | ✔️               |
+| CrossQ       | ✔️    | ❌         | ❌              | ❌            | ✔️               |
+| QR-DQN       | ❌    | ✔️         | ❌              | ❌            | ✔️               |
+| RecurrentPPO | ✔️    | ✔️         | ✔️              | ✔️            | ✔️               |
+| TQC          | ✔️    | ❌         | ❌              | ❌            | ✔️               |
+| TRPO         | ✔️    | ✔️         | ✔️              | ✔️            | ✔️               |
+
+:::{note}
+`Tuple` observation spaces are not supported by any environment,
+however, single-level `Dict` spaces are supported.
+:::
+
+Actions `gym.spaces`:
+
+- `Box`: A N-dimensional box that contains every point in the action
+  space.
+- `Discrete`: A list of possible actions, where each timestep only
+  one of the actions can be used.
+- `MultiDiscrete`: A list of possible actions, where each timestep only one action of each discrete set can be used.
+- `MultiBinary`: A list of possible actions, where each timestep any of the actions can be used in any combination.

docs/guide/algos.md

@@ -0,0 +1,126 @@
+(examples)=
+
+# Examples
+
+## TQC
+
+Train a Truncated Quantile Critics (TQC) agent on the Pendulum environment.
+
+```python
+from sb3_contrib import TQC
+
+model = TQC("MlpPolicy", "Pendulum-v1", top_quantiles_to_drop_per_net=2, verbose=1)
+model.learn(total_timesteps=10_000, log_interval=4)
+model.save("tqc_pendulum")
+```
+
+## QR-DQN
+
+Train a Quantile Regression DQN (QR-DQN) agent on the CartPole environment.
+
+```python
+from sb3_contrib import QRDQN
+
+policy_kwargs = dict(n_quantiles=50)
+model = QRDQN("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
+model.learn(total_timesteps=10_000, log_interval=4)
+model.save("qrdqn_cartpole")
+```
+
+## MaskablePPO
+
+Train a PPO with invalid action masking agent on a toy environment.
+
+:::{warning}
+You must use `MaskableEvalCallback` from `sb3_contrib.common.maskable.callbacks` instead of the base `EvalCallback` to properly evaluate a model with action masks.
+Similarly, you must use `evaluate_policy` from `sb3_contrib.common.maskable.evaluation` instead of the SB3 one.
+:::
+
+```python
+from sb3_contrib import MaskablePPO
+from sb3_contrib.common.envs import InvalidActionEnvDiscrete
+
+env = InvalidActionEnvDiscrete(dim=80, n_invalid_actions=60)
+model = MaskablePPO("MlpPolicy", env, verbose=1)
+model.learn(5000)
+model.save("maskable_toy_env")
+```
+
+## TRPO
+
+Train a Trust Region Policy Optimization (TRPO) agent on the Pendulum environment.
+
+```python
+from sb3_contrib import TRPO
+
+model = TRPO("MlpPolicy", "Pendulum-v1", gamma=0.9, verbose=1)
+model.learn(total_timesteps=100_000, log_interval=4)
+model.save("trpo_pendulum")
+```
+
+## ARS
+
+Train an agent using Augmented Random Search (ARS) agent on the Pendulum environment
+
+```python
+from sb3_contrib import ARS
+
+model = ARS("LinearPolicy", "Pendulum-v1", verbose=1)
+model.learn(total_timesteps=10000, log_interval=4)
+model.save("ars_pendulum")
+```
+
+## RecurrentPPO
+
+Train a PPO agent with a recurrent policy on the CartPole environment.
+
+:::{note}
+It is particularly important to pass the `lstm_states`
+and `episode_start` argument to the `predict()` method,
+so the cell and hidden states of the LSTM are correctly updated.
+:::
+
+```python
+import numpy as np
+
+from sb3_contrib import RecurrentPPO
+
+model = RecurrentPPO("MlpLstmPolicy", "CartPole-v1", verbose=1)
+model.learn(5000)
+
+vec_env = model.get_env()
+obs = vec_env.reset()
+# Cell and hidden state of the LSTM
+lstm_states = None
+num_envs = 1
+# Episode start signals are used to reset the lstm states
+episode_starts = np.ones((num_envs,), dtype=bool)
+while True:
+    action, lstm_states = model.predict(obs, state=lstm_states, episode_start=episode_starts, deterministic=True)
+    # Note: vectorized environment resets automatically
+    obs, rewards, dones, info = vec_env.step(action)
+    episode_starts = dones
+    vec_env.render("human")
+```
+
+## CrossQ
+
+Train a CrossQ agent on the Pendulum environment.
+
+```python
+from sb3_contrib import CrossQ
+
+model = CrossQ(
+    "MlpPolicy",
+    "Pendulum-v1",
+    verbose=1,
+    policy_kwargs=dict(
+        net_arch=dict(
+            pi=[256, 256],
+            qf=[1024, 1024],
+        )
+    ),
+)
+model.learn(total_timesteps=5_000, log_interval=4)
+model.save("crossq_pendulum")
+```