[doctest] Provide definition of `default_evaluator` and other default objects

[vfdev-5]

📚 Documentation

It was already discussed and also pointed out in the forum that it is unclear how default objects are defined inside doctest examples. For example: https://pytorch.org/ignite/generated/ignite.metrics.Accuracy.html#ignite.metrics.Accuracy

metric = Accuracy()
metric.attach(default_evaluator, "accuracy")
y_true = torch.Tensor([1, 0, 1, 1, 0, 1])
y_pred = torch.Tensor([1, 0, 1, 0, 1, 1])
state = default_evaluator.run([[y_pred, y_true]])
print(state.metrics["accuracy"])

We can think of different solutions:

• 1. for all doctests, add a link on

     ignite/docs/source/conf.py

     Lines 334 to 378 in 111fc7a

     	doctest_global_setup = """
     	from collections import OrderedDict
     	import torch
     	from torch import nn, optim
     	from ignite.engine import *
     	from ignite.handlers import *
     	from ignite.metrics import *
     	from ignite.utils import *
     	from ignite.contrib.metrics.regression import *
     	from ignite.contrib.metrics import *
     	# create default evaluator for doctests
     	def eval_step(engine, batch):
     	return batch
     	default_evaluator = Engine(eval_step)
     	# create default optimizer for doctests
     	param_tensor = torch.zeros([1], requires_grad=True)
     	default_optimizer = torch.optim.SGD([param_tensor], lr=0.1)
     	# create default trainer for doctests
     	# as handlers could be attached to the trainer,
     	# each test must defined his own trainer using `.. testsetup:`
     	def get_default_trainer():
     	def train_step(engine, batch):
     	return 0.0
     	return Engine(train_step)
     	# create default model for doctests
     	default_model = nn.Sequential(OrderedDict([
     	('base', nn.Linear(4, 2)),
     	('fc', nn.Linear(2, 1))
     	]))
     	manual_seed(666)
     	"""

• 2. visible code is as above but if code is copied we add automatically all defaults (not good as a solution)
• 3. other ideas ?

[sdesrozis]

4. Don't hide any piece of code except import ? This is the more verbose approach but the less confusing.

[soma2000-lang]

@vfdev-5 Please assignme this.I would like to work on this

[vfdev-5]

@soma2000-lang sounds good, I can assign it to you. Which solution you plan to provide ?

[nishantb06]

@vfdev-5 , @sdesrozis I have a solution in mind - please let me know if this is feasible or not. For each metric we can envelop the code in a function and define the evaluator inside it. Now default_evaluator is a local variable and should not interfere with the one defined in conf.py. And then simply call the function to get the output -(Example shown below)

def func_name():
      
      def eval_step(engine, batch): 
            return batch 
  
      default_evaluator = Engine(eval_step) 

      metric = Accuracy()
      metric.attach(default_evaluator, "accuracy")
      y_true = torch.Tensor([1, 0, 1, 1, 0, 1])
      y_pred = torch.Tensor([1, 0, 1, 0, 1, 1])
      state = default_evaluator.run([[y_pred, y_true]])
      print(state.metrics["accuracy"])

func_name()

[sdesrozis]

@nishantb06 Thanks for this suggestion. You are right, we can exploit function's scope to avoid troubles. I would say that we could think about decorators too which provide good abstraction. Actually I don't have anything in mind but I will think about it.

Anyway, let's see whether any other ideas pop up.

[vfdev-5]

@nishantb06 thansk for a suggestion. If I understand correctly, you propose to show in examples func_name definition and func_name(). I think this would be very strange to see in the docs.

We can avoid using func_name and just define default_evaluator clearly in each docstring example. This is almost the same as the idea (4) from #2436 (comment)

[nishantb06]

@vfdev-5 I understand your point which is why I tried to implement idea (4) and built the docs locally. Following is a screenshot attached -

Do you think something like this would be suitable for a PR?

[sdesrozis]

And why not having a dedicated module to wrap snippets used in the doctest ?

from ignite.doctest import * # or whatever else

default_engine = get_engine_for_doctest()

...

It would be exploitable by users.

[vfdev-5]

And why not having a dedicated module to wrap snippets used in the doctest ?

I'm not sure that it is really necessary to introduce a dummy module into the library to solve a doctest representation issue.

---

Let's go with the idea (1) as we are passing almost 20 days to solve this issue :)

• to each docstring we add a text with a link on conf.py defaults.

Variant A:
Thus, for accuracy docs it will be:

    Examples:

        Default definitions in examples below can be found `here <https://github.com/pytorch/ignite/blob/111fc7a0c57bb4d96d94f300f8c2f188f7bda148/docs/source/conf.py#L334-L378>`_.

        Binary case

        .. testcode:: 1

            metric = Accuracy()
            metric.attach(default_evaluator, "accuracy")
            y_true = torch.Tensor([1, 0, 1, 1, 0, 1])
            y_pred = torch.Tensor([1, 0, 1, 0, 1, 1])
            state = default_evaluator.run([[y_pred, y_true]])
            print(state.metrics["accuracy"])

        .. testoutput:: 1

            0.6666...

Variant B. Ideal solution would be to find how to make this code foldable (IMO, it shouldn't be easy). So that once rendered we could have something like:

Examples

Default definitions in examples below can be found here.

from collections import OrderedDict
import torch
from torch import nn, optim
from ignite.engine import *
from ignite.handlers import *
from ignite.metrics import *
from ignite.utils import *
from ignite.contrib.metrics.regression import *
from ignite.contrib.metrics import *
# create default evaluator for doctests
def eval_step(engine, batch):
    return batch
default_evaluator = Engine(eval_step)
# create default optimizer for doctests
param_tensor = torch.zeros([1], requires_grad=True)
default_optimizer = torch.optim.SGD([param_tensor], lr=0.1)
# create default trainer for doctests
# as handlers could be attached to the trainer,
# each test must defined his own trainer using `.. testsetup:`
def get_default_trainer():
    def train_step(engine, batch):
        return 0.0
    return Engine(train_step)
# create default model for doctests
default_model = nn.Sequential(OrderedDict([
    ('base', nn.Linear(4, 2)),
    ('fc', nn.Linear(2, 1))
]))
manual_seed(666)

Binary case

metric = Accuracy()
metric.attach(default_evaluator, "accuracy")
y_true = torch.Tensor([1, 0, 1, 1, 0, 1])
y_pred = torch.Tensor([1, 0, 1, 0, 1, 1])
state = default_evaluator.run([[y_pred, y_true]])
print(state.metrics["accuracy"])

Foldable stuff in GH and HTML i suppose can be done as

<details>
<summary>
Default definitions in examples below can be found here.
</summary>

python code

</details>

The main question is how to make that in sphinx. Ideally, we do not want to have all declarations to be present at each docstring, it would be better to predefine them somehow and while rendering the docs, add to the place where they should be...

[DevPranjal]

Hi @vfdev-5 @sdesrozis, I was trying a few options to resolve this problem

Ideal solution would be to find how to make this code foldable (IMO, it shouldn't be easy). So that once rendered we could have something like

I ended up with this using the sphinx_togglebutton extension

screen-capture.mp4

Currently, I am using code docstring below:

.. toggle::
   .. testcode::
        <defaults>
...

But as mentioned,

Ideally, we do not want to have all declarations to be present at each docstring, it would be better to predefine them somehow and while rendering the docs, add to the place where they should be

I tried the below, but was not able to get it to work:

.. toggle::        
     .. include:: defaults.rst
...