Doctest for PiecewiseLinear lr scheduler

[DevPranjal]

Addresses #2266

Description: Added doctest for PiecewiseLinear lr scheduler

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[DevPranjal]

I'm new to writing doctests and felt this doctest is pretty large, but was not able to think of a smaller and cleaner test.
Could anyone please suggest something better? @vfdev-5 @sdesrozis @ydcjeff @Ishan-Kumar2

Also, I have added a few functions that might be redundant in the future doctests to the config (as suggested here)

[sdesrozis]

@DevPranjal Thank you for your work.

I'm wondering if having global but hidden methods like generate_data_with_noise is a good option. IMO the examples must work as copy/past. What do you think ?

[DevPranjal]

I am with the same opinion here, maybe we can think of a cleaner test? Or does this work well (with the methods defined within the example)?

[sdesrozis]

I am with the same opinion here, maybe we can think of a cleaner test? Or does this work well (with the methods defined within the example)?

Maybe you could use a simple model

model = nn.Linear(1, 1)

and define a dataset as a list of scalars. It would worth adding outputs, for instance the learning rates.

[DevPranjal]

Hello @sdesrozis, I have made the changes... Could you please review them? Also about the test output... I thought of adding the weights and biases of the network (which were supposed to be close to 1 and -1 respectively according to the example I added) but that would not be representative of the test.

It would worth adding outputs, for instance the learning rates.

Could you also please elaborate on this? I am a bit confused here.

Thanks for the patience :)

[sdesrozis]

@DevPranjal Sorry for the late answer, I'm quite busy at the moment. Thus, thank you for your patience !

I will review and provide you a more detailed explanation asap.

[sdesrozis]

@DevPranjal Very sorry for the delay... I was so busy :(

Here is a code that you could use to add some outputs

import torch

from ignite.engine import Events, Engine
from ignite.handlers.param_scheduler import PiecewiseLinear

tensor = torch.zeros([1], requires_grad=True)
optimizer = torch.optim.SGD([tensor], lr=0)

milestones_values = [(1, 1.0), (3, 0.8), (5, 0.2)]
scheduler = PiecewiseLinear(optimizer, "lr", milestones_values=milestones_values)
        
def save_lr(engine):
    lrs.append(optimizer.param_groups[0]["lr"])

trainer = Engine(lambda engine, batch: None)
trainer.add_event_handler(Events.ITERATION_COMPLETED, scheduler)
trainer.add_event_handler(Events.ITERATION_COMPLETED, save_lr)

lrs = []
trainer.run([0] * 6, max_epochs=1)
print(lrs)
# >>> [1.0, 1.0, 0.9, 0.8, 0.5, 0.2]

lrs = []
trainer.run([0], max_epochs=1)
print(lrs)
# >>> [0.2]

[DevPranjal]

Thanks a lot for this snippet! I am really impressed by the ignite API here, want to learn more such use cases in the future :)

[sdesrozis]

Thanks a lot for this snippet! I am really impressed by the ignite API here, want to learn more such use cases in the future :)

This is the right place to be :)

[sdesrozis]

@DevPranjal Are you still working on this ? Thanks !

[DevPranjal]

Sorry for the delay... We have our finals coming up at college and hence I have been busy this week. I will make the changes as soon as possible.

[DevPranjal]

@sdesrozis, I have added the suggested code, please review. Thanks!

[sdesrozis]

Let me a little time to merge a PR I have done on this topic. I have introduced some features we could use in this PR. I keep you in touch.

[sdesrozis]

@DevPranjal Could you update this PR accordingly to #2327 ? Thanks !!

[DevPranjal]

Yes, I have been following #2327... And now that it is merged, I will update this one by tomorrow :)

[DevPranjal]

ignite/ignite/handlers/param_scheduler.py

Lines 978 to 988 in 60e2af4

	.. code-block:: python
	scheduler = PiecewiseLinear(optimizer, "lr",
	milestones_values=[(10, 0.5), (20, 0.45), (21, 0.3), (30, 0.1), (40, 0.1)])
	# Attach to the trainer
	trainer.add_event_handler(Events.ITERATION_STARTED, scheduler)
	#
	# Sets the learning rate to 0.5 over the first 10 iterations, then decreases linearly from 0.5 to 0.45 between
	# 10th and 20th iterations. Next there is a jump to 0.3 at the 21st iteration and LR decreases linearly
	# from 0.3 to 0.1 between 21st and 30th iterations and remains 0.1 until the end of the iterations.
	#

Should I remove this snippet? @sdesrozis

Also, sorry for the mess in commit history

[sdesrozis]

Your test should reflect the current test, even adding a few comments to be very explicit about what is done.

Please look #2327 with attention, it's mostly the same.

[DevPranjal]

@sdesrozis Gentle ping, in case you missed the last commit. I have added the comments as suggested

[vfdev-5]

@ydcjeff can you please also review this PR ?

[sdesrozis]

@DevPranjal Thank you ! LGTM !