Keeping track of the best model checkpoint

[miguelvr]

🚀 Feature

There is currently no way of keeping track of the best checkpoint in classy vision, with a focus on a specific meter.

Motivation / Pitch

Knowing which checkpoint performs better against a validation set is a common practice in ML, it makes sense to automate this process for the user.

Alternatives

The only way I've managed to do it so far was by analyzing the contents of the saved checkpoints for a specific meter or by manually looking at the training logs. Both of these are very tedious processes.

Additional Context

I think this can be implemented by a ClassyHook that watches the values of a user-specified meter and saves the best phase with a given strategy (max/min). The downside I see is that in order to be accurate, this needs to be synced across all the workers in the distributed case.

Results can then be logged or dumped to a file in the master worker.

[mannatsingh]

This is something we do plan on supporting, thanks for filing a feature request, @miguelvr !

The approach I had in mind was to just support this feature within the CheckpointHook, so that it would save the "best" checkpoint with a special filename.

[miguelvr]

I don't mind doing a PR for that as well. I'm using the framework for training a lot of models at the moment and this is something that'll have to do anyway.