First draft of multi-objective optimization

[mfeurer]

This is the very first implementation of multi-objective optimization in Auto-sklearn, solving issue #1317. It allows users passing in a list of metrics instead of a single metric. Auto-sklearn then solves a multi-objective optimization problem by using SMAC's new multi-objective capabilities.

This PR has several known limitation, some of which can/will be fixed in a future PR, and some for which it is unclear how to approach them in the first place:

1. The ensemble optimizes only the first metric:
   This will be addressed in a future PR.
2. _load_best_individual_model() returns the best model according to the first metric:
   It is unclear how to make this multi-objective as this is a clear fallback solution. My only two suggestions would be 1) to always return the first metric and document that the first metric is the tie breaking metric (currently implemented), or 2) use the Copula mixing approach demonstrated in Figure 6 of http://proceedings.mlr.press/v119/salinas20a/salinas20a.pdf to return a scalarized solution. However, this can lead to unpredictable selections.
3. score():
   This function suffers from the same issue as load_best_individual_model as there is no clear definition which model is the best.
4. _get_runhistory_models_performance():
   This function suffers from the same issue as load_best_individual_model as there is no clear definition which model is the best.
5. sprint_statistics()
   This function suffers from the same issue as load_best_individual_model as there is no clear definition which model is the best.
6. refit()
   This function suffers from the same issue as load_best_individual_model as there is no clear definition which model is the best.
7. Leaderboard:
   • The first metric is currently reported as "cost" -> If there is more than one metric, we report them as cost_0, cost_1, etc
   • The entries will be sorted by the first metric by default, with all additional metrics being tie breakers
   • The current sorting ["cost", "rank"] appears to me to be the same as would be sorting by ["cost"] as "rank" should have the same order. I therefore propose to use the "model_id" as a secondary metric here
8. cv_results_
   Updated to follow the output of scikit-learn, see Attributes in here
9. Accessing the Pareto front of individual models and ensembles
   It is so far completely unclear how to achieve this. Picking a model from the Pareto front would make Auto-sklearn non-interactive. Therefore, it might be a good idea to add a function to return all ensembles on the Pareto front as "raw" ensemble objects that can be further used by the user, or to one after the other load different models as the main model.

TODOs:

• Generalize code and wrap metric always in a list to have internal methods always work on dicts or lists of metrics (reduces a lot of if-statements)
• write unit tests
• pass a metric to meta-learning
• fix unit tests
• make the following functions multi-objective:
  • load_best_individual_model
  • leaderboard
  • score
  • _get_runhistory_models_performance
  • cv_results
  • sprint_statistics
  • refit
• update examples
• add multi-objective example
• sub-sampling (i.e. successive halving)

[eddiebergman]

Don't need to change, just if you see it and get a chance :) I think it looks a little Cleaner with as little square brackets [] from Optional[] and Union[] as possible but I'd merge without it. It also translates better with types in sphinx docs if you use the same notation in the docstrings.

[eddiebergman]

Not neccessary, just something to know Optional[X] == Union[X, None] == X | None
i..e you could write Scorer | Sequence[Scorer] | None = None

[mfeurer]

That's look nice, will do.

[eddiebergman]

Like wise here, Union[X | Y] == X | Y, the | essentially is just the infix operator for Union in the same way you have + instead of add(x, y).

i.e. metric: Scorer | Sequence[Scorer]

[mfeurer]

Thanks for catching.

[eddiebergman]

Sorry, more files I didn't see. I'm starting to wonder if it makes sense to have something like a MetricGroup class? A lot of the code changes seem to just be handling that case of 1, many or None metrics.

[eddiebergman]

I assume this is a point of API conflict? It would be good to know about all the metrics for a model but at the end of the day, we currently only support one and so we choose the first?

[mfeurer]

Yes, it would be good to know about all the metrics. I will look into returning multiple metrics here (should be possible).

[mfeurer]

Please see my comment wrt this in the PR comment at the top.

[codecov]

Codecov Report

Merging #1455 (d432e07) into development (daa9ad6) will increase coverage by 0.00%.
The diff coverage is 81.95%.

@@              Coverage Diff              @@
##           development    #1455    +/-   ##
=============================================
  Coverage        84.31%   84.32%            
=============================================
  Files              147      147            
  Lines            11284    11397   +113     
  Branches          1934     1986    +52     
=============================================
+ Hits              9514     9610    +96     
- Misses            1256     1263     +7     
- Partials           514      524    +10     

[mfeurer]

Alright, I think the functionality of this PR is complete for now. I will add unit tests after another round of feedback.

[eddiebergman]

Replies to the list of items in the PR first.

1. The ensemble optimizes only the first metric:
   This will be addressed in a future PR.

Sounds good, we need to know what to do with metrics in the ensemble first.

2. _load_best_individual_model() returns the best model according to the first metric:
   It is unclear how to make this multi-objective as this is a clear fallback solution. My only two suggestions would be 1) to always return the first metric and document that the first metric is the tie breaking metric (currently implemented), or 2) use the Copula mixing approach demonstrated in Figure 6 of http://proceedings.mlr.press/v119/salinas20a/salinas20a.pdf to return a scalarized solution. However, this can lead to unpredictable selection

An extended version of the first solution, (cost_0, cost_1, ...) <= (cost_0, cost_1, ...) such that if cost_0 is equal, it will then compare the second one and so on. This means the order of the metrics is the order of comparison and means they must be preserved throughout.

3. 4. 5. 6. ...

Same answer, without a method to present choices we have the requirement to select for the user. I would go with the above solution

7. Leaderboard

Seems good, we could even extract the metrics name through f.__name__ but this means programatically using leaderboard becomes difficult.

8. cv_results_

Sure

9. Ensembles

In theory there's nothing that prevents us from making a selection at predict as we already do, using the above methods to force a selection. For giving a choice, yes we need to somehow wrap up the ensembles with their scores. This below solution would give a lot of choice but maybe there's a better way to present the info?

[
    ((cost_0, cost_1, ...), ens0),
    ((cost_0, cost_1, ...), ens1),
    ((cost_0, cost_1, ...), ens2),
]

I'll do a review now

[eddiebergman]

[Non-Blocking, Q]
Using the same metric twice seems like it should raise an error to me. Given that the behavior is no error raised, the output is good though. Should we keep it as is?

[mfeurer]

Do you mean we should allow the same metric to be present in both metrics and scoring_functions, but not the same metric twice in one of them?

[eddiebergman]

I guess so, I didn't really think about the same one twice, once in metrics and once in scoring_functions.

What you stated seems reasonable to me but not a cause to block the PR if it's difficult. I can't imagine a scenario where you would want the same metric, i.e. acc twice in metrics but I could imagine a scenario where you have acc in metrics and acc in scoring_functions.

The scenario where I see this being used is purely to make getting out scores for autosklearn to be done in one place, i.e. you specify acc and metric_z for the optimization with metrics and you specify acc, balanced_acc and f1_score for the scoring_functions when you later want to evaluate the autosklearn run.