y_pred when metric is roc_auc

[mglowacki100]

NannyML is interesting tool. My use case is scoring not strictly binary prediction. I've a question regarding relevant example:
https://nannyml.readthedocs.io/en/stable/examples/california_housing.html#performance-estimation namely it seems that y_pred is required, but for roc_auc calculation we only need y_pred_proba. So is y_pred really required or could it be filled by random values?

[nnansters]

Hey @mglowacki100 , good question.

You are completely correct, roc_auc only requires the y_pred_proba values. Asking for y_pred is a consequence of us allowing multiple metrics to be calculated at once. Hence we'll need all the information those other metrics might require, such as y_pred. The way we've implemented it at this time is a bit lacking since we do expect a y_pred column to be present even when you're only estimating roc_auc. It will not be used, however, so you can pass a random values column there when only estimating roc_auc.

We are thinking about splitting up the metrics into separate runnable pieces, which would allow to only ask for parameters we actually need.

Hope that helps, let me know if you need anything else!