Skip to content

Change the _maxCalibrationExamples default on CalibratorUtils #5415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antoniovs1029 merged 2 commits into from
Sep 30, 2020
Merged

Change the _maxCalibrationExamples default on CalibratorUtils #5415

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0182155
Change the _maxCalibrationExamples default
antoniovs1029 Sep 30, 2020
d08a3d8
Improving comments
antoniovs1029 Sep 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion src/Microsoft.ML.Data/Prediction/Calibrator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -838,7 +838,9 @@ internal static object Create(IHostEnvironment env, ModelLoadContext ctx, object
internal static class CalibratorUtils
{
// maximum number of rows passed to the calibrator.
private const int _maxCalibrationExamples = 1000000;
// if 0, we'll actually look through the whole dataset to
// when training the calibrator
Copy link
Contributor

@harishsk harishsk Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Earlier, you had explained to me that if this value is zero, we would look through the whole dataset, but still only use a million rows (randomly selected) for calibration. Is that correct?

Can you please clarify the exact behavior in comments?

Copy link
Contributor Author

@antoniovs1029 antoniovs1029 Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is correct for PlattCalibrator, but depending of the calibrator the behavior is different. If this is 0, the only thing that does happen for all the calibrators is that we'll look through all the dataset. I explained this further on the other comment I left, so I don't think it's necessary to clarify it more in here.

private const int _maxCalibrationExamples = 0;

private static bool NeedCalibration(IHostEnvironment env, IChannel ch, ICalibratorTrainer calibrator,
ITrainer trainer, IPredictor predictor, RoleMappedSchema schema)
Expand Down Expand Up @@ -988,6 +990,10 @@ public static ICalibrator TrainCalibrator(IHostEnvironment env, IChannel ch, ICa
caliTrainer.ProcessTrainingExample(score, label > 0, weight);

if (maxRows > 0 && ++num >= maxRows)
// If maxRows was 0, we'll process all of the rows in the dataset
// Notice that depending of the calibrator, "processing" means
// only using N random rows of the ones that where processed
Copy link
Contributor

@harishsk harishsk Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, typo: "depending on the calibrator"

// to actually train the calibrator.
break;
}
}
Expand Down