Extend Scorer #1475
Extend Scorer #1475
Conversation
Codecov Report
@@ Coverage Diff @@
## development #1475 +/- ##
===============================================
- Coverage 84.33% 84.27% -0.06%
===============================================
Files 151 151
Lines 11437 11488 +51
Branches 1984 1994 +10
===============================================
+ Hits 9645 9682 +37
- Misses 1265 1275 +10
- Partials 527 531 +4 |
mfeurer
left a comment
mfeurer
left a comment
There was a problem hiding this comment.
This looks good in general, but it would be great if this would be accompanied by some unit tests, for example for the train evaluator and the metrics module.
| def make_scorer( | ||
| name: str, | ||
| score_func: Callable, | ||
| *, |
| def metric_which_needs_x(solution, prediction, x_data, extra_argument): | ||
| # custom function defining accuracy | ||
| assert x_data is not None | ||
| return np.mean(solution[extra_argument, :] == prediction[extra_argument, :]) |
There was a problem hiding this comment.
Can we illustrate with some arbitrary use of x_data for the example, i.e. make rows with NaNs count for twice as much?
There was a problem hiding this comment.
I updated the example with a more meaningful metric
mfeurer
left a comment
mfeurer
left a comment
There was a problem hiding this comment.
This looks all good, a new unit test in the file test_train_evaluator.py, which ensures that the data is correctly emitted to disk, would be great to have, though.
|
I will run the benchmark on this, it's hard to implement this feature into the |
Co-authored-by: Matthias Feurer <[email protected]>
* FIX minor * update submodule * update submodule * ADD pass xdata to metric * update submodule * Fix tests * update submodule * ADD example * UPDATE example * ADD extra method to concat data * RM note * Fix minor * ADD more types for xdata * FIX unittests * FIX variable naming bug * change varible name; fix docstring * Rename variable * FIX example * Update examples/40_advanced/example_metrics.py Co-authored-by: Matthias Feurer <[email protected]> Co-authored-by: Matthias Feurer <[email protected]>
This PR enables ASKL to work with a wider range of metrics and scorers. Concretely, it does the following: