Skip to content

[PR #1537 Patch 4] Separate users' evaluation metric and the internal validation metric#1571

Merged
sxjscience merged 9 commits intoautogluon:masterfrom
zhiqiangdon:mm
Feb 25, 2022
Merged

[PR #1537 Patch 4] Separate users' evaluation metric and the internal validation metric#1571
sxjscience merged 9 commits intoautogluon:masterfrom
zhiqiangdon:mm

Conversation

@zhiqiangdon
Copy link
Copy Markdown
Contributor

This PR is to separate users' provided evaluation metric and the internal validation metric. If a user provides metric r2, we will use rmse as the internal validation metric since torchmetrics.R2Score encounters errors for batch size 1.

  1. Always keep the evaluation metric provided by users. It can be used in .evaluate().
  2. Use a separate internal validation metric, which may be different from the provided evaluation metric.
  3. Rename the names of evaluation and validation metrics so that their names are both distinguishable and accurate, especially in the context of deep learning.
  4. Clean some unused log keys.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Comment thread text/src/autogluon/text/automm/predictor.py Outdated
@szha
Copy link
Copy Markdown

szha commented Feb 25, 2022

Job PR-1571-2 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1571/2/index.html

@szha
Copy link
Copy Markdown

szha commented Feb 25, 2022

Job PR-1571-3 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1571/3/index.html

@sxjscience sxjscience merged commit c9c33c9 into autogluon:master Feb 25, 2022
@zhiqiangdon zhiqiangdon deleted the mm branch February 26, 2022 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants