-
Notifications
You must be signed in to change notification settings - Fork 3k
Add ASR task for SUPERB #2619
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ASR task for SUPERB #2619
Conversation
|
Wait until #2620 is merged before pushing the README tags in this PR |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#2620 is already merged into master. You can merge master into this branch.
albertvillanova
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
One question: aren't you adding task_templates to the _info method (and to the dataset_infos.json?
great catch! i've now added the asr task template (along with a mapping from superb task -> template) and updated the |
albertvillanova
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good!
I have a suggested refactoring... Tell me what you think! :)
Co-authored-by: Albert Villanova del Moral <[email protected]>
your approach is much more elegant - i've included your suggestions 🙏 |
albertvillanova
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you.
This PR starts building up the SUPERB benchmark by including the ASR task as described in the SUPERB paper and
s3prlinstructions.Usage:
I've used the GLUE benchmark as a guide for filling out the README.
To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.
Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)