Skip to content

Adding ROPES, RACE tasks#1234

Merged
jeswan merged 35 commits intonyu-mll:masterfrom
zphang:add_test_sets2
Jan 1, 2021
Merged

Adding ROPES, RACE tasks#1234
jeswan merged 35 commits intonyu-mll:masterfrom
zphang:add_test_sets2

Conversation

@zphang
Copy link
Collaborator

@zphang zphang commented Nov 12, 2020

Ropes

  • RoBERTa-base: EM=45.7, F1=53.8
  • RoBERTa-large: EM=55.9, F1=62.0

RACE

Results forthcoming.

Other Notes

  • RACE isn't publicly downloadable
  • Ropes requires tweaking the SQuAD featurization itself because of its special format (the answer span can be within the question itself)

@codecov
Copy link

codecov bot commented Nov 12, 2020

Codecov Report

Merging #1234 (80bd7b8) into master (e5fbea4) will decrease coverage by 0.55%.
The diff coverage is 29.41%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1234      +/-   ##
==========================================
- Coverage   56.43%   55.88%   -0.56%     
==========================================
  Files         147      149       +2     
  Lines       10598    10819     +221     
==========================================
+ Hits         5981     6046      +65     
- Misses       4617     4773     +156     
Impacted Files Coverage Δ
jiant/scripts/download_data/constants.py 100.00% <ø> (ø)
...pts/download_data/dl_datasets/hf_datasets_tasks.py 100.00% <ø> (ø)
jiant/tasks/evaluate/core.py 36.17% <ø> (ø)
...t/scripts/download_data/dl_datasets/files_tasks.py 7.54% <9.09%> (+0.04%) ⬆️
jiant/tasks/lib/ropes.py 20.39% <20.39%> (ø)
jiant/tasks/lib/race.py 55.35% <55.35%> (ø)
jiant/tasks/retrieval.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5fbea4...80bd7b8. Read the comment docs.

@jeswan
Copy link
Collaborator

jeswan commented Nov 24, 2020

@zphang Do you have RACE results you can post here?

@HarshTrivedi
Copy link
Collaborator

@zphang @jeswan RACE is downloadable (at least now) via HF datasets. I've added it in a different branch.

@zphang
Copy link
Collaborator Author

zphang commented Dec 31, 2020

Following up on Harsh's update, RACE is now downloadable, and the PR has been updated to reflect this.

Results (val):

  • RoBERTa-base
    • RACE (middle): 68.2
    • RACE (high): 64.8
    • RACE (all): 66.4
  • RoBERTa-large
    • RACE (middle): 85.7
    • RACE (high): 79.7
    • RACE (all): 79.7

Compare to published results, on the test set:

  • RACE (middle): 86.5
  • RACE (high): 81.3
  • RACE (all): 83.2

Both Harsh and I encountered some training instability (requires multiple restarts).

@jeswan jeswan merged commit 1ab34a4 into nyu-mll:master Jan 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants