Skip to content

Adding Acceptability judgment and SentEval tasks#1271

Merged
zphang merged 6 commits intonyu-mll:masterfrom
zphang:acceptability
Feb 2, 2021
Merged

Adding Acceptability judgment and SentEval tasks#1271
zphang merged 6 commits intonyu-mll:masterfrom
zphang:acceptability

Conversation

@zphang
Copy link
Collaborator

@zphang zphang commented Feb 1, 2021

Results using roberta-base / epochs=40, batch_size=16, lr=1e-5

  • acceptability_coord: 0.913
  • acceptability_definiteness: 0.942
  • acceptability_eos: 0.854
  • acceptability_whwords: 0.900

Results using roberta-base / epochs=3, batch_size=16, lr=1e-5

  • senteval_bigram_shift: 0.973
  • senteval_coordination_inversion: 0.857
  • senteval_obj_number: 0.965
  • senteval_odd_man_out: 0.743
  • senteval_past_present: 0.928
  • senteval_sentence_length: 0.876
  • senteval_subj_number: 0.954
  • senteval_top_constituents: 0.910
  • senteval_tree_depth: 0.681
  • senteval_word_content: 0.995

@codecov
Copy link

codecov bot commented Feb 1, 2021

Codecov Report

Merging #1271 (f083e81) into master (56ceae5) will increase coverage by 1.25%.
The diff coverage is 86.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1271      +/-   ##
==========================================
+ Coverage   47.21%   48.47%   +1.25%     
==========================================
  Files         149      163      +14     
  Lines       10877    11214     +337     
==========================================
+ Hits         5136     5436     +300     
- Misses       5741     5778      +37     
Impacted Files Coverage Δ
jiant/scripts/download_data/constants.py 100.00% <ø> (ø)
jiant/scripts/download_data/runscript.py 29.41% <0.00%> (ø)
jiant/tasks/evaluate/core.py 32.89% <ø> (ø)
...t/scripts/download_data/dl_datasets/files_tasks.py 7.69% <10.52%> (+0.12%) ⬆️
jiant/tasks/lib/senteval/base.py 77.96% <38.46%> (ø)
jiant/tasks/lib/acceptability_judgement/base.py 73.84% <73.84%> (ø)
jiant/tasks/lib/senteval/bigram_shift.py 95.65% <95.65%> (ø)
jiant/tasks/lib/senteval/coordination_inversion.py 95.65% <95.65%> (ø)
jiant/tasks/lib/senteval/obj_number.py 95.65% <95.65%> (ø)
jiant/tasks/lib/senteval/odd_man_out.py 95.65% <95.65%> (ø)
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 56ceae5...f083e81. Read the comment docs.

@zphang zphang merged commit 84f2f5a into nyu-mll:master Feb 2, 2021
@zphang zphang deleted the acceptability branch February 2, 2021 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants