v0.8.6

What's new

• Pinned datasets, tokenizers, pyarrow versions.

Commits

92e17d1 Pinning dependencies to known good versions. (#15)
73e9db1 Add release process instructions

v0.8.5

What's new

Removed 👋

• Remove sklearn and numpy as depedencies. Manual implementation of F1 score.

Commits

ed1b21b remove sklearn/numpy deps (#14)

v0.8.4

What's new

• Add BOS token, when the BOS token exists in the tokenizer

Commits

v0.8.3

What's new

• Fix speed problem for BPB/RC tasks

Commits

v0.8.2

What's new

• Add few-shot HumanEval
• Fix prompting setup for MT MBPP. For more details, see allenai/oe-eval-internal#489

Commits

d3d2448 add few show he
696f7ae update fixed mbpp (#10)
8ead415 detach bpb tensor (prevents warning)

v0.8.1

What's new

• Add mt MBPP and Minerva Math 500

Commits

v0.8.0

What's new

• Add fast MCQA

Commits

b635bb9 Add fast MCQA (#8)

v0.7.2

What's new

• Add basic skills evals

Commits

v0.7.1

What's new

• Fix normalization to match the OLMES standard

Commits

7363946 fix bias in eval per-char and per-byte normalization (#6)

v0.7.0

What's new

• Add in-loop GSM, Minerva, MBPP, HumanEval

Commits

2f01ec8 Add in-loop gen tasks (#5)