[Eval] Evaluation of abstract reasoning capabilities of language models with the Raven Matrices test (openai#1078)

# Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, __failure to follow
the guidelines below will result in the PR being closed automatically__.
Note that even if the criteria are met, that does not guarantee the PR
will be merged nor GPT-4 access granted. 🚨

__PLEASE READ THIS__:

In order for a PR to be merged, it must fail on GPT-4. We are aware that
right now, users do not have access, so you will not be able to tell if
the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep
in mind as we run the eval, if GPT-4 gets higher than 90% on the eval,
we will likely reject since GPT-4 is already capable of completing the
task.

We plan to roll out a way for users submitting evals to see the eval
performance on GPT-4 soon. Stay tuned! Until then, you will not be able
to see the eval performance on GPT-4. **Starting April 10, the minimum
eval count is 15 samples, we hope this makes it easier to create and
contribute evals.**

Also, pelase note that we're using **Git LFS** for storing the JSON
files, so please make sure that you move the JSON file to Git LFS before
submitting a PR. Details on how to use Git LFS are available
[here](https://git-lfs.com).

## Eval details 📑
### Eval name
Raven Matrices

### Eval description

This benchmark evaluates the ability of a language to perform abstract
reasoning using a text-based version of the Raven Matrices test. The
task consist of finding a pattern from a set of choices that completes a
sequence of eight previous samples. We provide various types of
matrices, either under natural language or symbolic formats, with
multiple-choices and open-ended settings.

### What makes this a useful eval?

Abstract reasoning is an useful task to evaluate the ability of a
language model to extract a pattern from few examples. The abstract
nature of the pattern requires the model to find the most generic
pattern, allowing to test the generalization capacities of language
models. Abstract reasoning is a task on which current language models do
not perform well, although they have been under-evaluated in the
research.

## Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general,
we are seeking cases where the model does not do a good job despite
being capable of generating a good response (note that there are some
things large language models cannot do, so those would not make good
evals).

Your eval should be:

- [X] Thematically consistent: The eval should be thematically
consistent. We'd like to see a number of prompts all demonstrating some
particular failure mode. For example, we can create an eval on cases
where the model fails to reason about the physical world.
- [X] Contains failures where a human can do the task, but either GPT-4
or GPT-3.5-Turbo could not.
- [X] Includes good signal around what is the right behavior. This means
either a correct answer for `Basic` evals or the `Fact` Model-graded
eval, or an exhaustive rubric for evaluating answers for the `Criteria`
Model-graded eval.
- [X] **Include at least 15 high quality examples.**

If there is anything else that makes your eval worth including, please
document it below.

### Unique eval value

Our eval contains an extensive list of high-quality samples from a
challenging and under-evaluated task, with several levels of difficulty
and different formats.

## Eval structure 🏗️

Your eval should
- [X] Check that your data is in `evals/registry/data/{name}`
- [X] Check that your yaml is registered at
`evals/registry/evals/{name}.yaml`
- [X] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing
eval classes. You may still write custom eval classes for your own
cases, and we may consider merging them in the future.)

## Final checklist 👀

### Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic
and data under the same MIT license as this repository. You must have
adequate rights to upload any data used in an Eval. OpenAI reserves the
right to use this data in future service improvements to our product.
Contributions to OpenAI Evals will be subject to our usual Usage
Policies (https://platform.openai.com/docs/usage-policies).

- [X] I agree that my submission will be made available under an MIT
license and complies with OpenAI's usage policies.

### Email address validation

If your submission is accepted, we will be granting GPT-4 access to a
limited number of contributors. Access will be given to the email
address associated with the merged pull request.

- [X] I acknowledge that GPT-4 access will only be granted, if
applicable, to the email address used for my merged pull request.

### Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission,
help improve our models, and gain access to GPT-4. However, due to the
requirements mentioned above and high volume of submissions, we will not
be able to accept all submissions and thus not grant everyone who opens
a PR GPT-4 access. We know this is disappointing, but we hope to set the
right expectation before you open this PR.

- [X] I understand that opening a PR, even if it meets the requirements
above, does not guarantee the PR will be merged nor GPT-4 access
granted.

### Submit eval

- [X] I have filled out all required fields of this form
- [X] I have used **Git LFS** for the Eval JSON data
- [ ] (Ignore if not submitting code) I have run `pip install
pre-commit; pre-commit install` and have verified that `black`, `isort`,
and `autoflake` are running when I commit and push

Failure to fill out all required fields will result in the PR being
closed.

### Eval JSON data 

Since we are using Git LFS, we are asking eval submitters to add in as
many Eval Samples (at least 5) from their contribution here:

<details>
  <summary>View evals in JSON</summary>

  ### Eval
  ```jsonl
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Pick the letter in front of the correct pattern
that logically follows in the sequence from the answer set. Patterns in
the sequence are preceded by a number from 1 to 8. Patterns in the
answer set are preceded by a letter from A to H. Only return the letter
in front of the correct pattern."}, {"role": "user", "content": "1. On
an image, a large orange circle rotated at 90 degrees. "}, {"role":
"user", "content": "2. On an image, a giant orange pentagon rotated at
90 degrees. "}, {"role": "user", "content": "3. On an image, a small red
triangle rotated at 90 degrees. "}, {"role": "user", "content": "4. On
an image, a small orange circle rotated at 135 degrees. "}, {"role":
"user", "content": "5. On an image, a large orange pentagon rotated at
135 degrees. "}, {"role": "user", "content": "6. On an image, a giant
red triangle rotated at 135 degrees. "}, {"role": "user", "content": "7.
On an image, a giant red circle rotated at -45 degrees. "}, {"role":
"user", "content": "8. On an image, a small red pentagon rotated at -45
degrees. "}, {"role": "user", "content": "A. On an image, a large red
triangle rotated at -45 degrees. "}, {"role": "user", "content": "B. On
an image, a large red circle rotated at -45 degrees. "}, {"role":
"user", "content": "C. On an image, a large red hexagon rotated at -45
degrees. "}, {"role": "user", "content": "D. On an image, a medium red
triangle rotated at -45 degrees. "}, {"role": "user", "content": "E. On
an image, a large orange triangle rotated at -45 degrees. "}, {"role":
"user", "content": "F. On an image, a large red pentagon rotated at -45
degrees. "}, {"role": "user", "content": "G. On an image, a large pink
triangle rotated at -45 degrees. "}, {"role": "user", "content": "H. On
an image, a large lime triangle rotated at -45 degrees. "}, {"role":
"user", "content": "The answer is "}], "ideal": "A"}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Pick the letter in front of the correct pattern
that logically follows in the sequence from the answer set. Patterns in
the sequence are preceded by a number from 1 to 8. Patterns in the
answer set are preceded by a letter from A to H. Only return the letter
in front of the correct pattern."}, {"role": "user", "content": "1. On
an image, a huge purple triangle rotated at 180 degrees in the bottom
right, a small purple triangle rotated at -45 degrees in the top left, a
large purple triangle rotated at 45 degrees in the bottom left. "},
{"role": "user", "content": "2. On an image, a huge pink circle rotated
at 180 degrees in the bottom right, a small pink circle rotated at -45
degrees in the bottom left, a large pink circle rotated at 45 degrees in
the top right. "}, {"role": "user", "content": "3. On an image, a huge
white square rotated at 180 degrees in the bottom right, a small white
square rotated at -45 degrees in the top left, a large white square
rotated at 45 degrees in the top right. "}, {"role": "user", "content":
"4. On an image, a large lime circle rotated at 0 degrees in the bottom
right, a tiny lime circle rotated at 90 degrees in the top left, a giant
lime circle rotated at -45 degrees in the top right. "}, {"role":
"user", "content": "5. On an image, a large green square rotated at 0
degrees in the bottom right, a tiny green square rotated at 90 degrees
in the top left, a giant green square rotated at -45 degrees in the
bottom left. "}, {"role": "user", "content": "6. On an image, a large
cyan triangle rotated at 0 degrees in the bottom right, a tiny cyan
triangle rotated at 90 degrees in the bottom left, a giant cyan triangle
rotated at -45 degrees in the top right. "}, {"role": "user", "content":
"7. On an image, a huge lime square rotated at 135 degrees in the bottom
right, a tiny lime square rotated at 0 degrees in the bottom left, a
tiny lime square rotated at 135 degrees in the top right. "}, {"role":
"user", "content": "8. On an image, a huge green triangle rotated at 135
degrees in the bottom right, a tiny green triangle rotated at 0 degrees
in the top left, a tiny green triangle rotated at 135 degrees in the top
right. "}, {"role": "user", "content": "A. On an image, a huge cyan
pentagon rotated at 135 degrees in the bottom right, a tiny cyan
triangle rotated at 0 degrees in the top left, a tiny cyan triangle
rotated at 135 degrees in the bottom left. "}, {"role": "user",
"content": "B. On an image, a huge cyan circle rotated at 135 degrees in
the top right, a tiny cyan circle rotated at 0 degrees in the bottom
left, a tiny cyan circle rotated at 135 degrees in the top left. "},
{"role": "user", "content": "C. On an image, a huge cyan square rotated
at 135 degrees in the bottom right, a tiny cyan hexagon rotated at 0
degrees in the top left, a tiny cyan hexagon rotated at 135 degrees in
the bottom left. "}, {"role": "user", "content": "D. On an image, a huge
cyan circle rotated at 135 degrees in the top left, a tiny cyan circle
rotated at 0 degrees in the bottom right, a tiny cyan circle rotated at
135 degrees in the top right. "}, {"role": "user", "content": "E. On an
image, a huge cyan circle rotated at 135 degrees in the bottom right, a
tiny cyan circle rotated at 0 degrees in the top left, a tiny cyan
circle rotated at 135 degrees in the bottom left. "}, {"role": "user",
"content": "F. On an image, a huge yellow circle rotated at 135 degrees
in the bottom right, a tiny lime circle rotated at 0 degrees in the top
left, a tiny orange circle rotated at 135 degrees in the bottom left.
"}, {"role": "user", "content": "G. On an image, a huge cyan circle
rotated at 135 degrees in the bottom left, a tiny cyan circle rotated at
0 degrees in the top right, a tiny cyan circle rotated at 135 degrees in
the bottom right. "}, {"role": "user", "content": "H. On an image, a
huge cyan hexagon rotated at 135 degrees in the bottom right, a tiny
cyan pentagon rotated at 0 degrees in the top left, a tiny cyan pentagon
rotated at 135 degrees in the bottom left. "}, {"role": "user",
"content": "The answer is "}], "ideal": "E"}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Pick the letter in front of the correct pattern
that logically follows in the sequence from the answer set. Patterns in
the sequence are preceded by a number from 1 to 8. Patterns in the
answer set are preceded by a letter from A to H. Only return the letter
in front of the correct pattern."}, {"role": "user", "content": "1. On
an image, a small red circle rotated at -135 degrees in the top left.
"}, {"role": "user", "content": "2. On an image, a small red hexagon
rotated at -135 degrees in the top right. "}, {"role": "user",
"content": "3. On an image, a small red triangle rotated at -135 degrees
in the center. "}, {"role": "user", "content": "4. On an image, a giant
cyan hexagon rotated at -135 degrees in the top center. "}, {"role":
"user", "content": "5. On an image, a giant cyan triangle rotated at
-135 degrees in the center left. "}, {"role": "user", "content": "6. On
an image, a giant cyan circle rotated at -135 degrees in the center
right. "}, {"role": "user", "content": "7. On an image, a tiny green
triangle rotated at -45 degrees in the center, a tiny green triangle
rotated at -45 degrees in the bottom left, a tiny green triangle rotated
at -45 degrees in the center left. "}, {"role": "user", "content": "8.
On an image, a tiny green circle rotated at -45 degrees in the bottom
left, a tiny green circle rotated at -45 degrees in the bottom right, a
tiny green circle rotated at -45 degrees in the center right. "},
{"role": "user", "content": "A. On an image, a huge yellow circle
rotated at -45 degrees in the bottom center, a large green square
rotated at 180 degrees in the center left, a small red triangle rotated
at -45 degrees in the top center, a medium pink triangle rotated at -45
degrees in the center, a small green pentagon rotated at 135 degrees in
the bottom right, a giant lime triangle rotated at 180 degrees in the
top left, a large blue pentagon rotated at -90 degrees in the center
right. "}, {"role": "user", "content": "B. On an image, a tiny green
circle rotated at -45 degrees in the bottom right, a tiny green triangle
rotated at -45 degrees in the top center, a tiny green triangle rotated
at -45 degrees in the bottom center. "}, {"role": "user", "content": "C.
On an image, a tiny green triangle rotated at -45 degrees in the bottom
right, a tiny green square rotated at -45 degrees in the top center, a
tiny green circle rotated at -45 degrees in the bottom center. "},
{"role": "user", "content": "D. On an image, a large green hexagon
rotated at -45 degrees in the bottom right, a giant green hexagon
rotated at -45 degrees in the top center, a small green hexagon rotated
at -45 degrees in the bottom center. "}, {"role": "user", "content": "E.
On an image, a huge green hexagon rotated at -45 degrees in the bottom
right, a medium green hexagon rotated at -45 degrees in the top center,
a large green hexagon rotated at -45 degrees in the bottom center. "},
{"role": "user", "content": "F. On an image, a tiny green hexagon
rotated at -45 degrees in the center right, a tiny green hexagon rotated
at -45 degrees in the center, a tiny green hexagon rotated at -45
degrees in the top center. "}, {"role": "user", "content": "G. On an
image, a tiny green pentagon rotated at -45 degrees in the bottom right,
a tiny green circle rotated at -45 degrees in the top center, a tiny
green pentagon rotated at -45 degrees in the bottom center. "}, {"role":
"user", "content": "H. On an image, a tiny green hexagon rotated at -45
degrees in the bottom right, a tiny green hexagon rotated at -45 degrees
in the top center, a tiny green hexagon rotated at -45 degrees in the
bottom center. "}, {"role": "user", "content": "The answer is "}],
"ideal": "H"}

{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Pick the letter in front of the correct pattern
that logically follows in the sequence from the answer set. Patterns in
the sequence are preceded by a number from 1 to 8. Patterns in the
answer set are preceded by a letter from A to H. Only return the letter
in front of the correct pattern."}, {"role": "user", "content": "1. [(D,
B, F, F,)] "}, {"role": "user", "content": "2. [(F, B, D, F,)] "},
{"role": "user", "content": "3. [(B, A, B, F,)] "}, {"role": "user",
"content": "4. [(B, B, F, G,)] "}, {"role": "user", "content": "5. [(D,
B, D, G,)] "}, {"role": "user", "content": "6. [(F, A, B, G,)] "},
{"role": "user", "content": "7. [(F, A, F, C,)] "}, {"role": "user",
"content": "8. [(B, A, D, C,)] "}, {"role": "user", "content": "A. [(D,
A, B, C,)] "}, {"role": "user", "content": "B. [(D, A, F, C,)] "},
{"role": "user", "content": "C. [(D, A, E, C,)] "}, {"role": "user",
"content": "D. [(C, A, B, C,)] "}, {"role": "user", "content": "E. [(D,
B, B, C,)] "}, {"role": "user", "content": "F. [(D, A, D, C,)] "},
{"role": "user", "content": "G. [(D, I, B, C,)] "}, {"role": "user",
"content": "H. [(D, D, B, C,)] "}, {"role": "user", "content": "The
answer is "}], "ideal": "A"}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Pick the letter in front of the correct pattern
that logically follows in the sequence from the answer set. Patterns in
the sequence are preceded by a number from 1 to 8. Patterns in the
answer set are preceded by a letter from A to H. Only return the letter
in front of the correct pattern."}, {"role": "user", "content": "1. [(E,
H, B, H, BR), (B, H, B, C, TL), (D, H, B, E, BL)] "}, {"role": "user",
"content": "2. [(E, I, F, H, BR), (B, I, F, C, BL), (D, I, F, E, TR)]
"}, {"role": "user", "content": "3. [(E, J, C, H, BR), (B, J, C, C, TL),
(D, J, C, E, TR)] "}, {"role": "user", "content": "4. [(D, D, F, D, BR),
(A, D, F, F, TL), (F, D, F, C, TR)] "}, {"role": "user", "content": "5.
[(D, E, C, D, BR), (A, E, C, F, TL), (F, E, C, C, BL)] "}, {"role":
"user", "content": "6. [(D, F, B, D, BR), (A, F, B, F, BL), (F, F, B, C,
TR)] "}, {"role": "user", "content": "7. [(E, D, C, G, BR), (A, D, C, D,
BL), (A, D, C, G, TR)] "}, {"role": "user", "content": "8. [(E, E, B, G,
BR), (A, E, B, D, TL), (A, E, B, G, TR)] "}, {"role": "user", "content":
"A. [(E, F, D, G, BR), (A, F, B, D, TL), (A, F, B, G, BL)] "}, {"role":
"user", "content": "B. [(E, F, F, G, TR), (A, F, F, D, BL), (A, F, F, G,
TL)] "}, {"role": "user", "content": "C. [(E, F, C, G, BR), (A, F, E, D,
TL), (A, F, E, G, BL)] "}, {"role": "user", "content": "D. [(E, F, F, G,
TL), (A, F, F, D, BR), (A, F, F, G, TR)] "}, {"role": "user", "content":
"E. [(E, F, F, G, BR), (A, F, F, D, TL), (A, F, F, G, BL)] "}, {"role":
"user", "content": "F. [(E, C, F, G, BR), (A, D, F, D, TL), (A, B, F, G,
BL)] "}, {"role": "user", "content": "G. [(E, F, F, G, BL), (A, F, F, D,
TR), (A, F, F, G, BR)] "}, {"role": "user", "content": "H. [(E, F, E, G,
BR), (A, F, D, D, TL), (A, F, D, G, BL)] "}, {"role": "user", "content":
"The answer is "}], "ideal": "E"}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Pick the letter in front of the correct pattern
that logically follows in the sequence from the answer set. Patterns in
the sequence are preceded by a number from 1 to 8. Patterns in the
answer set are preceded by a letter from A to H. Only return the letter
in front of the correct pattern."}, {"role": "user", "content": "1. [(B,
A, F, A, TL)] "}, {"role": "user", "content": "2. [(B, A, E, A, TR)] "},
{"role": "user", "content": "3. [(B, A, B, A, C)] "}, {"role": "user",
"content": "4. [(F, F, E, A, TC)] "}, {"role": "user", "content": "5.
[(F, F, B, A, CL)] "}, {"role": "user", "content": "6. [(F, F, F, A,
CR)] "}, {"role": "user", "content": "7. [(A, E, B, C, C), (A, E, B, C,
BL), (A, E, B, C, CL)] "}, {"role": "user", "content": "8. [(A, E, F, C,
BL), (A, E, F, C, BR), (A, E, F, C, CR)] "}, {"role": "user", "content":
"A. [(E, C, F, C, BC), (D, E, C, H, CL), (B, A, B, C, TC), (C, I, B, C,
C), (B, E, D, G, BR), (F, D, B, H, TL), (D, G, D, B, CR)] "}, {"role":
"user", "content": "B. [(A, E, F, C, BR), (A, E, B, C, TC), (A, E, B, C,
BC)] "}, {"role": "user", "content": "C. [(A, E, B, C, BR), (A, E, C, C,
TC), (A, E, F, C, BC)] "}, {"role": "user", "content": "D. [(D, E, E, C,
BR), (F, E, E, C, TC), (B, E, E, C, BC)] "}, {"role": "user", "content":
"E. [(E, E, E, C, BR), (C, E, E, C, TC), (D, E, E, C, BC)] "}, {"role":
"user", "content": "F. [(A, E, E, C, CR), (A, E, E, C, C), (A, E, E, C,
TC)] "}, {"role": "user", "content": "G. [(A, E, D, C, BR), (A, E, F, C,
TC), (A, E, D, C, BC)] "}, {"role": "user", "content": "H. [(A, E, E, C,
BR), (A, E, E, C, TC), (A, E, E, C, BC)] "}, {"role": "user", "content":
"The answer is "}], "ideal": "H"}

{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Write the correct pattern with the same format
as in the examples. Patterns in the sequence are preceded by a number
from 1 to 8. "}, {"role": "user", "content": "1. On an image, a large
orange circle rotated at 90 degrees. "}, {"role": "user", "content": "2.
On an image, a giant orange pentagon rotated at 90 degrees. "}, {"role":
"user", "content": "3. On an image, a small red triangle rotated at 90
degrees. "}, {"role": "user", "content": "4. On an image, a small orange
circle rotated at 135 degrees. "}, {"role": "user", "content": "5. On an
image, a large orange pentagon rotated at 135 degrees. "}, {"role":
"user", "content": "6. On an image, a giant red triangle rotated at 135
degrees. "}, {"role": "user", "content": "7. On an image, a giant red
circle rotated at -45 degrees. "}, {"role": "user", "content": "8. On an
image, a small red pentagon rotated at -45 degrees. "}, {"role": "user",
"content": "The pattern that logically follows is:\n9. "}], "ideal": "On
an image, a large red triangle rotated at -45 degrees. "}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Write the correct pattern with the same format
as in the examples. Patterns in the sequence are preceded by a number
from 1 to 8. "}, {"role": "user", "content": "1. On an image, a huge
purple triangle rotated at 180 degrees in the bottom right, a small
purple triangle rotated at -45 degrees in the top left, a large purple
triangle rotated at 45 degrees in the bottom left. "}, {"role": "user",
"content": "2. On an image, a huge pink circle rotated at 180 degrees in
the bottom right, a small pink circle rotated at -45 degrees in the
bottom left, a large pink circle rotated at 45 degrees in the top right.
"}, {"role": "user", "content": "3. On an image, a huge white square
rotated at 180 degrees in the bottom right, a small white square rotated
at -45 degrees in the top left, a large white square rotated at 45
degrees in the top right. "}, {"role": "user", "content": "4. On an
image, a large lime circle rotated at 0 degrees in the bottom right, a
tiny lime circle rotated at 90 degrees in the top left, a giant lime
circle rotated at -45 degrees in the top right. "}, {"role": "user",
"content": "5. On an image, a large green square rotated at 0 degrees in
the bottom right, a tiny green square rotated at 90 degrees in the top
left, a giant green square rotated at -45 degrees in the bottom left.
"}, {"role": "user", "content": "6. On an image, a large cyan triangle
rotated at 0 degrees in the bottom right, a tiny cyan triangle rotated
at 90 degrees in the bottom left, a giant cyan triangle rotated at -45
degrees in the top right. "}, {"role": "user", "content": "7. On an
image, a huge lime square rotated at 135 degrees in the bottom right, a
tiny lime square rotated at 0 degrees in the bottom left, a tiny lime
square rotated at 135 degrees in the top right. "}, {"role": "user",
"content": "8. On an image, a huge green triangle rotated at 135 degrees
in the bottom right, a tiny green triangle rotated at 0 degrees in the
top left, a tiny green triangle rotated at 135 degrees in the top right.
"}, {"role": "user", "content": "The pattern that logically follows
is:\n9. "}], "ideal": "On an image, a huge cyan circle rotated at 135
degrees in the bottom right, a tiny cyan circle rotated at 0 degrees in
the top left, a tiny cyan circle rotated at 135 degrees in the bottom
left. "}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Write the correct pattern with the same format
as in the examples. Patterns in the sequence are preceded by a number
from 1 to 8. "}, {"role": "user", "content": "1. On an image, a small
red circle rotated at -135 degrees in the top left. "}, {"role": "user",
"content": "2. On an image, a small red hexagon rotated at -135 degrees
in the top right. "}, {"role": "user", "content": "3. On an image, a
small red triangle rotated at -135 degrees in the center. "}, {"role":
"user", "content": "4. On an image, a giant cyan hexagon rotated at -135
degrees in the top center. "}, {"role": "user", "content": "5. On an
image, a giant cyan triangle rotated at -135 degrees in the center left.
"}, {"role": "user", "content": "6. On an image, a giant cyan circle
rotated at -135 degrees in the center right. "}, {"role": "user",
"content": "7. On an image, a tiny green triangle rotated at -45 degrees
in the center, a tiny green triangle rotated at -45 degrees in the
bottom left, a tiny green triangle rotated at -45 degrees in the center
left. "}, {"role": "user", "content": "8. On an image, a tiny green
circle rotated at -45 degrees in the bottom left, a tiny green circle
rotated at -45 degrees in the bottom right, a tiny green circle rotated
at -45 degrees in the center right. "}, {"role": "user", "content": "The
pattern that logically follows is:\n9. "}], "ideal": "On an image, a
tiny green hexagon rotated at -45 degrees in the bottom right, a tiny
green hexagon rotated at -45 degrees in the top center, a tiny green
hexagon rotated at -45 degrees in the bottom center. "}

{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Write the correct pattern with the same format
as in the examples. Patterns in the sequence are preceded by a number
from 1 to 8. "}, {"role": "user", "content": "1. [(D, B, F, F,)] "},
{"role": "user", "content": "2. [(F, B, D, F,)] "}, {"role": "user",
"content": "3. [(B, A, B, F,)] "}, {"role": "user", "content": "4. [(B,
B, F, G,)] "}, {"role": "user", "content": "5. [(D, B, D, G,)] "},
{"role": "user", "content": "6. [(F, A, B, G,)] "}, {"role": "user",
"content": "7. [(F, A, F, C,)] "}, {"role": "user", "content": "8. [(B,
A, D, C,)] "}, {"role": "user", "content": "The pattern that logically
follows is:\n9. "}], "ideal": "[(D, A, B, C,)] "}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Write the correct pattern with the same format
as in the examples. Patterns in the sequence are preceded by a number
from 1 to 8. "}, {"role": "user", "content": "1. [(E, H, B, H, BR), (B,
H, B, C, TL), (D, H, B, E, BL)] "}, {"role": "user", "content": "2. [(E,
I, F, H, BR), (B, I, F, C, BL), (D, I, F, E, TR)] "}, {"role": "user",
"content": "3. [(E, J, C, H, BR), (B, J, C, C, TL), (D, J, C, E, TR)]
"}, {"role": "user", "content": "4. [(D, D, F, D, BR), (A, D, F, F, TL),
(F, D, F, C, TR)] "}, {"role": "user", "content": "5. [(D, E, C, D, BR),
(A, E, C, F, TL), (F, E, C, C, BL)] "}, {"role": "user", "content": "6.
[(D, F, B, D, BR), (A, F, B, F, BL), (F, F, B, C, TR)] "}, {"role":
"user", "content": "7. [(E, D, C, G, BR), (A, D, C, D, BL), (A, D, C, G,
TR)] "}, {"role": "user", "content": "8. [(E, E, B, G, BR), (A, E, B, D,
TL), (A, E, B, G, TR)] "}, {"role": "user", "content": "The pattern that
logically follows is:\n9. "}], "ideal": "[(E, F, F, G, BR), (A, F, F, D,
TL), (A, F, F, G, BL)] "}
{"input": [{"role": "system", "content": "Find the pattern number 9 that
completes the sequence. Write the correct pattern with the same format
as in the examples. Patterns in the sequence are preceded by a number
from 1 to 8. "}, {"role": "user", "content": "1. [(B, A, F, A, TL)] "},
{"role": "user", "content": "2. [(B, A, E, A, TR)] "}, {"role": "user",
"content": "3. [(B, A, B, A, C)] "}, {"role": "user", "content": "4.
[(F, F, E, A, TC)] "}, {"role": "user", "content": "5. [(F, F, B, A,
CL)] "}, {"role": "user", "content": "6. [(F, F, F, A, CR)] "}, {"role":
"user", "content": "7. [(A, E, B, C, C), (A, E, B, C, BL), (A, E, B, C,
CL)] "}, {"role": "user", "content": "8. [(A, E, F, C, BL), (A, E, F, C,
BR), (A, E, F, C, CR)] "}, {"role": "user", "content": "The pattern that
logically follows is:\n9. "}], "ideal": "[(A, E, E, C, BR), (A, E, E, C,
TC), (A, E, E, C, BC)] "}
  ```
</details>

Author: ggendro

evals/registry/data/raven-matrices/symbolic-open/center_single.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c84632d5b8e597ff50a4e9a959bfc6df2cab42798aa1b955978cd11666134a2b
+size 1759999

evals/registry/data/raven-matrices/symbolic-open/distribute_four.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8e8109a535682ee2c04ea01b4bed7fe21de7d0b709f00fb81c8fc4af8b8b1f3
+size 2219071

evals/registry/data/raven-matrices/symbolic-open/distribute_nine.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:948b2b5ac3ad0e75696bd5ebd17ae5c42dbdc617cf5e6fc12601246a881171cc
+size 2996485

evals/registry/data/raven-matrices/symbolic-open/in_center_single_out_center_single.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4860028a0c3a6e4fd886c81d31b95899d89ec426e37a6273e78e39bd126899ec
+size 2695999

evals/registry/data/raven-matrices/symbolic-open/in_distribute_four_out_center_single.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31998c9f1371fe10d32edf1f714b3b521918111ca5fb38848887c6e370b57523
+size 3148789

evals/registry/data/raven-matrices/symbolic-open/left_center_single_right_center_single.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0fe84cbc56576c987fdcb4b054b2e90ad4f712196ff42f1cd946d9e25986f7af
+size 2083999

evals/registry/data/raven-matrices/symbolic-open/up_center_single_down_center_single.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67d69873a1ed555b00a4bf0b78ac0507381e778763d4ceec93c1b8a7abd12ab8
+size 2389999

evals/registry/data/raven-matrices/symbolic/center_single.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0b7f3af9212d611e1fd86978218d2fc800407c9c86b2eb6a6271c4c7dd83abd
+size 2741999

evals/registry/data/raven-matrices/symbolic/distribute_four.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f8126d64f49fe285a6143d7e04d5400e4aedd17091134cc837ff329f9fd0fa4
+size 3577943

evals/registry/data/raven-matrices/symbolic/distribute_nine.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a6e5c6fedbd921c4b35c1bea277567de2b9fc43e3554892ae3a0501cfc5dbd8
+size 4949952