Add korean_dialects eval

[racheroni]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access be granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject it since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. Starting April 10, the minimum eval count is 15 samples, we hope this makes it easier to create and contribute evals.

Also, please note that we're using Git LFS for storing the JSON files, so please make sure that you move the JSON file to Git LFS before submitting a PR. Details on how to use Git LFS are available here.

Eval details 📑

Eval name

korean_dialects

Eval description

The eval aims to assess the model's ability to identify the specific South Korean dialect a sentence belongs to.

What makes this a useful eval?

This eval provides the opportunity to understand how well GPT models can classify South Korean dialects. The dialects within South Korea are fairly distinct in terms of pronunciation, vocabulary, grammar, and intonation. Being able to determine the dialect can help in providing social, cultural, and historical insights.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 15 high-quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

Insert what makes your eval high quality that was not mentioned above. (Not required)

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your YAML is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the commits on the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgment

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and the high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access be granted.

Submit eval

• I have filled out all required fields of this form
• I have used Git LFS for the Eval JSON data
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"모두 다 오세요."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"벌써 열시 반이에요."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"창피해서 말도 못해요."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"그럼요 가능하죠."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"똑바로 해주세요."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"이거 조금 짜."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"간신히 도착했습니다."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"야 이거 좀 별로다."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"와 정말 많다."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"거기 구멍에 잘 끼워보세요."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"진짜 피곤해요."}],"ideal":"서울"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"마카 모예."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"하마 열시 반이잖소."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"남새시러운기 왜그르나."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"해봐요 그 될껄?"}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"거 똑떼이 해야될끼라요."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"이기 쫌 짜구워."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"거 간신히 도착했잖소."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"아 매련도 읍싸."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"마이 있잖소."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"구녕에 똑띠 끼워봐요."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"쌔가 빠진다야."}],"ideal":"강원도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"야 싹 다 온나."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"벌써 열시 반이가."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"하모 된다카이."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"단디 해라이."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"이거 쫌 짭다."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"내 강가이 도착했다."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"그 영 파이다."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"억수로 많다."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"구녕에 단디 낑가라이."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"아따 대다."}],"ideal":"경상도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"싹 다 와부쇼."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"폴쎄 열시 반이되브렀네."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"아따 하먼 된당께."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"똑떨어지게 해보랑게."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"포도시 도착했시야."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"야 이거 물짜야 못 쓰것다이."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"아따 겁나게 많네."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"구녁에 잘 찡거보쇼 딱 맞게 그게 맞소?"}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"오메 된 거 걍 오만 삭신이 아퍼 죽겄소이."}],"ideal":"전라도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"거기 다 와봐유."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"벌써 열시 반인겨?"}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"될겨."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"할껴 알 할껴."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"갠시히 왔네."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"글쎄 잘 모르겄어."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"뭐여 뭐가 이렇게 많은겨."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"거기 구멍에 잘 좀 낌어봐."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"진짜 대간햐."}],"ideal":"충청도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"발써 열시 반되수과."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"키여 맞수다게."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"졸바로 해줍써."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"제우 제우 와수다."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"영 벨론게게."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"잘도 하영이 숨게데."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"트멍더래 잘 쫍지라."}],"ideal":"제주도"}
{"input":[{"role":"system","content":"You will be prompted with a Korean sentence to determine which South Korean dialect the sentence belongs to. This is multiple choice problem where your answer is one of the following six dialects: 강원도, 경상도, 전라도, 제주도, 충청도, or 서울. Return just the dialect with no other words or punctuation."},{"role":"user","content":"잘도 버침게."}],"ideal":"제주도"}  

[usama-openai]

Thanks for submitting this eval! This PR looks good. I'm approving this PR.

[usama-openai]

You should see GPT-4 API access enabled in your account in the next few days.