medication_dose

[mickaw2]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access be granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject it since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. Starting April 10, the minimum eval count is 15 samples, we hope this makes it easier to create and contribute evals.

Also, please note that we're using Git LFS for storing the JSON files, so please make sure that you move the JSON file to Git LFS before submitting a PR. Details on how to use Git LFS are available here.

Eval details 📑

Eval name

medication_dose

Eval description

This tests the LLM ability to identify medication doses that are inappropriate for therapeutic uses.

What makes this a useful eval?

For LLM to have a role in medical applications, the ability to recognize medication doses that are out of therapeutic range is a very important function. I have found that these models frequently fail at this task.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 15 high-quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

LLM have tremendous possibilities for medical applications. Many future applications will need the LLM to be able to identify appropriate medication dose ranges. I have found that current LLM frequently fail at this task and this is an area where improvement would be important.

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your YAML is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgment

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and the high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access be granted.

Submit eval

• I have filled out all required fields of this form
• I have used Git LFS for the Eval JSON data
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Metronidazole 500 mg, Icatibant 30 mg, Haegarda 3000 IU, Docusate 100 mg, Levofloxacin 750 mcg, Famotidine 40 mg, Fentanyl 100 mcg, Budesonide 0.25 mg, Metoprolol 12.5 mg, Atenolol 50 mg, Labetalol 100 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Levofloxacin"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Docusate 100 mg, Diazepam 5 mg, Lanoxin 250 mcg, Privigen 25 g, Doxycycline 100 mg, CellCept 1.5 g, Xolair 300 mcg, Hydrocortisone 1%, Keflex 250 mg, Naloxone 400 mcg, Famotidine 40 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Xolair"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Budesonide 0.25 mg, Theophylline 450 mg, Humira 160 mg, Haegarda 3000 IU, Neoral 50 mg, Metronidazole 500 mg, Formoterol 20 mcg, Advair 500 mcg, Zosyn 3.375 mg, Furosemide 20 mg, Dilantin 100 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Zosyn"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Breo 200/25 mcg, Acetaminophen 1000 mg, Claratin 10 mg, Gammunex 100 gram, Levothyroxine 125 mg, Albuterol 108 mcg, Lovenox 40 mg, Betapace 120 mg, Levofloxacin 500 mg, Nystatin 100000 U, Warfarin 6.5 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Levothyroxine"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Azithromycin 500 mg, Heparin 5000 U, Atenolol 50 mg, Betapace 120 mg, Budesonide 0.25 mg, Privigen 25 g, Furosemide 20 mg, Humira 160 mg, Keflex 250 mg, Verapamil 40 mg, Symbicort 500/4.5 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Symbicort"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Famotidine 40 mg, Digoxin 0.125 mg, Rifampin 150 mg, Albuterol 108 mcg, Allegra 60 mg, Azithromycin 5 mg, Spiriva 1.25 mcg, Warfarin 6.5 mg, Nasacort 220 mcg, Cetirizine 10 mg, Azithromycin 500 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Azithromycin"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Acyclovir 200 mg, Patanol 0.2%, Famotidine 40 mg, Heparin 5000 U, Gammunex 100 gram, Prednisone 40 mg, Amitriptyline 25 mg, Betapace 120 mg, Sotolol 80 mg, Cetirizine 100 mg, Loperamide 2 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Cetirizine"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Fasenra 30 mg, Sotolol 80 mg, Levothyroxine 125 mcg, Digoxin 0.125 mg, Budesonide 1 gm, Loperamide 2 mg, Humira 160 mg, Patanol 0.2%, Dilantin 100 mg, Rifampin 150 mg, Keflex 250 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Budesonide"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Fasenra 130 mg, Doxycycline 100 mg, Patanol 0.2%, Pulmicort 1 mg, Flonase 50 mcg, Tiotropium 2.5 mcg, Nasacort 220 mcg, Acetaminophen 1000 mg, Icatibant 30 mg, Cetirizine 10 mg, Theophylline 450 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Fasenra"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Plaquenil 400 mg, Atorvastatin 2000 mg, Lovenox 40 mg, Keflex 250 mg, Pulmicort 1 mg, Versed 5 mg, Spiriva 1.25 mcg, Cyclosporine 100 mg, Ventolin 90 mcg, Icatibant 30 mg, Loperamide 2 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Atorvastatin"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Tiotropium 2.5 mcg, Plaquenil 400 mg, Azelastine 137 mg, Haegarda 3000 IU, Albendazole 400 mg, Phenytoin 30 mg, Naloxone 400 mcg, Symbicort 160/4.5 mcg, Isoniazid 100 mg, Diazepam 5 mg, Metformin 500 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Azelastine"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Budesonide 0.25 mg, Fexofenadine 180 mg, Keflex 250 mg, Fexofenadine 1.8 mg, Allegra 60 mg, Flonase 50 mcg, Rhinocort 32 mcg, Azithromycin 500 mg, Sotolol 80 mg, Fluoxetine 20 mg, Pulmicort 1 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Fexofenadine"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Flecainide 50 mg, Claratin 10 mg, Sotolol 80 mg, Atenolol 50 mg, Metformin 500 mg, Flonase 50 mcg, Advair 5 mcg, Labetalol 100 mg, Nasacort 220 mcg, Fentanyl 100 mcg, Budesonide 0.25 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Advair"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Prednisone 40 g, Digoxin 0.125 mg, Budesonide 0.25 mg, Icatibant 30 mg, Betapace 120 mg, Fluconazole 50 mg, Patanol 0.2%, Heparin 5000 U, Coumadin 3.5 mg, Metformin 500 mg, Fasenra 30 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Prednisone"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Fluconazole 50 mg, Furosemide 20 mg, Verapamil 40 mg, Nystatin 100000 U, Augmentin 875 mg, Augmentin 8.75 mg, Theophylline 450 mg, Lanoxin 250 mcg, Neoral 50 mg, Nasacort 220 mcg, Diazepam 5 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Augmentin"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Haegarda 3000 IU, Lithium 600 mg, Acyclovir 200 mg, CellCept 1.5 g, Fasenra 30 mg, Metformin 500 mg, Albendazole 400 mg, Advair 500 mcg, Zofran 4 mg, Ciprofloxacin 500 mg, Haegarda 3000 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Haegarda"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Orapred 15 mg, Carvedilol 3.125 mg, Lanoxin 250 mcg, Lovenox 40 mg, Loperamide 2 mg, Hydrocortisone 1%, Diazepam 5 mg, Zosyn 3.375 g, Warfarin 6.5 mg, Synthroid 88 mg, Cetirizine 10 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Synthroid"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Patanol 0.2%, Icatibant 30 g, Gammunex 100 gram, Versed 5 mg, Formoterol 20 mcg, Rhinocort 32 mcg, Metformin 500 mg, Motrin 800 mg,, Enoxaprin 30 mg, Metoprolol 12.5 mg, Xolair 300 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Icatibant"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? CellCept 1.5 g, Flonase 50 mcg, Trimethoprim/Sulfamethoxazole 160 mg/800 mg, Fluconazole 50 mg, Verapamil 40 mg, Loperamide 2 mg, Prednisone 40 mg, Pepcid 20 mg, Plaquenil 4 mcg, Nystatin 100000 U, Labetalol 100 mg"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Plaquenil"}
{"input": [{"role": "system", "content": "Answer this question as concisely as possible. Of the following medications and doses which is incorrect for a standard dose for that medication? Carvedilol 3.125 mg, Digoxin 0.125 mg, Fexofenadine 180 mg, Nasacort 220 mcg, Zofran 4 mg, Fluoxetine 20 mg, Mupirocin 2%, CellCept 1.5 g, Keflex 250 mg, Atorvastatin 20 mg, Amitriptyline 25 g"}, {"role": "user", "content": "The medication that is not within a standard dose range is: "}], "ideal": "Amitriptyline"}

[usama-openai]

Thanks for the contribution. I would like to request the following changes:

Kindly remove all the .gitattributes files. You don't need to add this file or make any changes to existing ones. If git lfs is properly installed, the required files will automatically be pushed as lfs files.

We would love to review the PR again after the suggested changes.

[mickaw2]

I have removed both .gitattributes files. Please let me know if you need me to do anything else.

[usama-openai]

Instead of removing evals/registry/data/medication_dose/.gitattributes, you've removed .gitattributes, which is the main attributes file for this repo. Knidly fix this issue.

[mickaw2]

My apologies! Should be fixed now. Please let me know if anything else you need.

[usama-openai]

Now .yaml file is pushed as LFS but it should be pushed as regular file. Reverting 400b8a0a846c7c06ba1115d0d242185b9afd9e2a should most probably fix this.

[mickaw2]

OK, did that thanks.

[usama-openai]

This PR looks in good shape now. I'm approving this PR.

[mickaw2]

Thanks!

[usama-openai]

You should see GPT-4 API access enabled in your account in the next few days.

[mickaw2]

Thank you!

…

On Thu, Jun 8, 2023 at 3:07 AM Usama ***@***.***> wrote: You should see GPT-4 API access enabled in your account in the next few days. — Reply to this email directly, view it on GitHub <#1095 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7CHIW5HS5KV6PWTMYLPWILXKF2ZJANCNFSM6AAAAAAYZXJBT4> . You are receiving this because you commented.Message ID: ***@***.***>

[mickaw2]

Andrew, I was checking when I would be able to access the chat GPT4 API and how I will know when it is activated. Thanks!

…

On Thu, 8 Jun 2023 at 05:47, William Micka ***@***.***> wrote: Thank you! On Thu, Jun 8, 2023 at 3:07 AM Usama ***@***.***> wrote: > You should see GPT-4 API access enabled in your account in the next few > days. > > — > Reply to this email directly, view it on GitHub > <#1095 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/A7CHIW5HS5KV6PWTMYLPWILXKF2ZJANCNFSM6AAAAAAYZXJBT4> > . > You are receiving this because you commented.Message ID: > ***@***.***> >