Skip to content

Dataset/quac dataset #484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 3, 2023
Merged

Dataset/quac dataset #484

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bbf7a4c
add prompt
Prikshit7766 Jun 2, 2023
25eaf6d
validation data added and path added
Prikshit7766 Jun 2, 2023
512d56c
added quac-train and tiny
RakshitKhajuria Jun 2, 2023
339ecbb
added train to path
RakshitKhajuria Jun 2, 2023
8c248e9
removed quac from custom prompt
RakshitKhajuria Jun 2, 2023
b4c5243
Update and rename QUAC-train.jsonl to Quac-train.jsonl
Prikshit7766 Jun 2, 2023
649a4f4
added new quac train
RakshitKhajuria Jun 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11,567 changes: 11,567 additions & 0 deletions nlptest/data/Quac/QUAC-train.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

50 changes: 50 additions & 0 deletions nlptest/data/Quac/Quac-train-tiny.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

50 changes: 50 additions & 0 deletions nlptest/data/Quac/Quac-val-tiny.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions nlptest/data/Quac/Quac-val.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion nlptest/datahandler/datasource.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,13 +140,16 @@ def _load_dataset(cls, dataset_name: str):
'MMLU-dev-tiny' : script_dir[:-7]+'/MMLU/MMLU-dev-tiny.jsonl',
'MMLU-test-tiny' : script_dir[:-7]+'/MMLU/MMLU-test-tiny.jsonl',
'MMLU-val-tiny' : script_dir[:-7]+'/MMLU/MMLU-val-tiny.jsonl',

'OpenBookQA-test' : script_dir[:-7]+'/OpenBookQA/OpenBookQA-test.jsonl',
'OpenBookQA-train' : script_dir[:-7]+'/OpenBookQA/OpenBookQA-train.jsonl',
'OpenBookQA-dev' : script_dir[:-7]+'/OpenBookQA/OpenBookQA-dev.jsonl',
'OpenBookQA-test-tiny' : script_dir[:-7]+'/OpenBookQA/OpenBookQA-test-tiny.jsonl',
'OpenBookQA-train-tiny' : script_dir[:-7]+'/OpenBookQA/OpenBookQA-train-tiny.jsonl',
'OpenBookQA-dev-tiny' : script_dir[:-7]+'/OpenBookQA/OpenBookQA-dev-tiny.jsonl',
'Quac-val' : script_dir[:-7]+'/Quac/Quac-val.jsonl',
'Quac-val-tiny' : script_dir[:-7]+'/Quac/Quac-val-tiny.jsonl',
'Quac-train' : script_dir[:-7]+'/Quac/Quac-train.jsonl',
'Quac-train-tiny' : script_dir[:-7]+'/Quac/Quac-train-tiny.jsonl',
}
return datasets_info[dataset_name]

Expand Down
3 changes: 2 additions & 1 deletion nlptest/transform/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7202,7 +7202,8 @@ def get_entity_representation_proportions(entity_representation):
"xsum": "You are an intelligent Context summarizer. Please read the following context carefully. After understanding its content, create a concise summary, capturing the essential themes and key details. Please ensure that the summary does not end abruptly and remains within the max_tokens word limit. Context: {context}\n\n Summary: ",
"truthfulqa": "As an intelligent bot, your primary mission is to analyze the question provided and offer a concise answer that directly addresses the query at hand. Context: {context}\n Question: {question}\n Answer:",
"mmlu": "You are an AI bot specializing in providing accurate and concise answers to questions. You will be presented with a question and multiple-choice answer options. Your task is to choose the correct answer. Context: {context}\n Question: {question}\n Answer:",
"openbookqa": "You are an AI bot specializing in providing accurate and concise answers to questions. You will be presented with a question and multiple-choice answer options. Your task is to choose the correct answer. Context: {context}\n Question: {question}\n Answer:"
"openbookqa": "You are an AI bot specializing in providing accurate and concise answers to questions. You will be presented with a question and multiple-choice answer options. Your task is to choose the correct answer. Context: {context}\n Question: {question}\n Answer:",
"quac": "You are an intelligent bot specialized in question answering. Your goal is to provide accurate and concise answers. Please read the context below carefully, which contains information relevant to the questions. For each question, provide the corresponding answers using above context . Context: {context}\n Question: {question}\n Answer:",
}

nth = {
Expand Down
6 changes: 4 additions & 2 deletions nlptest/utils/custom_types/sample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@
"xsum": "You are an intelligent Context summarizer. Please read the following context carefully. After understanding its content, create a concise summary, capturing the essential themes and key details. Please ensure that the summary does not end abruptly and remains within the max_tokens word limit. Context: {context}\n\n Summary: ",
"truthfulqa": "As an intelligent bot, your primary mission is to analyze the question provided and offer a concise answer that directly addresses the query at hand. Context: {context}\n Question: {question}\n Answer:",
"mmlu": "You are an AI bot specializing in providing accurate and concise answers to questions. You will be presented with a question and multiple-choice answer options. Your task is to choose the correct answer. Context: {context}\n Question: {question}\n Answer:",
"openbookqa": "You are an AI bot specializing in providing accurate and concise answers to questions. You will be presented with a question and multiple-choice answer options. Your task is to choose the correct answer. Context: {context}\n Question: {question}\n Answer:" }
"openbookqa": "You are an AI bot specializing in providing accurate and concise answers to questions. You will be presented with a question and multiple-choice answer options. Your task is to choose the correct answer. Context: {context}\n Question: {question}\n Answer:" ,
"quac": "You are an intelligent bot specialized in question answering. Your goal is to provide accurate and concise answers. Please read the context below carefully, which contains information relevant to the questions. For each question, provide the corresponding answers using above context . Context: {context}\n Question: {question}\n Answer:",
}

class BaseSample(BaseModel):
"""
Expand Down Expand Up @@ -398,7 +400,7 @@ def is_pass(self) -> bool:
from langchain.prompts import PromptTemplate

""""""
if self.dataset_name not in ['BoolQ', 'TruthfulQA']:
if self.dataset_name not in ['BoolQ', 'TruthfulQA', 'Quac']:
PROMPT = PromptTemplate(input_variables=["query", "answer", "result"], template=qa_prompt_template)
eval_chain = QAEvalChain.from_llm(llm=llm_model.model_class.model, prompt=PROMPT)
inputs = [{
Expand Down