Prompts for climate_fever

[drugilsberg]

Created five template prompt to evaluate claim/evidence pairs from the dataset.
One main note.
Looking at the schema from HuggingFace datasets (https://huggingface.co/datasets/climate_fever#data-fields), the labels should be indexed as follows:

• claim_label: a int feature, overall label assigned to claim (based on evidence majority vote). The label correspond to 0: "refutes", 1: "supports" and 2: "not enough info"
• evidence_label: a int feature, micro-verdict label. The label correspond to 0: "refutes", 1: "supports" and 2: "not enough info"
  But after some inspection I realized that 0 is usually associated to SUPPORTS and 1 is associated to REFUTES.
  The template I created reflects this.

[drugilsberg]

hey @VictorSanh what are the next steps, should I add some more details or are there any changes I should work on?

[VictorSanh]

Yes, i think you are right, can you open a PR on datasets to correct this? + I also see that claim_label can have 3 as a value (Disputed)... I don't see disputed being used in the individual evidences though.

This looks great thank you!

Could you add some more templates?
For instance, one could be:

Here's a statement and accompanying evidence. Does the evidence {{"supports"}}, {{"refutes"}}, or provide {{"not enough info"}} on climate change?
Statement: {{claim}}
Evidence: {{evidence[...]}}|||
your_label

[drugilsberg]

Yes, i think you are right, can you open a PR on datasets to correct this? + I also see that claim_label can have 3 as a value (Disputed)... I don't see disputed being used in the individual evidences though.

This looks great thank you!

Could you add some more templates?
For instance, one could be:

Here's a statement and accompanying evidence. Does the evidence {{"supports"}}, {{"refutes"}}, or provide {{"not enough info"}} on climate change?
Statement: {{claim}}
Evidence: {{evidence[...]}}|||
your_label

Exactly it is not used for the evidence. I will also open the PR on datasets and add some more templates.

[drugilsberg]

Yes, i think you are right, can you open a PR on datasets to correct this? + I also see that claim_label can have 3 as a value (Disputed)... I don't see disputed being used in the individual evidences though.
This looks great thank you!
Could you add some more templates?
For instance, one could be:

Here's a statement and accompanying evidence. Does the evidence {{"supports"}}, {{"refutes"}}, or provide {{"not enough info"}} on climate change?
Statement: {{claim}}
Evidence: {{evidence[...]}}|||
your_label

Exactly it is not used for the evidence. I will also open the PR on datasets and add some more templates.

Thanks again for the quick feedback @VictorSanh. I added some new templates and fixed the labeling:

• paired templates as suggested in the PR review.
• a template considering all evidence statements together.

Will now open the PR on datasets.

[drugilsberg]

@VictorSanh As promised here the PR in datasets: huggingface/datasets#2464.

[VictorSanh]

merging now, please update the spreadsheet!