Question about C-STANCE eval #4
Description
Deer Author:
Thanks for your great work. But I noticed that your code in metric.py:
def caculate_accuracy(results, data): scores = 0 for output_id in range(len(results)): prediction = results[output_id] target = data[output_id] if prediction == "" or target == "": continue if prediction == target: scores += 1 avg_score = scores / len(results) return avg_score
I checked the lora-trained model's output, it will not contain only the answer "A", "B", or "C" but some other words, or reason to get the answer, so the prediction == target will always be False, that is, the C_STANCE eval will always be acc 0. How do you solve this problem?