Add align_labels_with_mapping function

[lewtun]

This PR adds a helper function to align the label2id mapping between a datasets.Dataset and a classifier (e.g. a transformer with a PretrainedConfig.label2id dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from datasets.

An example where this is needed is if we naively try to evaluate microsoft/deberta-base-mnli on mnli because the model config has the following mappings:

  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }

while the mnli dataset has the contradiction and neutral labels swapped:

id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}

As a result, we get a much lower accuracy during evaluation:

from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)

cc @thomwolf @lhoestq

[lewtun]

@lhoestq i think this is ready for another review 🙂

[lhoestq]

Thanks for adding this method and writing the test !
Just have 2 comments:

[lewtun]

@lhoestq thanks for the feedback - it's now integrated :)

i also added a comment about sorting the input label IDs

[lhoestq]

Thanks ! Looks all good now :)

We will also need to have the DatasetDict.align_labels_with_mapping method. Let me quickly add it

[lhoestq]

Created the PR here: #2510

[lewtun]

Thanks ! Looks all good now :)

We will also need to have the DatasetDict.align_labels_with_mapping method. Let me quickly add it

thanks a lot! i always forget about DatasetDict - will be happy when it's just one "dataset" object :)

[retoj]

So, there seems to be a problem with the function align_labels_with_mapping for models like this: https://huggingface.co/huggingface/distilbert-base-uncased-finetuned-mnli]. At least with this model, but perhaps also with others, the model.config.label2id values are of type str not int, which crashes said function. After manually converting the model.config.label2id values to int, the script runs smoothly.