`eval_on_start` triggers `AttributeError` in JupyterLab

[matteosantama]

System Info

transformers: '4.44.0'
Python: '3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]'

Who can help?

@muellerz @SunMarc

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Here is a self-contained script that triggers the error:

model_id = "distilbert/distilbert-base-uncased"

model = AutoModelForSequenceClassification.from_pretrained(
    model_id, num_labels=2
)
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)

ds = (
    Dataset.from_dict({"text": ["Hello world!"], "labels": [0]})
    .map(
        lambda b: tokenizer(b["text"], truncation=True),
        batched=True,
    )
)


trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="temp",
        eval_strategy="steps",
        save_strategy="no",
        eval_on_start=True,  # <-- setting to False avoids the error
    ),
    tokenizer=tokenizer,
    train_dataset=ds,
    eval_dataset=ds,
    compute_metrics=lambda _: {"metric": 1.0},
)
trainer.train()

The error I get is

AttributeError: 'NotebookTrainingTracker' object has no attribute 'value'

which seems related to the fact that I'm running the code in a Jupyter notebook.

Expected behavior

I should be able to use the eval_on_start parameter without triggering an error.

[fshp971]

Hi,

I have found that the cause of this exception is that in Jupyter Notebook, for a transformers.Trainer, when its argument eval_on_start is set to True, running Trainer.train() will lead to the following function execution order:

1. NotebookProgressCallback.on_train_begin()
2. NotebookProgressCallback.on_prediction_step()
3. NotebookProgressCallback.on_train_end()

Running NotebookProgressCallback.on_prediction_step() directly after NotebookProgressCallback.on_train_begin() will result in two attributes of NotebookProgressBar, i.e., value and label, being used before assigned (see methods update(), update_bar() and display() of NotebookProgressBar in transformers/utils/notebook.py for details).

I have open a PR to fix this.