Some weights not initialized in pre-trained RobertaForMaskedLM

[HarshTrivedi]

The bug is similar to #2202.

I am trying to evaluate MLM perplexity (without training/finetuning) using Roberta with run_language_modeling.py (from the official example). However, some weights seems to be reinitialized instead of getting loading from the pretrained Roberta checkpoint.

To Reproduce (with master branch):

import logging
logging.basicConfig(level=logging.INFO)
from transformers import RobertaForMaskedLM
_ = RobertaForMaskedLM.from_pretrained('roberta-base')

It gives the following warning message:

WARNING:transformers.modeling_utils:Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at roberta-base and are newly initialized: ['roberta.embeddings.position_ids', 'lm_head.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

The perplexities I get on direct evaluation on Wikitext-2/103 datasets are also much higher than the official Roberta implementation from fairseq. I suspect this could be the reason.

[LysandreJik]

Hello! These warnings are not important, as these weights are not necessary (the position IDs is a buffer that is initialized if not defined, and the lm head decoder bias already exists in the lm head decoder linear layer).

On the master branch we've updated the warnings to only list those that could have an impact, so running your code on the current master branch results in:

WARNING:transformers.modeling_utils:Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at roberta-base and are newly initialized: ['lm_head.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

So nothing wrong here! Did you make sure you were using the exact same implementation of perplexity calculation than Fairseq?

[HarshTrivedi]

Thanks for the clarification! The difference in fairseq and transformers ppl is coming from different implementation - e^cross_entropy (in transformers) vs 2^cross_entropy (in fairseq). Nothing wrong here! : )

[cloudygoose]

@LysandreJik
Hi, did not understand "the lm head decoder bias already exists in the lm head decoder linear layer", but it still warnings that lm_head.decoder.bias is newly initialized? I'm confused, sorry. Could you elaborate more about why this is not a problem?

[LysandreJik]

Hi @cloudygoose, I recommend you take a look at the following class:

transformers/src/transformers/models/roberta/modeling_roberta.py

Lines 1065 to 1087 in b8462b5

	class RobertaLMHead(nn.Module):
	"""Roberta Head for masked language modeling."""
	def __init__(self, config):
	super().__init__()
	self.dense = nn.Linear(config.hidden_size, config.hidden_size)
	self.layer_norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
	self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
	self.bias = nn.Parameter(torch.zeros(config.vocab_size))
	# Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
	self.decoder.bias = self.bias
	def forward(self, features, **kwargs):
	x = self.dense(features)
	x = gelu(x)
	x = self.layer_norm(x)
	# project back to size of vocabulary with bias
	x = self.decoder(x)
	return x

The error tells you that the following weight: lm_head.decoder.bias was not initialized from the model checkpoint: the model checkpoint did not contain that weight.

However, the weight lm_head.bias isn't in the error because it was correctly initialized. If you take a look at the last line of the initialization of the class above, you'll see:

         # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings` 
         self.decoder.bias = self.bias 

Therefore, the lm_head.decoder.bias weight that was not initialized is now set to the value of self.bias, which is correctly initialized.

Let me know if something isn't clear.

[gchhablani]

Hi @LysandreJik,

I am trying to make a custom model based on Roberta. I use RobertaModel internally.

The warning is:

Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModelForTokenAndSpans: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']
- This IS expected if you are initializing RobertaModelForTokenAndSpans from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModelForTokenAndSpans from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaModelForTokenAndSpans were not initialized from the model checkpoint at roberta-large and are newly initialized: ['roberta.embeddings.position_ids', 'classifier.weight', 'classifier.bias', 'qa_outputs.weight', 'qa_outputs.bias']

classifier and qa_outputs are my own layers. I assume from the previous release (4.2.2), this warning has not been removed?
Will having roberta.embeddings.position_ids not initialized from roberta affect things in any way?

[LysandreJik]

This warning shouldn't be removed, it's telling you what it initializes randomly, and what isn't used. Apparently it's not an issue in your case since you're aware of it, so that's great!

Not an issue for the position IDs, this warning should have been removed in version v4.3.2, though!

[gchhablani]

I meant the warning about for position IDs.
Thanks a lot @LysandreJik :)

[LysandreJik]

Happy to help!