warning about weight_g/weight_v missing on WeightNorm on PyTorch

[kamilakesbi]

This PR addresses issue #26796:

• On models using WeightNorm (such as Hubert or Wav2vec2), a warning message appears indicating that some weights have not been properly initialised (weight_v and weight_g parameters).

This is because Pytorch migrates the weight_v and weight_g params of WeightNorm to original0 and original1.

The state dict is converted correctly thanks to PR #24030, but we still get a warning message when loading the model.

We can solve this by replacing weight_g and weight_v keys with corresponding updated keys in _load_pretrained_model as done in this PR.

Who can review:

cc @sanchit-gandhi

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sanchit-gandhi]

I like the elegance of this fix @kamilakesbi! My only concern is that it assumes that all models are using the new parametrizations format, e.g. the one used in Wav2Vec2:

transformers/src/transformers/models/wav2vec2/modeling_wav2vec2.py

Lines 377 to 379 in 9d6c064

	weight_norm = nn.utils.weight_norm
	if hasattr(nn.utils.parametrizations, "weight_norm"):
	weight_norm = nn.utils.parametrizations.weight_norm

Doing a quick search of the codebase, this is not the case for all models, e.g. for EnCodec:

transformers/src/transformers/models/encodec/modeling_encodec.py

Line 107 in 9d6c064

self.conv = nn.utils.weight_norm(self.conv)

=> have we checked the weight norm params are loaded without warning when we use the legacy implementation? Ideally, we would only "fix" these keys if we know we're loading a legacy weight norm param into a parametrization one

[kamilakesbi]

@sanchit-gandhi there are indeed a few models for which the new parametrizations format is missing:

• Univnet
• sew_d
• sew
• Seamless_m4t
• Seamless_m4t v2
• fast_speech2_conformer
• Encodec

I've udpated the corresponding modeling_files.

The problem may arise with future model integrations! I'm thinking of the recent dac for example.

We should probably add a warning suggesting to users to implement the new parameterisation format with weight_norm when adding a new model!

[amyeroberts]

cc @ylacombe

[ylacombe]

Superseded by #33275