-
Notifications
You must be signed in to change notification settings - Fork 99
Open
Description
It seems that in the current version of master in the he_init function, gain is being passed for argument a.
But as per Pytorch code in both version 1.0 and 0.4.1, kaiming_normal_() expects a to be negative slope and has nonlinearity argument separately. So just following function call should do:
kaiming_normal(layer.weight, nonlinearity=nonlinearity, a=param)
Pytorch source code link: https://github.com/pytorch/pytorch/blob/v0.4.1/torch/nn/init.py#L296
I have tested it on some layers, just above call gives the expected value to std() of layer.weight.
Your code might have been based on some different version of pytorch that did expect to pass gain separately but thought should you a heads up just in case.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels