Skip to content

Possible bug in he_init function #29

@ngoyal2707

Description

@ngoyal2707

It seems that in the current version of master in the he_init function, gain is being passed for argument a.
But as per Pytorch code in both version 1.0 and 0.4.1, kaiming_normal_() expects a to be negative slope and has nonlinearity argument separately. So just following function call should do:

kaiming_normal(layer.weight, nonlinearity=nonlinearity, a=param) 

Pytorch source code link: https://github.com/pytorch/pytorch/blob/v0.4.1/torch/nn/init.py#L296

I have tested it on some layers, just above call gives the expected value to std() of layer.weight.

Your code might have been based on some different version of pytorch that did expect to pass gain separately but thought should you a heads up just in case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions