|
linear.weight.data = W_ - W_.mean(dim=-2, keepdim=True) |
Hello. I have a question concerning how to bake mean subtraction from LayerNorm into the Linear layer.
I have managed to solve by hand that it is possible to merge mean subtraction from the layernorm into the linear layer by subtracting the mean of each column of the weight matrix.
However, because the nn.Linear class holds the weights transposed for memory contiguity, I think that one should do
W_ - W_.mean(dim=-1, keepdim=True) instead of W_ - W_.mean(dim=-2, keepdim=True) to subtract from the columns of the weights.
To summarize, since nn.Linear does [email protected], I think that the dimensions should be flipped.
Please correct me if I am wrong.
QuaRot/fake_quant/rotation_utils.py
Line 36 in 5008669
Hello. I have a question concerning how to bake mean subtraction from LayerNorm into the Linear layer.
I have managed to solve by hand that it is possible to merge mean subtraction from the layernorm into the linear layer by subtracting the mean of each column of the weight matrix.
However, because the
nn.Linearclass holds the weights transposed for memory contiguity, I think that one should doW_ - W_.mean(dim=-1, keepdim=True)instead ofW_ - W_.mean(dim=-2, keepdim=True)to subtract from the columns of the weights.To summarize, since
nn.Lineardoes[email protected], I think that the dimensions should be flipped.Please correct me if I am wrong.