Skip to content

[NPU] delete useless GELU in gelu grad npu op #33872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 4, 2021
Merged

[NPU] delete useless GELU in gelu grad npu op #33872

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1e5695c
delete useless GELU in gelu npu op
pangyoki Jun 30, 2021
084173c
add description
pangyoki Jun 30, 2021
2b30114
fix format
pangyoki Jun 30, 2021
13741a1
add check_grad in gelu unittest
pangyoki Jul 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions paddle/fluid/operators/gelu_op_npu.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,14 @@ class GeluGradNPUKernel : public framework::OpKernel<T> {
ctx.template device_context<paddle::platform::NPUDeviceContext>()
.stream();

Tensor out(x->type());
out.mutable_data<T>(x->dims(), place);
const auto& runner_out = NpuOpRunner("Gelu", {*x}, {out}, {});
runner_out.Run(stream);

// NOTE(pangyoki): In the original implementation of GeluGrad op, the input
// is {*dout, *x, out}, where out = Gelu(x). However, we find that variable
// `out` was not actually used. In order to improve performance, the
// useless GELU operation was deleted.
// We directly use `*dout` as a placeholder to replace `out`, it will not
// be used in calculations.
const auto& runner_dx =
NpuOpRunner("GeluGrad", {*dout, *x, out}, {*dx}, {});
NpuOpRunner("GeluGrad", {*dout, *x, *dout}, {*dx}, {});
runner_dx.Run(stream);
}
};
Expand Down
15 changes: 6 additions & 9 deletions python/paddle/fluid/tests/unittests/npu/test_gelu_op_npu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,9 @@ def init_dtype(self):
def test_check_output(self):
self.check_output_with_place(self.place, check_dygraph=False, atol=1e-3)

# TODO(ascendrc): Add grad test
# def test_check_grad(self):
# if self.dtype == np.float16:
# return
# self.check_grad(['X'], 'Out')
#
def test_check_grad(self):
self.check_grad_with_place(
self.place, ['X'], 'Out', check_dygraph=False)


@unittest.skipIf(not paddle.is_compiled_with_npu(),
Expand Down Expand Up @@ -115,10 +112,10 @@ def _test(self, run_npu=True):
name="label", shape=[32, 1], dtype='int64')

c = paddle.multiply(a, b)
d = fluid.layers.gelu(c)

fc_1 = fluid.layers.fc(input=d, size=128)
prediction = fluid.layers.fc(input=fc_1, size=2, act='softmax')
fc_1 = fluid.layers.fc(input=c, size=128)
fc_1_gelu = fluid.layers.gelu(fc_1)
prediction = fluid.layers.fc(input=fc_1_gelu, size=2, act='softmax')

cost = fluid.layers.cross_entropy(input=prediction, label=label)
loss = fluid.layers.reduce_mean(cost)
Expand Down