Unify the implementation of activation operation #32348
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
ZzSean
force-pushed
the
activation_op_impl
branch
from
bd411e5 to
59d62d1
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
from
adfeb25 to
1d41980
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
from
beb7ed2 to
5c3cf4d
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
4 times, most recently
from
d910b6d to
711a097
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
2 times, most recently
from
1e5c724 to
78058c6
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
from
c81aa21 to
52d8151
Compare
ZzSean
force-pushed
the
activation_op_impl
branch
from
c8942d9 to
d8da01f
Compare
Xreki
reviewed
Apr 26, 2021
ZzSean
force-pushed
the
activation_op_impl
branch
from
767cbb6 to
f67e8a4
Compare
Xreki
reviewed
Apr 26, 2021
| CT dout = static_cast<CT>(args[0]); | ||
| CT x = static_cast<CT>(args[1]); | ||
| CT temp1 = one + exp(-x); | ||
| CT temp2 = x * exp(-x); |
| __device__ __forceinline__ T operator()(const T* args) const { | ||
| CT x = static_cast<CT>(args[0]); | ||
| CT temp = x > zero ? zero : -x; | ||
| return T(-temp - log(exp(-temp) + exp(-x - temp))); |
Contributor
There was a problem hiding this comment.
既然用的都是-temp，那temp计算的时候是不是就可以不要这个负号？
Contributor
Author
There was a problem hiding this comment.
为了保持与原实现和公式的统一，还是先不改了
| CT dout = static_cast<CT>(args[0]); | ||
| CT x = static_cast<CT>(args[1]); | ||
| CT temp = x > zero ? zero : -x; | ||
| return T(dout * (exp(-x - temp) / (exp(-temp) + exp(-x - temp)))); |
Contributor
There was a problem hiding this comment.
- 同上：既然用的都是
-temp，那temp计算的时候是不是就可以不要这个负号？ - 分子、分母都会用到
exp(-x - temp)，是不是可以提取出来？
| // Inputs: args[0], the input x | ||
| __device__ __forceinline__ T operator()(const T* args) const { | ||
| CT x = static_cast<CT>(args[0]); | ||
| return x >= zero ? args[0] : T(static_cast<CT>(alpha) * (exp(x) - one)); |
Contributor
There was a problem hiding this comment.
| __device__ __forceinline__ T operator()(const T* args) const { | ||
| CT dout = static_cast<CT>(args[0]); | ||
| CT x = static_cast<CT>(args[1]); | ||
| return x >= zero ? args[0] : T(dout * static_cast<CT>(alpha) * exp(x)); |
Contributor
There was a problem hiding this comment.
| auto temp1 = x < static_cast<T>(threshold * -1.f); | ||
| auto temp2 = x > static_cast<T>(threshold); | ||
| out.device(d) = x * (temp1 + temp2).template cast<T>(); | ||
| out.device(d) = x * (temp1 || temp2).template cast<T>(); |
Contributor
Author
There was a problem hiding this comment.
单测里有threshold为负的情况，用加号就会变成两倍
| ThresholdedReluFunctor, | ||
| ThresholdedReluGradFunctor); | ||
| REGISTER_ACTIVATION_GPU_KERNEL(hard_swish, HardSwish, HardSwishFunctor, | ||
| HardSwishGradFunctor); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Describe
Unify the implementation of activation operation
本次提交共修改激活算子25个，包括其前向和反向，其中
elu每种类型算子的性能提升近似，因此选取每个类别中的一个算子作为示例进行描述，如下表：
case配置：[16, 128, 257, 257]