Support backward of backward for Relu and add a new gradient checker by comparing theoretical and numerical Jacobian.

[qingqing01]

1. Use Relu to get through the whole process from C++ to Python.
   • Add double backward op for Relu (named relu_grad_grad op).
   • Add register for relu_grad_grad.

Add ReluDoubleGradMaker when register relu_grad:

REGISTER_OPERATOR(relu_grad, ops::ActivationOpGrad,
                  paddle::framework::SingleOpInplaceInToOut,
                  ops::ReluDoubleGradMaker);

2. Add a new gradient checker by comparing theoretical and numerical Jacobian.

   • support to check program, not one op.
   • support to check gradients by API grad_check.
   • support to check double gradients by API double_grad_check.

3. Add gradients testing for relu_grad_grad.

4. fluid.backward.calc_gradient is used to append the backward pass, which is similar interface to tf.gradients.

For example:

import paddle.fluid as pd
from paddle.fluid.backward import calc_gradient

# relu
x = pd.layers.data(name='x', shape=[2,4], dtype='float32', append_batch_size=False)
x.stop_gradient=False
y = pd.layers.relu(x)
z = calc_gradient([y], x)
c = calc_gradient(z, x)

print(pd.default_main_program())

Note, this PR doest not use macro to register double grad for activations since only add one for relu. If adding more in the future, will use macro for more general.

[wanghaoshuang]

为什么不用np.product呢？

[qingqing01]

Done, Thanks!

[wanghaoshuang]

这行应该删掉，和下面一行重复了。

[qingqing01]

Done. Thanks!

[wanghaoshuang]

这个也可以删掉了吧？

[qingqing01]

Done.

[qingqing01]

@sneaxiy I have deleted DoubleGradVarName.
@wanghaoshuang Update code according your comments.

Thanks!

[qingqing01]

[16:03:43]W:	 [Step 1/1] + API_CHANGE=python/paddle/fluid/tests/unittests/CMakeLists.txt
[16:03:43]W:	 [Step 1/1] + echo 'checking CMakeLists.txt change, PR: 16862, changes: python/paddle/fluid/tests/unittests/CMakeLists.txt'
[16:03:43]W:	 [Step 1/1] + '[' python/paddle/fluid/tests/unittests/CMakeLists.txt ']'
[16:03:43]W:	 [Step 1/1] + '[' 16862 '!=' '' ']'
[16:03:43]W:	 [Step 1/1] + '[' CMakeLists.txt == paddle/fluid/API.spec ']'
[16:03:43]W:	 [Step 1/1] + '[' CMakeLists.txt == CMakeLists.txt ']'
[16:03:43] :	 [Step 1/1] checking CMakeLists.txt change, PR: 16862, changes: python/paddle/fluid/tests/unittests/CMakeLists.txt
[16:03:43]W:	 [Step 1/1] ++ python /paddle/tools/check_pr_approval.py 1 6836917 46782768 30176695
[16:03:43]W:	 [Step 1/1] ++ curl -H 'Authorization: token 5ae37427a95d9e51fb0ea3c71121b75d5c2ebefa' 'https://api.github.com/repos/PaddlePaddle/Paddle/pulls/16862/reviews?per_page=10000'
[16:03:43]W:	 [Step 1/1]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[16:03:43]W:	 [Step 1/1]                                  Dload  Upload   Total   Spent    Left  Speed
[16:03:43]W:	 [Step 1/1] 
[16:03:44]W:	 [Step 1/1]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[16:03:45]W:	 [Step 1/1]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[16:03:45]W:	 [Step 1/1]   1  9156    1   125    0     0     64      0  0:02:23  0:00:01  0:02:22    64
[16:03:45]W:	 [Step 1/1] 100  9156  100  9156    0     0   4742      0  0:00:01  0:00:01 --:--:--  4744
[16:03:45] :	 [Step 1/1] current pr 16862 got approvals: FALSE
[16:03:45] :	 [Step 1/1] You must have one RD (luotao1 or chengduoZH or XiaoguangHu01) approval for the cmakelist change! CMakeLists.txt
[16:03:45]W:	 [Step 1/1] + APPROVALS=FALSE

The unit testing failed since Python on Windows CI have a same python package. Rename decorators to decorator_helper. So change the python/paddle/fluid/tests/unittests/CMakeLists.txt.

Please help to review. Thanks! @chengduoZH @luotao1 @XiaoguangHu01

[luotao1]

LGTM for cmakelist.txt

[luotao1]

@tianshuo78520a It seems #16944 don't fix this problem #16862 (comment)