Incorporate cudnn_lstm into LSTM api

[guoshengCS]

PR types

New features

PR changes

Others

Describe

Incorporate cudnn_lstm into LSTM api

1. 在LSTM中集成cudnn_lstm
2. 将基类RNNMixin修改为RNNBase，将RNNCell中的param暴露在RNNBase中
3. 为coalesce_tensor_op增加use_align属性，以便在参数转换中小tensor合并得到大tensor时忽略小tensor中的memory chunk
4. 修改cudnn_lstm kernel：对in_h/in_c为可选的求导，调整cudnn_lstm中dropout的seed设置，test时重置weightlist中参数的指针避免检测到不连续而每次拷贝。

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[iclementine]

1. coalsectensor 拼接大 tensor 目前不保证内存连续，可能需要修改。
2. 如果 cudnn lstm 完成了小 weight list 的调用方式，那么是否还必须在 python 端持有一个 fused tensor？（可能需要测试一下？）

[guoshengCS]

Thanks. Done
为coalsec_tensor_op增加use_align属性，默认使用原有的保留小tensor的完整chunk，可选的在合并时小tensor连续存放。

[iclementine]

一个小建议：等 cudnn_lstm_op 实现了 python 接口（functional 接口）之后改调用那个接口。

[guoshengCS]

Thanks. 这个会在cudnn_lstm_op支持functional后调整。

[iclementine]

建议把三个改 RNN 网络(SimpleRNN, GRU, LSTM) init 的内容统一成一个，放到 RNNMixin 里面。因为目前这三个类的 init 方法基本是相同的，只要用 cls 区分，稍作修改可能就可以。

RNNMixin 如有必要也可以改名 RNNBase, 因为它目前并不公开。

[guoshengCS]

Thanks.
RNNMixin 改名 RNNBase 并调整了SimpleRNN, GRU, LSTM对RNNBase的调用

[guoshengCS]

测试代码

import paddle
import paddle.fluid as fluid
import numpy as np
paddle.disable_static()
paddle.manual_seed(123)
x = paddle.randn((4, 10, 16))
x.stop_gradient = False
prev_h = paddle.randn((4, 4, 32))
prev_c = paddle.randn((4, 4, 32))
seq_len = paddle.to_tensor(np.array([10,6,8,5]))
mask = fluid.layers.sequence_mask(seq_len, maxlen=10, dtype=prev_h.dtype)
mask = paddle.unsqueeze(mask, [2])
rnn = paddle.nn.LSTM(16, 32, 2, direction="bidirectional")
y, (h, c) = rnn(x, (prev_h, prev_c), seq_len)
y = y * mask
loss = paddle.mean(y)
loss.backward()
optimizer = paddle.optimizer.Adam(learning_rate=0.1, parameters=rnn.parameters())
optimizer.step()
print(rnn[0].cell_fw.weight_hh)

多卡

import paddle
import paddle.fluid as fluid
import paddle.distributed as dist
import numpy as np

def train():
    paddle.disable_static()
    paddle.distributed.init_parallel_env()
    paddle.manual_seed(123)
    x = paddle.randn((4, 10, 16))
    x.stop_gradient = False
    prev_h = paddle.randn((4, 4, 32))
    prev_c = paddle.randn((4, 4, 32))
    seq_len = paddle.to_tensor(np.array([10,6,8,5]))
    mask = fluid.layers.sequence_mask(seq_len, maxlen=10, dtype=prev_h.dtype)
    mask = paddle.unsqueeze(mask, [2])
    rnn = paddle.nn.LSTM(16, 32, 2, direction="bidirectional")#, dropout=0.0)
    dp_layer = paddle.DataParallel(rnn)
    y, (h, c) = dp_layer(x, (prev_h, prev_c), seq_len)
    y = y * mask
    loss = paddle.mean(y)
    loss = dp_layer.scale_loss(loss)
    loss.backward()
    dp_layer.apply_collective_grads()
    optimizer = paddle.optimizer.Adam(learning_rate=0.1, parameters=rnn.parameters())
    optimizer.step()
    print(dp_layer._layers[0].cell_fw.weight_hh)

if __name__ == '__main__':
    dist.spawn(train, nprocs=2)

可以通过修改python api RNNBase.__init__代码中的self.could_use_cudnn = False对比cudnn与非cudnn版本

保存预测：

import paddle
import paddle.fluid as fluid
import paddle.distributed as dist
import numpy as np

class Net(paddle.nn.Layer):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = paddle.nn.LSTM(
            16, 32, 2, direction="bidirectional", dropout=0.1)

    def forward(self, input):
        return self.rnn1(input)

def train():
    paddle.disable_static()
    paddle.distributed.init_parallel_env()
    paddle.manual_seed(123)
    np.random.seed(123)
    x_np = np.random.rand(4, 10, 16).astype("float32")
    x = paddle.randn((4, 10, 16))
    x = paddle.to_tensor(x_np)
    x.stop_gradient = False
    prev_h = paddle.randn((4, 4, 32))
    prev_c = paddle.randn((4, 4, 32))
    seq_len = paddle.to_tensor(np.array([10,6,8,5]))
    mask = fluid.layers.sequence_mask(seq_len, maxlen=10, dtype=prev_h.dtype)
    mask = paddle.unsqueeze(mask, [2])
    rnn = Net()
    dp_layer = paddle.DataParallel(rnn)
    y, (h, c) = dp_layer(x)
    y = y * mask
    loss = paddle.mean(y)
    loss = dp_layer.scale_loss(loss)
    loss.backward()
    dp_layer.apply_collective_grads()
    optimizer = paddle.optimizer.Adam(learning_rate=0.1, parameters=rnn.parameters())
    optimizer.step()
    dp_layer.eval()
    y, (h, c) = dp_layer(x)
    print(y)
    dp_layer.train()
    if dist.get_rank() == 0:
        rnn = paddle.jit.to_static(rnn, [paddle.static.InputSpec(shape=[None, None, 16])])
        print(rnn.forward.concrete_program.main_program)
        paddle.jit.save(rnn, "./infer")

        paddle.enable_static()
        place = fluid.CPUPlace() if not fluid.is_compiled_with_cuda(
                ) else fluid.CUDAPlace(0)
        new_scope = fluid.Scope()
        with fluid.scope_guard(new_scope):
            exe = fluid.Executor(place)
            [inference_program, feed_target_names, fetch_targets] = (
                fluid.io.load_inference_model(
                    dirname="./", executor=exe, model_filename="infer.pdmodel", params_filename="infer.pdiparams"))
            results = exe.run(inference_program,
                              feed={feed_target_names[0]: x_np.astype("float32")},
                              fetch_list=fetch_targets)
            print(results)
            print(y.numpy() == results[0])  # eval与infer结果相同

if __name__ == '__main__':
    dist.spawn(train, nprocs=2)

[guoshengCS]

2020-09-17 12:23:16 ****************
2020-09-17 12:23:16 0. You must have one RD (XiaoguangHu01 or lanxianghit) and one TPM (saxon-zh or jzhang533 or swtkiwi or Heeenrrry or TCChenlong) approval for the api change for the management reason of API interface.
2020-09-17 12:23:16 1. You must have one TPM (saxon-zh or jzhang533 or swtkiwi or Heeenrrry or TCChenlong) approval for the api change for the management reason of API document.
2020-09-17 12:23:16 2. You must have one RD (zhiqiu (Recommend) or phlrain) approval for the api change for the opreator-related api without 'core.ops'.
2020-09-17 12:23:16 For more details, please click [https://github.com/PaddlePaddle/Paddle/wiki/paddle_api_development_manual.md]
2020-09-17 12:23:16 Related APIs: paddle.nn.SimpleRNN.flatten_parameters
2020-09-17 12:23:16 paddle.nn.LSTM.flatten_parameters
2020-09-17 12:23:16 paddle.nn.GRU.flatten_parameters
2020-09-17 12:23:16 
2020-09-17 12:23:16 There are 3 approved errors.
2020-09-17 12:23:16 ****************

• API修改还请 @XiaoguangHu01 @jzhang533 确认
• cudnn_lstm由于有输入输出使用同一变量的内容(dropout的state)，这里在动态图时使用了append_op未使用 core.ops，还请 @phlrain 确认

[jzhang533]

• SimpleRNN & SimpleRNNCell 有activation参数，需要相应的更新文档当中的公式和说明。
• num_layers=1, activation="tanh", direction="forward", dropout=0., time_major=False, 这几个参数的顺序建议做一下调整： num_layers, direction, time_major, dropout, activation 这样的顺序，可能会好一些，前三个都是跟shape相关的，放一起容易理解； activation是SimpleRNN独有的，放最后比较合适，同时，文档中可以建议用户用keyword arguments调用；
• direction='bidirectional` 时，我们只有concat这一种merge的方式，建议在这个参数的说明里说明一下。
• 1165 行有typo

[ZeyuChen]

对cudnn_lstm的Kernel名称需要讨论，避免将来cpu和gpu kernel部署的统一造成困扰。

[ZeyuChen]

如果这里op名字限制为cudnn_lstm, 对将来cpu kernel名字要统一是否会带来困扰，毕竟cudnn不是cpu关键字。
用lstm_v2之类的是否更合适？

[guoshengCS]

• SimpleRNN & SimpleRNNCell 有activation参数，需要相应的更新文档当中的公式和说明。
• num_layers=1, activation="tanh", direction="forward", dropout=0., time_major=False, 这几个参数的顺序建议做一下调整： num_layers, direction, time_major, dropout, activation 这样的顺序，可能会好一些，前三个都是跟shape相关的，放一起容易理解； activation是SimpleRNN独有的，放最后比较合适，同时，文档中可以建议用户用keyword arguments调用；
• direction='bidirectional` 时，我们只有concat这一种merge的方式，建议在这个参数的说明里说明一下。
• 1165 行有typo

@jzhang533 Done, thanks.

[jzhang533]

lgtm

[jzhang533]

lgtm

[XiaoguangHu01]

LGTM

[zhangting2020]

LGTM

[jzhang533]

lgtm

[zhiqiu]

flatten_parameters can add dygraph branch in the future.