Add a new op: paddle.linalg.multi_dot by zkh2016 · Pull Request #35224 · PaddlePaddle/Paddle

zkh2016 · 2021-08-27T11:46:41Z

PR types

New features

PR changes

OPs

Describe

Add the multi_dot to paddle linear algebra library:

Example:

>>> x0_data = np.random.random((3,2)).astype("float32")
>>> x1_data = np.random.random((2,4)).astype("float32")
>>> x1_data = np.random.random((4,5)).astype("float32")
>>> x0_data = np.random.random((3,2)).astype("float32")
>>> x1_data = np.random.random((2,4)).astype("float32")
>>> x2_data = np.random.random((4,5)).astype("float32")
>>> x0 = paddle.to_tensor(x0_data)
>>> x1 = paddle.to_tensor(x1_data)
>>> x2 = paddle.to_tensor(x2_data)
>>> out = paddle.linalg.multi_dot([x0, x1, x2])
>>> out
Tensor(shape=[3, 5], dtype=float32, place=CUDAPlace(0), stop_gradient=True,
       [[1.47845721, 1.24809611, 1.35647595, 2.11938763, 1.63790154],
        [1.39540243, 0.84205949, 1.09351981, 1.72926235, 1.36388993],
        [1.40417647, 1.54325879, 1.48727894, 2.30167985, 1.74950039]])
>>>

paddle-bot-old · 2021-08-27T11:46:53Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

…tGrad

xingfeng01 · 2021-08-31T07:18:14Z

paddle/fluid/operators/multi_dot_op.cc

+}
+
+/**
+ * @brief multi matrix dot by a chain order


加些注释

xingfeng01 · 2021-08-31T07:21:09Z

paddle/fluid/operators/multi_dot_op.cc

+  }
+};
+
+template <typename DeviceContext, typename T>


加些计算逻辑的注释

xingfeng01 · 2021-08-31T07:24:44Z

paddle/fluid/operators/multi_dot_op.cc

+    auto order = GetOrder(ins, ins_dims);
+    auto n = ins.size();
+    std::vector<framework::Tensor> results(n * n);
+    MatChainMul<DeviceContext, T>(ctx, ins, ins_dims, order, 0, n - 1, true,


是否可以使用前向结果？

可以设置成AsIntermediate作为中间结果，在前向的时候保存下来，放到后面优化的时候改。

xingfeng01 · 2021-08-31T07:25:19Z

paddle/fluid/operators/multi_dot_op.cc

+                  ops::MultiDotOpDoubleGradMaker<paddle::framework::OpDesc>,
+                  ops::MultiDotOpDoubleGradMaker<paddle::imperative::OpBase>);
+
+REGISTER_OP_CPU_KERNEL(


cpu版不支持fp16

xingfeng01 · 2021-08-31T07:27:17Z

python/paddle/tensor/linalg.py

+
+
+def multi_dot(x, name=None):
+    """


改写一下语言

xingfeng01 · 2021-08-31T07:29:03Z

python/paddle/fluid/tests/unittests/test_multi_dot_op.py

+paddle.enable_static()
+
+
+class TestMultiDotOp(OpTest):


加些注释说明下函数作用

xingfeng01 · 2021-08-31T07:30:02Z

python/paddle/fluid/tests/unittests/test_multi_dot_op.py

+            self.assertRaises(ValueError, paddle.multi_dot, [x5, x6, x7])
+
+
+class API_TestMultiDot(unittest.TestCase):


名字格式改一下

xingfeng01 · 2021-08-31T07:30:22Z

python/paddle/fluid/tests/unittests/white_list/check_shape_white_list.py

    'cvm',
    'cudnn_lstm',
    'rnn',
+    'multi_dot',


check 一下白名单

已经找zhupengyang确认过可以加

ZeyuChen

Need to optimize the comments of API and add more comments for the get order algorithm.

ZeyuChen · 2021-09-01T11:24:17Z

python/paddle/tensor/linalg.py

+
+def multi_dot(x, name=None):
+    """
+    Compute the dot product of tow or more matrix in a single function call, while automatically selecting the fastest evaluation order.


ZeyuChen · 2021-09-01T11:25:29Z

python/paddle/tensor/linalg.py

+
+    Supports inputs of float, double and float16 dtypes. This function does not support batched inputs.
+
+    Every tensor in x must be 2D, except for the first and last which may be 1D. if the first tensor is a 1D vector of shape(n, ) it is treated as row vector of shape(1, n), similarly if the last tensor is a 1D vector of shape(n, ), it is treated as a column vector of shape(n, 1).


Every tensor in x must be 2D
x要加单括号标明变量

ZeyuChen · 2021-09-01T11:26:53Z

python/paddle/tensor/linalg.py

+    If the first and last tensors are matrices, the output will be a matrix. However, if either is a 1D vector, then the output will be a 1D vector.
+
+    The cost of multiplying two matrices with shapes (a, b) and (b, c) is a * b * c. Given matrices A, B, C with shapes (10, 100), (100, 5), (5, 50) respectively, we can calculate the cost of different multiplication orders as follows:
+    - Cost((AB)C) = 10x100x5 + 10x5x50 = 7500


例子与pytorch一样，是否可以更换下？

ZeyuChen · 2021-09-01T11:28:05Z

python/paddle/tensor/linalg.py

+        B = paddle.to_tensor(B_data)
+        C = paddle.to_tensor(C_data)
+        out = paddle.multi_dot([A, B, C])
+        print(out.numpy().shape)


code examples 应该要给出结正确果，使用注释符号后给出

ZeyuChen · 2021-09-01T11:34:01Z

paddle/fluid/operators/multi_dot_op.cc

+
+  std::vector<uint64_t> m(n * n, 0);
+  std::vector<uint64_t> order(n * n);
+


对这一算法增加原理注释

python/paddle/fluid/tests/unittests/test_multi_dot_op.py

zhangting2020 · 2021-09-08T11:52:00Z

python/paddle/fluid/tests/unittests/test_multi_dot_op.py

+            x0 = fluid.data(name='x0', shape=[3, 2], dtype="float64")
+            x1 = fluid.data(name='x1', shape=[2, 3], dtype='float64')
+            result = paddle.multi_dot([x0, x1])
+            exe = fluid.Executor(fluid.CPUPlace())


这里fluid. Executor -> paddle.static.Executor.
凡是fluid.xx都修改下，参考下其他单测，或者是官网文档搜同名，用paddle.xx替换

zhangting2020 · 2021-09-08T11:53:04Z

python/paddle/tensor/linalg.py

 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at


这里包括下面，有一些和你的PR无关的修改，恢复成原样

这个恢复了，其他的是precommit自动格式化的。

zhangting2020 · 2021-09-08T11:55:59Z

python/paddle/tensor/linalg.py

+
+    Supports inputs of float, double and float16 dtypes. This function does not support batched inputs.
+
+    The input tensor in [x] must be 2D except for the first and last can be 1D. If the first tensor is a 1D vector of shape(n, ) it is treated as row vector of shape(1, n), similarly if the last tensor is a 1D vector of shape(n, ), it is treated as a column vector of shape(n, 1).


2D- > 2-D，1D->1-D

这里文档描述调整下换行，太长了

下面的也需要注意类似问题

xingfeng01 · 2021-09-13T04:27:25Z

LGTM

Xreki · 2021-09-13T05:48:54Z

paddle/fluid/operators/multi_dot_op.cc

+  framework::DDim out_dim;
+
+  if (first_dim.size() > 2) {
+    PADDLE_THROW(platform::errors::InvalidArgument(


直接用PADDLE_ENFORCE_GT(first_dim.size(), 2, ...

Xreki · 2021-09-13T05:49:49Z

paddle/fluid/operators/multi_dot_op.cc

+  }
+
+  auto last_dim = inputs_dims[n - 1];
+  if (last_dim.size() > 2) {


Xreki · 2021-09-13T05:56:00Z

paddle/fluid/operators/multi_dot_op.cc

+      const framework::OpKernelType& expected_kernel_type) const override {
+    return framework::OpKernelType(expected_kernel_type.data_type_,
+                                   tensor.place(), tensor.layout());
+  }


所有输入、输出的数据类型是一样的吧，这两个函数没有必要重写。

Xreki · 2021-09-13T05:58:05Z

paddle/fluid/operators/multi_dot_op.cc

+  framework::OpKernelType GetKernelTypeForVar(
+      const std::string& var_name, const framework::Tensor& tensor,
+      const framework::OpKernelType& expected_kernel_type) const {
+    if (framework::IsComplexType(expected_kernel_type.data_type_)) {


并没有注册复数类型的Kernel？

Xreki · 2021-09-13T05:59:05Z

paddle/fluid/operators/multi_dot_op.cc

+  void Make() override {
+    AddInput("X", "The input tensors of multi_dot operator.").AsDuplicable();
+    AddOutput("Out", "The output tensor of multi_dot operator");
+    AddAttr<bool>(


并没有实现MKLDNN类型的OpKernel，建议删除mkldnn所有相关的代码。

Xreki

LGTM

zhangting2020 · 2021-09-15T06:29:37Z

python/paddle/fluid/tests/unittests/test_multi_dot_op.py

+
+import unittest
+import numpy as np
+from op_test import OpTest, skip_check_grad_ci


skip_check_grad_ci 这个没有用到，记得删除

ok，下个PR我在去掉

ZeyuChen

LGTM

hong19860320

LGTM

zkh2016 added 7 commits August 23, 2021 08:16

add a fusion op: fused_residual_dropout_bias

bf318b8

simplify the code, andd opt reduce sum

507117a

resolve review comments and add comments to the code

462caa1

fused_dropout: optimize code structure to facilitate reuse

93e0638

Merge branch 'PaddlePaddle:develop' into develop

e2808ff

optimize code structure to facilitate reuse

036b430

Add a new op: paddle.linalg.multi_dot

6755aea

merge upstream, and resolved conflict

abc66c9

zkh2016 marked this pull request as draft August 30, 2021 02:49

fix the ci problem

1cc8aad

zkh2016 force-pushed the multi_dot branch from b3ae927 to 1cc8aad Compare August 30, 2021 06:30

zkh2016 added 4 commits August 30, 2021 10:30

modify the code according to the review comments

4d33b98

replace cudaMemcpy with TensorFromVector and TensorToVector in Dropou…

bd44d04

…tGrad

set dropout attr 'is_test':false

d2beab7

reduce the code to less than 1000 lines

40cd7ca

zkh2016 marked this pull request as ready for review August 31, 2021 07:14

xingfeng01 reviewed Aug 31, 2021

View reviewed changes

ZeyuChen reviewed Sep 1, 2021

View reviewed changes

zkh2016 added 8 commits September 2, 2021 02:42

add comment and modifying code according to the review comments

f342f00

optimize the code according to the review comments

5d2bbc8

use static_cast

934fcac

Merge branch 'develop' into multi_dot

09c55a8

fix the blocks for large shape

44610ea

fix the merge error

6c743f1

Merge remote-tracking branch 'upstream/develop' into develop

3133d33

merge upstream, and used new AlignedVector

1a83adb

dingjiaweiww previously approved these changes Sep 8, 2021

View reviewed changes

zhangting2020 reviewed Sep 8, 2021

View reviewed changes

replace fluid with paddle

eda910f

zkh2016 dismissed dingjiaweiww’s stale review via eda910f September 9, 2021 10:30

merge upstream

970ca84

zkh2016 requested a review from ZeyuChen September 10, 2021 11:25

xingfeng01 previously approved these changes Sep 13, 2021

View reviewed changes

zhupengyang previously approved these changes Sep 13, 2021

View reviewed changes

Xreki reviewed Sep 13, 2021

View reviewed changes

Merge branch 'develop' into multi_dot

ba0a92c

zkh2016 dismissed stale reviews from zhupengyang and xingfeng01 via ba0a92c September 13, 2021 07:29

zkh2016 and others added 4 commits September 13, 2021 10:55

Merge remote-tracking branch 'origin/develop' into multi_dot

433d65b

Merge branch 'PaddlePaddle:develop' into multi_dot

8def631

Merge branch 'multi_dot' of github.com:zkh2016/Paddle into multi_dot

335bed0

modify code according to the review

b647c1f

Xreki approved these changes Sep 15, 2021

View reviewed changes

xingfeng01 approved these changes Sep 15, 2021

View reviewed changes

zhupengyang approved these changes Sep 15, 2021

View reviewed changes

zhangting2020 reviewed Sep 15, 2021

View reviewed changes

zhangting2020 approved these changes Sep 15, 2021

View reviewed changes

dingjiaweiww approved these changes Sep 15, 2021

View reviewed changes

lanxianghit approved these changes Sep 15, 2021

View reviewed changes

ZeyuChen approved these changes Sep 15, 2021

View reviewed changes

hong19860320 approved these changes Sep 15, 2021

View reviewed changes

Xreki merged commit c9f7cff into PaddlePaddle:develop Sep 16, 2021

AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this pull request Sep 29, 2021

Add a new op: paddle.linalg.multi_dot (PaddlePaddle#35224)

99aff24

zkh2016 deleted the multi_dot branch August 19, 2022 04:05

		self.assertRaises(ValueError, paddle.multi_dot, [x5, x6, x7])


		class API_TestMultiDot(unittest.TestCase):


		Supports inputs of float, double and float16 dtypes. This function does not support batched inputs.

		Every tensor in x must be 2D, except for the first and last which may be 1D. if the first tensor is a 1D vector of shape(n, ) it is treated as row vector of shape(1, n), similarly if the last tensor is a 1D vector of shape(n, ), it is treated as a column vector of shape(n, 1).


		std::vector<uint64_t> m(n * n, 0);
		std::vector<uint64_t> order(n * n);


		Supports inputs of float, double and float16 dtypes. This function does not support batched inputs.

		The input tensor in [x] must be 2D except for the first and last can be 1D. If the first tensor is a 1D vector of shape(n, ) it is treated as row vector of shape(1, n), similarly if the last tensor is a 1D vector of shape(n, ), it is treated as a column vector of shape(n, 1).

Conversation

zkh2016 commented Aug 27, 2021

PR types

PR changes

Describe

Uh oh!

paddle-bot-old bot commented Aug 27, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZeyuChen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xingfeng01 commented Sep 13, 2021

Uh oh!