Support control flow in DataParallel by ForFishes · Pull Request #31625 · PaddlePaddle/Paddle

ForFishes · 2021-03-15T06:05:47Z

PR types

New features

PR changes

Others

Describe

Support control flow in DataParallel
DataParallel exposes the find_unused_parameters interface, which is used to detect whether there are unused parameters in the network. This is a compatible upgrade.

This PR also did the following work:

Improve the overall error message, and expose it earlier if it does not meet the execution requirements of DataParallel. For example: multiple forward calculations and one reverse calculation.
Support control flow branching.

For the same step, the gradients of different cards are inconsistent.
Different steps and gradients of the same card are inconsistent.
In the case of gradient accumulation, the above two cases.

Support sync parameters and buffers in reducer.

paddle-bot-old · 2021-03-15T06:05:58Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

wangxicoding · 2021-03-18T07:43:43Z

paddle/fluid/imperative/reducer.cc

-          // TODO(liuyuhui) support XPU set constant
-          VLOG(3) << "XPU doesn't support set_constant";
-        }
+      if (platform::is_xpu_place(group_tensor.place())) {


移到下面的else里面？

wangxicoding · 2021-03-18T07:47:10Z

paddle/fluid/imperative/reducer.cc

+      VLOG(3) << "Local used vars : "
+              << string::join_strings(local_used_vars_, ',');
+// TODO(liuyuhui): support bckl in using TensorToVector/TensorFromVector
+#if defined(PADDLE_WITH_NCCL)


把RCCL给加上？

wangxicoding · 2021-03-18T07:59:20Z

paddle/fluid/imperative/reducer.cc

    });
 #elif defined(PADDLE_WITH_RCCL) || defined(PADDLE_WITH_NCCL)
-    FusedAllReduceSchedule(run_order, group);
+    FusedAllReduceSchedule(run_order, group, next_group_);


BKCL的给漏掉了

BKCL的代码在上面，这里只有NCCL。

BKCL的没有加next_group_ =.=

额，所以单测挂了。已修改。

wangxicoding · 2021-03-18T08:15:49Z

paddle/fluid/imperative/reducer.cc


+  if (find_unused_vars_) {
+// TODO(liuyuhui) support xpu about Tensorcopy
+#if defined(PADDLE_WITH_NCCL)


RCCL给加上？

wangxicoding · 2021-03-18T08:16:29Z

python/paddle/distributed/fleet/base/distributed_strategy.py

+            strategy.find_unused_parameters = True
+        """
+
+        return self.strategy.sync_batch_norm


sync_batch_norm了

wangxicoding · 2021-03-18T08:17:22Z

python/paddle/distributed/fleet/base/fleet_base.py

-            last_comm_group_size_MB)
+            last_comm_group_size_MB,
+            find_unused_parameters=self._user_defined_strategy.
+            find_unused_parameters)


这个是自动排版了，参数并没有重复。

wangxicoding · 2021-03-18T10:06:48Z

paddle/fluid/imperative/bkcl_context.cc

 }

+void BKCLParallelContext::SynchronizeCompute() {
+  // TODO(wangxi16): [Performance optimize] Maybe need to put Wait and


这个TODO和上面那个TODO帮忙删一下0.0

vslyu · 2021-03-19T08:33:55Z

paddle/fluid/imperative/parallel_context.h

  virtual void WaitComm(int ring_id) = 0;

+  // synchorize compute stream
+  virtual void SynchronizeCompute() = 0;


亮哥，顺路把SynchronizeCompute在bkcl_context里面也加下哈～

已经添加。

paddle/fluid/imperative/reducer.cc

vslyu · 2021-03-19T09:26:34Z

paddle/fluid/imperative/reducer.cc

  }

+  if (find_unused_vars_) {
+// TODO(liuyuhui) support xpu about Tensorcopy


TensorCopy不是已经支持XPU的place了吗～

好的，不过由于前面那两个还不支持。所以还是下次xpu添加的时候顺便删一下吧。

vslyu · 2021-03-22T05:38:40Z

python/paddle/fluid/tests/unittests/test_parallel_dygraph_control_flow.py

+class TestFleetDygraphControlFlowSame(TestDygraphControlFlowSame):
+    def _setup_config(self):
+        self._sync_mode = False
+        self._nccl2_mode = True


Add new UT case about bkcl_mode for XPU?

目前还不支持xpu。

vslyu · 2021-03-22T05:39:26Z

python/paddle/fluid/tests/unittests/test_parallel_dygraph_unused_variables.py

        self._dygraph = True

-    def test_mnist(self):
+    def test_net(self):


Add new UT case about bkcl_mode for XPU?

同上，还不支持xpu。

hutuxian · 2021-03-22T12:01:47Z

paddle/fluid/imperative/reducer.cc

-    auto dtype = var->DataType();
-    auto place = var->Place();
+    const auto dtype = var->DataType();
+    const auto place = var->Place();


Use const&?

hutuxian · 2021-03-22T12:05:02Z

paddle/fluid/imperative/reducer.cc

 void Reducer::PrepareForBackward(
    const std::vector<std::shared_ptr<imperative::VarBase>> &outputs) {
-  VLOG(3) << "start reseting count..";
+  VLOG(3) << "start forward and reset count.";


Why start forward? This function is called after forward, right?

hutuxian · 2021-03-23T02:39:12Z

paddle/fluid/imperative/reducer.cc

-      auto tensor =
-          var_warpper->MutableVar()->GetMutable<framework::LoDTensor>();
+      auto var_base = vars_[var_index]->GradVarBase();
+      auto tensor = var_base->MutableVar()->GetMutable<framework::LoDTensor>();


what's the difference?

hutuxian · 2021-03-23T02:49:14Z

paddle/fluid/imperative/reducer.cc


-void Reducer::FusedAllReduceSchedule(int run_order, Group &group) {
+void Reducer::FusedAllReduceSchedule(const int run_order, Group &group,
+                                     const int curr_group_index) {


why add curr_group_index？

just for print log

hutuxian · 2021-03-23T02:57:23Z

paddle/fluid/imperative/reducer.h

+  std::vector<bool> vars_marked_ready_;
+
+  // Following variables are to help control flow,
+  std::vector<int> local_used_vars_;


Could we find better name, or add more comment to explain this vector.
Because after all-reduce, what it denotes is global_used_vars instead of local_used_vars

hutuxian

LGTM

XiaoguangHu01

LGTM

jzhang533

lgtm for find_unused_parameters api exposed in python

ForFishes added 4 commits March 11, 2021 15:17

fix reducer

e4f2752

add utest

5bf0086

fix untest

449e090

support control flow

26d1b48

ForFishes added 10 commits March 15, 2021 21:19

support control flow

a798b97

fix utest

45fed37

fix bug

32b2ab5

add utest

c6c55f4

fix utest

930a58f

fix untest

7fbb5a2

fix bckl

2583f1c

fix utest

70716fc

support fleet

2143ff5

fix comment

7706e9d

wangxicoding reviewed Mar 18, 2021

View reviewed changes

fix fleet api

a4212df

wangxicoding reviewed Mar 18, 2021

View reviewed changes

ForFishes added 12 commits March 18, 2021 22:55

fix utest

dae49d7

fix bug

2d4f4d5

fix comment

cc7fb0c

fix bug

ac3e2c0

fix files

3f541fe

fix utest

5f3a3fd

fix utest

97d08eb

add sync buffer and param

a356f61

add sync param

42e0c11

fix bug

39dab1a

fix utest

e4f5a4d

fix cmake

a807956

ForFishes added 11 commits March 20, 2021 15:19

fix coverage

8f67e1f

add gradient check

be4072b

fix coverage

93bac9a

fix utest

1ce2b2b

add test for nonevar && find_unused_parameters

e619554

supoort xpu in sync_parameters_buffers

037c225

fix xpu

8f25efa

fix ctest

9997354

add test for dataparallel

0dee9fd

fix utest

8ad02cc

fix comment

85a758b

vslyu reviewed Mar 22, 2021

View reviewed changes

hutuxian reviewed Mar 22, 2021

View reviewed changes

fix small bug for redcuer

7a9c0a9

hutuxian reviewed Mar 23, 2021

View reviewed changes

ForFishes added 3 commits March 23, 2021 13:09

fix parallel_dygraph_dataparallel

c3af47e

solve compute stream & comm stream conflict

d934171

fix the bug of sparse embedding

39caee0

hutuxian approved these changes Mar 30, 2021

View reviewed changes

XiaoguangHu01 approved these changes Mar 31, 2021

View reviewed changes

jzhang533 approved these changes Mar 31, 2021

View reviewed changes

ForFishes merged commit 8460698 into PaddlePaddle:develop Apr 1, 2021

ForFishes deleted the support_control_flow branch April 1, 2021 08:11

Conversation

ForFishes commented Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Describe

Uh oh!

paddle-bot-old bot commented Mar 15, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hutuxian Mar 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hutuxian left a comment

Choose a reason for hiding this comment

Uh oh!

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Uh oh!

jzhang533 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

ForFishes commented Mar 15, 2021 •

edited

Loading

hutuxian Mar 22, 2021 •

edited

Loading