Fuse AllReduce by chengduoZH · Pull Request #15921 · PaddlePaddle/Paddle

chengduoZH · 2019-02-25T13:10:34Z

Code separated from #15497
Fix part of #16061

	ResNet	ResNet	Transformer	Transformer
	4cards（s/batch）	8cards（s/batch）	4cards（step/s）	8cards（step/s）
before	0.149	0.1935	3.276035	2.255209
fuse all reduce	0.0985	0.177	3.405812	2.346058

For ResNet:

export FLAGS_fuse_parameter_memory_size=131072
export FLAGS_fuse_parameter_groups_size=3

For Transformer:

export FLAGS_fuse_parameter_memory_size=131072
export FLAGS_fuse_parameter_groups_size=10

test=develop

… fuse_all_reduce

test=develop

gongweibao · 2019-03-04T11:36:35Z

paddle/fluid/framework/details/build_strategy.cc

      viz_pass->Set<std::string>("graph_viz_path", new std::string(graph_path));
    }

-    if (strategy.fuse_elewise_add_act_ops_) {


Why delete this?

It is not deleted but only moved to here.

gongweibao · 2019-03-04T11:38:13Z

paddle/fluid/framework/details/multi_devices_graph_pass.cc

 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
+#include "paddle/fluid/framework/details/multi_devices_graph_pass.h"


This should be after the system header files.

gongweibao · 2019-03-04T11:39:28Z

paddle/fluid/framework/details/multi_devices_graph_pass.cc

 }

+bool MultiDevSSAGraphBuilderBase::DealWithSpecialOp(ir::Graph *result,
+                                                    ir::Node *node) const {


Why add an empty function?

Just move AllReduceSSAGraphBuilder::DealWithSpecialOp to here.

gongweibao · 2019-03-04T12:05:29Z

paddle/fluid/framework/details/fuse_all_reduce_op_pass.cc

+
+      outputs.insert(outputs.end(), op_handle.Outputs().begin(),
+                     op_handle.Outputs().end());
+      // Remove Input


Do we need to remove the pointer in kGraphVars attr?

Only the origin all_reduce nodes should be removed from graph，and it is done in line 132.

test=develop

… fuse_all_reduce test=develop

gongweibao · 2019-03-06T02:33:08Z

Do we need a unittest to make sure the fused allreduce on all trainers hold the same order?

chengduoZH · 2019-03-06T05:09:08Z

@gongweibao all_reduce_deps_pass will make sure the order of allreduce.

… fuse_all_reduce

test=develop

… fuse_all_reduce test=develop

gongweibao · 2019-03-14T12:31:17Z

Another question: Where is the address alignment implemented?

gongweibao · 2019-03-14T11:22:58Z

paddle/fluid/framework/details/alloc_continuous_space_for_grad_pass.cc

+        auto iter = vars.find(p_g.second);
+        PADDLE_ENFORCE(iter != vars.end());
+        PADDLE_ENFORCE_NOT_NULL(iter->second->Var());
+        PADDLE_ENFORCE_EQ(iter->second->Var()->GetType(),


能否先略过其他的类型？

目前阶段要求类型必须是lodtensor，下一个阶段可能会做这方面的改进。

gongweibao · 2019-03-15T05:53:26Z

paddle/fluid/framework/details/alloc_continuous_space_for_grad_pass.cc

+    // Run Only Once Programs
+    for (size_t i = 0; i < local_scopes.size(); ++i) {
+      for (auto &op_desc : program_desc.Block(0).AllOps()) {
+        auto op = OpRegistry::CreateOp(*op_desc);


Maybe we can set a new tensor member to avoid fused-tensor be resized.

fused_all_reduce_op_handle has address check, so it is unnecessary.

… fuse_all_reduce test=develop

test=develop

… fuse_all_reduce test=develop

test=develop

… fuse_all_reduce test=develop

gongweibao

LGTM+++
Gradient fusion is very useful for multi-machines training communication.Thanks!

chengduoZH force-pushed the fuse_all_reduce branch from f40edfb to 779a2f5 Compare February 26, 2019 01:46

fuse all_reduce

67badd2

test=develop

chengduoZH force-pushed the fuse_all_reduce branch from 779a2f5 to 67badd2 Compare February 26, 2019 01:52

chengduoZH requested review from Yancey0623, gongweibao, panyx0718 and typhoonzero February 28, 2019 10:47

chengduozh added 2 commits March 4, 2019 15:39

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

650685c

… fuse_all_reduce

add fuse_parameter_groups_size

609290c

test=develop

gongweibao requested changes Mar 4, 2019

View reviewed changes

chengduoZH force-pushed the fuse_all_reduce branch from 05d55ec to 0ede8fc Compare March 5, 2019 02:15

Polish code

cd49a17

test=develop

chengduoZH force-pushed the fuse_all_reduce branch from 00646cc to cd49a17 Compare March 5, 2019 04:52

chengduozh added 2 commits March 5, 2019 16:59

Fix travis-ci

b53f468

test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

f6bfcaa

… fuse_all_reduce test=develop

chengduozh added 4 commits March 13, 2019 11:17

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

6a17a82

… fuse_all_reduce

Add SetGroupAccordingToLayers and SetGroupAccordingToGroupSize

3de3d35

test=develop

Add SetGroupAccordingToMemorySize

f98fcdf

test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

67eac9f

… fuse_all_reduce test=develop

gongweibao reviewed Mar 14, 2019

View reviewed changes

gongweibao reviewed Mar 15, 2019

View reviewed changes

chengduozh added 3 commits March 17, 2019 22:12

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

3bbdd6e

… fuse_all_reduce test=develop

fix multi_devices_graph

7040e86

test=develop

reset params_grads

601dd3c

test=develop

chengduoZH force-pushed the fuse_all_reduce branch from f6b6639 to 601dd3c Compare March 18, 2019 08:22

chengduozh added 2 commits March 18, 2019 16:24

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

10e744d

… fuse_all_reduce test=develop

Polish code

61f25d0

test=develop

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

dca0b3e

… fuse_all_reduce test=develop

panyx0718 approved these changes Mar 20, 2019

View reviewed changes

gongweibao approved these changes Mar 20, 2019

View reviewed changes

chengduoZH merged commit f26ba5b into PaddlePaddle:develop Mar 20, 2019

Conversation

chengduoZH commented Feb 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gongweibao commented Mar 6, 2019

Uh oh!

chengduoZH commented Mar 6, 2019

Uh oh!

gongweibao commented Mar 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gongweibao left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chengduoZH commented Feb 25, 2019 •

edited

Loading

gongweibao commented Mar 14, 2019 •

edited

Loading