Skip to content

Fix allreducedep bug#16443

Merged
gongweibao merged 3 commits intoPaddlePaddle:developfrom
gongweibao:allreduceorder
Mar 26, 2019
Merged

Fix allreducedep bug#16443
gongweibao merged 3 commits intoPaddlePaddle:developfrom
gongweibao:allreduceorder

Conversation

@gongweibao
Copy link
Contributor

@gongweibao gongweibao commented Mar 25, 2019

Error:
image

Now:
image

性能对比:

bert step/s
before 0.86*32
after 1.47*32

test=develop
test=develop
auto outputs = op_desc->Outputs();
for (auto& o_it : outputs) {
for (auto& v : o_it.second) { // values
vars[v] = order;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you record the gradient but not parameter here? because the gradient's name may be changed after mem_opt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the op_role_var attr

const BuildStrategy &build_strategy,
ir::Graph *graph)
: member_(new ParallelExecutorPrivate(places)) {
VLOG(10) << "enter ParallelExecutor";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should remove this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@gongweibao gongweibao merged commit ec6519e into PaddlePaddle:develop Mar 26, 2019
@gongweibao gongweibao deleted the allreduceorder branch January 17, 2021 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants