Skip to content

[bug fix] avoiding multiple initialization of gloo for fleet in dygraph mode #27706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guru4elephant merged 16 commits into from
Sep 30, 2020
Merged

[bug fix] avoiding multiple initialization of gloo for fleet in dygraph mode #27706

Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
831182f
add double grad for expand, test=develop
Sep 8, 2020
8d5c17d
update, test=develop
Sep 8, 2020
d0905a6
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 10, 2020
9181c2d
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 10, 2020
d9fcbe6
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 15, 2020
c0616ca
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 16, 2020
65c2cbf
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 18, 2020
f2d426f
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 21, 2020
640f16d
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 21, 2020
33ee9b3
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 22, 2020
6d703d7
Merge branch 'develop' of https://github.com/sandyhouse/Paddle into d…
Sep 28, 2020
102354e
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 29, 2020
02cd424
bug fix for fleet dygraph, test=develop
Sep 29, 2020
0db0cae
python/paddle/distributed/fleet/base/role_maker.py
Sep 29, 2020
a57574c
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
Sep 29, 2020
7bc86fc
update, test=develop
Sep 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion python/paddle/distributed/fleet/base/role_maker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import warnings
from multiprocessing import Process, Manager

import paddle
import paddle.fluid as fluid
from paddle.distributed.fleet.base.private_helper_function import wait_server_ready

Expand Down Expand Up @@ -869,7 +870,8 @@ def _generate_role(self):
else:
self._collective_env()
self._role_is_generated = True
self._gloo_init()
if not paddle.fluid.framework.in_dygraph_mode():
Copy link
Member

@guru4elephant guru4elephant Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems in dygraph mode, we do not need to do gloo_init, but actually it is initialized in other places. In general, this is not a good design, and should be refactored with init_parallel_env in the future.

self._gloo_init()


class UserDefinedRoleMaker(PaddleCloudRoleMaker):
Expand Down