Enable program passes on Fleet APIs by sneaxiy · Pull Request #34955 · PaddlePaddle/Paddle

sneaxiy · 2021-08-17T03:44:20Z

PR types

New features

PR changes

Others

Describe

Enable program passes on Fleet APIs. Related doc PR: PaddlePaddle/docs#3854

paddle-bot-old · 2021-08-17T03:44:58Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

zhiqiu · 2021-08-26T06:33:15Z

python/paddle/distributed/fleet/base/fleet_base.py


+def apply_ir_passes(main_program, startup_program, config):
+    build_strategy = config._user_defined_strategy.build_strategy._copy()
+    if not paddle.fluid.core.globals()['FLAGS_apply_pass_to_program']:


You can use _global_flags() to replace paddle.fluid.core.globals()

zhiqiu · 2021-08-26T06:33:43Z

paddle/fluid/framework/ir/pass.cc

 #include "paddle/fluid/platform/mkldnn_helper.h"
 #endif

+DEFINE_bool(apply_pass_to_program, false,


Better put it in flags.cc

zhiqiu · 2021-08-26T06:43:59Z

python/paddle/fluid/optimizer.py

                            'bias': 0.0,
                            'bias_after_scale': False
                        })
+                    new_grad.op._set_attr(op_maker.kOpRoleAttrName(),


Use main_program._optimized_guard()?

Some operators are marked as kBackward. Not applicable to use main_program._optimized_guard().

sandyhouse

LGTM

JZ-LIANG

LGTM

zhiqiu

LGTM

lanxianghit · 2021-09-07T03:26:41Z

python/paddle/fluid/tests/unittests/CMakeLists.txt

    if(WITH_DISTRIBUTE)
        set_tests_properties(test_new_group_api PROPERTIES TIMEOUT 120)
-        set_tests_properties(test_pipeline PROPERTIES TIMEOUT 120)
+        set_tests_properties(test_pipeline PROPERTIES TIMEOUT 240)


尽量不要改单测时间，优先看一下是否有优化方法，否则CI的负担会很大

已对单测进行拆分，但拆分后仍有有一个超时时间为120s的单测test_ir_pass_pipeline。

单测超时问题请 @kolinwei 确认一下，谢谢

lanxianghit · 2021-09-07T03:28:16Z

paddle/fluid/platform/flags.cc

+ *          Fleet APIs.
+ * Note: Apply IR pass to program. Be only useful when using Fleet APIs.
+ */
+DEFINE_bool(apply_pass_to_program, false,


官网上关于flag的文档是否对应更新了？

PR链接：PaddlePaddle/docs#3854

… program_pass_fleet

lanxianghit

LGTM

XieYunshen

LGTM

lanxianghit

LGTM

* add fleet api for program pass * turn on apply pass for CI test * fix disable fuse_all_optimizer bug * try to test ci * fix CI * fill unspecified op role * fix fuse_allreduce * add ut to improve coverage * remove useless change * improve c++ coverage * follow some comments * test ir pass pipeline * update doc * reduce ut time again

wangxicoding · 2021-09-13T13:46:31Z

python/paddle/distributed/fleet/meta_optimizers/raw_program_optimizer.py

+        block = self.main_program.global_block()
+
+        last_backward_op_idx = None
+        for i, op in enumerate(reversed(gm_block.ops)):


应该是reversed(list(enumerate(gm_block.ops)))吧

Done in #35704 .

wangxicoding · 2021-09-13T13:48:59Z

python/paddle/distributed/fleet/meta_optimizers/raw_program_optimizer.py

+            return
+
+        gm_block._insert_op(
+            last_backward_op_idx,


last_backward_op_idx + 1吧，插入到最后一个last_backward_op_idx后面，optimize前面。此时last_backward_op_idx的默认值应该是-1

Done in #35704 .

wangxicoding · 2021-09-13T13:56:06Z

python/paddle/distributed/fleet/meta_optimizers/raw_program_optimizer.py

+                outputs={'Out': g},
+                attrs={
+                    'ring_id': ring_id,
+                    OP_ROLE_KEY: OpRole.Backward,


这个用Optimize可能更准确些，PE里面放到backward主要可以overlap。不过这个没啥影响就是了，pipeline有自己的gradient merge，其它的也不会用。

嗯，这个OpRole目前设置太随意了。或者说没有统一规范，依需求来设置...

* add fleet api for program pass * turn on apply pass for CI test * fix disable fuse_all_optimizer bug * try to test ci * fix CI * fill unspecified op role * fix fuse_allreduce * add ut to improve coverage * remove useless change * improve c++ coverage * follow some comments * test ir pass pipeline * update doc * reduce ut time again

sneaxiy added 2 commits August 17, 2021 03:40

add fleet api for program pass

2f5a3f6

turn on apply pass for CI test

8b51433

sneaxiy added 8 commits August 17, 2021 04:33

fix disable fuse_all_optimizer bug

e613119

try to test ci

0b1419e

fix CI

45b9c13

Merge develop

ee291dc

fill unspecified op role

55bc7d0

fix fuse_allreduce

7c3b3ad

add ut to improve coverage

9ba5727

remove useless change

c77cb8f

sneaxiy changed the title ~~[WIP] Program pass fleet~~ Enable program passes on Fleet APIs Aug 24, 2021

improve c++ coverage

4497db0

sneaxiy requested review from JZ-LIANG, XieYunshen, gongweibao, lanxianghit, sandyhouse, wangxicoding, zhhsplendid and zhiqiu and removed request for sandyhouse and wangxicoding August 25, 2021 08:51

zhiqiu reviewed Aug 26, 2021

View reviewed changes

sandyhouse previously approved these changes Aug 26, 2021

View reviewed changes

JZ-LIANG previously approved these changes Aug 31, 2021

View reviewed changes

follow some comments

156ceeb

sneaxiy dismissed JZ-LIANG’s stale review via 156ceeb September 6, 2021 02:55

sneaxiy dismissed sandyhouse’s stale review via 156ceeb September 6, 2021 02:55

zhiqiu previously approved these changes Sep 7, 2021

View reviewed changes

lanxianghit reviewed Sep 7, 2021

View reviewed changes

sneaxiy dismissed zhiqiu’s stale review via 3e20dc3 September 7, 2021 03:46

test ir pass pipeline

c123166

sneaxiy force-pushed the program_pass_fleet branch from 3e20dc3 to c123166 Compare September 7, 2021 03:47

sneaxiy added 2 commits September 7, 2021 12:19

update doc

09b7c58

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

c986244

… program_pass_fleet

sneaxiy mentioned this pull request Sep 7, 2021

Add FLAGS_apply_pass_to_program doc PaddlePaddle/docs#3854

Merged

lanxianghit previously approved these changes Sep 7, 2021

View reviewed changes

XieYunshen previously approved these changes Sep 7, 2021

View reviewed changes

reduce ut time again

f0a2a0a

sneaxiy dismissed stale reviews from XieYunshen and lanxianghit via f0a2a0a September 7, 2021 10:48

PaddlePaddle locked and limited conversation to collaborators Sep 7, 2021

PaddlePaddle unlocked this conversation Sep 7, 2021

sneaxiy requested a review from xiegegege September 8, 2021 00:03

lanxianghit approved these changes Sep 8, 2021

View reviewed changes

XieYunshen approved these changes Sep 8, 2021

View reviewed changes

xiegegege approved these changes Sep 8, 2021

View reviewed changes

sneaxiy merged commit 5f36988 into PaddlePaddle:develop Sep 8, 2021

sneaxiy deleted the program_pass_fleet branch September 8, 2021 03:58

wangxicoding reviewed Sep 13, 2021

View reviewed changes

sneaxiy mentioned this pull request Sep 14, 2021

Fix RawProgramOptimizer bug #35704

Merged

Conversation

sneaxiy commented Aug 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Describe

Uh oh!

paddle-bot-old bot commented Aug 17, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sandyhouse left a comment

Choose a reason for hiding this comment

Uh oh!

JZ-LIANG left a comment

Choose a reason for hiding this comment

Uh oh!

zhiqiu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lanxianghit left a comment

Choose a reason for hiding this comment

Uh oh!

XieYunshen left a comment

Choose a reason for hiding this comment

Uh oh!

lanxianghit left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sneaxiy Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

sneaxiy commented Aug 17, 2021 •

edited

Loading

sneaxiy Sep 14, 2021 •

edited

Loading