【paddle.fleet】add auto parallel L1 implementations

[guru4elephant]

PR types

New features

PR changes

APIs

Describe

make distributed strategy easy to config with the following example

import paddle
import paddle.distributed.fleet as fleet
fleet.init(is_collective=True)
input_x = paddle.fluid.layers.data(name="x", shape=[32], dtype='float32')
input_y = paddle.fluid.layers.data(name="y", shape=[1], dtype='int64')

fc_1 = paddle.fluid.layers.fc(input=input_x, size=64, act='tanh')
fc_2 = paddle.fluid.layers.fc(input=fc_1, size=64, act='tanh')
prediction = paddle.fluid.layers.fc(input=[fc_2], size=2, act='softmax')
cost = paddle.fluid.layers.cross_entropy(input=prediction, label=input_y)
avg_cost = paddle.fluid.layers.mean(x=cost)

strategy = paddle.distributed.fleet.DistributedStrategy()
strategy.auto = True
optimizer = paddle.fluid.optimizer.SGD(learning_rate=0.01)
optimizer = fleet.distributed_optimizer(optimizer, strategy=strategy)
optimizer.minimize(avg_cost)

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[paddle-bot-old]

✅ This PR's description meets the template requirements!
Please wait for other CI results.

[gavin1332]

This keyword is a little hard to be understood. Could we name it 'reset_auto_flag' as what it exactly does?

[guru4elephant]

this is just a check, whether a user is using strictly auto configuration. Here is a strictly auto configuration.

strategy = DistributedStrategy()
strategy.auto = True

Here is a case that is not a strict auto configuration.

strategy = DistributedStrategy()
strategy.amp = True
strategy.auto = True

[gavin1332]

I think the global auto flag should not be shared with other strategies. For it is hard to be controlled. E.g. two strategies have auto capacity both, and one requires auto but the other not.

So I suggest to assign auto the highest priority, which shields other strategies . And if one strategy could be 'auto' separately, it should has its own auto configuration.

[gavin1332]

It is not good to see the the is_strict_auto decorator broadcast everywhere.

[wangxicoding]

Will auto parallel open all the options that can be turned on? I think some options should not be turned on by default, otherwise accuracy may be lost.
For example, DGC may require warm-up which needs to be set by users, otherwise accuracy may be lost. Auto parallel needs to ensure accuracy without loss.

[guru4elephant]

depends on the network condition. I think current dgc is not intelligent that we should improve to remove the warmup thing. The enable_strategy interface does not require all meta optimizers to be valid.

[guru4elephant]

I think the global auto flag should not be shared with other strategies. For it is hard to be controlled. E.g. two strategies have auto capacity both, and one requires auto but the other not.

So I suggest to assign auto the highest priority, which shields other strategies . And if one strategy could be 'auto' separately, it should has its own auto configuration.

The aim of auto is to make the interface easy to learn and use, not to show we can do different strategies automatically.
It is recommended to remove user defined configurations for each meta optimizer, otherwise meta optimizers should not be opened if user defined configurations exist.

[guru4elephant]

It is not good to see the the is_strict_auto decorator broadcast everywhere.

if you set localsgd with auto, the strict auto will not be valid.

strategy = DistributedStrategy()
strategy.localsgd = True
strategy.auto = True

[gavin1332]

LGTM

[wangxicoding]

LGTM