Add basic hook classes for dygraph & implement reduce hook

[chenwhql]

PR types

New features

PR changes

Others

Describe

Add basic hook classes for dygraph & implement reduce hook

执行逻辑设计

1. 由前向VarBase拿到前向VariableWrapper, 通过VariableWrapper的接口注册LeafGradHook

   void AddGradVarLeafBackwardHook(std::unique_ptr<GradAccumulatorPostHook>&& hook) {
       auto leaf_hooks = GetGradVarLeafHooksSafely();
       leaf_hooks->add_backward_hook(std::move(hook));
   }
   

2. 反向执行Engine准备执行环境时将hook关联到GradientAccumulator

   if (var->HasLeafHooks()) {
     VLOG(3) << "Grad variable wrapper (" << var->Name() <<") has leaf grad hooks.";
       PADDLE_ENFORCE_NE(
           var->HasGradNode(), true,
           platform::errors::PermissionDenied(
               "Only leaf Tensor's gradient can append hook to Gradientaccumulator."));
       accumulator->SetPostHooks(var->GetLeafHooks());
   }
   

3. 当反向执行梯度累加完成时，执行关联的hook

// 无需梯度累加时
for (auto* accumulator : no_need_run_accumulators_) {
 if (accumulator->HasPostHooks()) {
    accumulator->CallPostHooks();
  }
}
// 需要梯度累加时
void IncreaseCurCnt() {
  ++cur_cnt_;
  VLOG(3) << "IncreaseCurCnt: cur_cnt " << cur_cnt_ << ", ref_cnt " << ref_cnt_;
  // After all tmp gradient being accumulated to grad var, run hooks
  if (AccumulateCompleted() && HasPostHooks()) {
    CallPostHooks();
  }
}

简单hook示例

// need impl in C++
// w is a parameter std::shared_ptr<VarBase>
auto w_shared = w->SharedVar();
w_shared->->AddGradVarLeafBackwardHook(
      std::unique_ptr<LambdaGradAccumulatorPostHook>(
          new LambdaGradAccumulatorPostHook([=](VariableWrapper* grad) {
            auto* grad_tensor =
                grad->MutableVar()->GetMutable<framework::LoDTensor>();
            for (int i = 0; i < grad_tensor->numel(); ++i) {
              grad_tensor->mutable_data<float>(place)[i] *= 2.0;
            }
          })));

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[zhwesky2010]

May be it can call 'leaf_var_hooks_' , and can be better distinguished from 'backward_hooks_' . After all, both of them are hooks for backward. Isn't 'backward_hooks_' here for Allreduce/Reduce only?

[chenwhql]

1. my opinion: the class name LeafVarHookPackage alreaady hold theleaf var info, the hooks in LeafVarHookPackage are leaf_var_hooks_ , using long member name cause information redundancy and also make the interface name longer, such as LeafVarHookPackage.add_leaf_var_hook()
2. backward_hooks_ mean the hooks of whole backward process, because it relay on leaf var, so we can only put it here now, may be we should add AccumulateGrad dummy OpNode and move backward_hooks_ outside, I wiil perfect the comments here

[chenwhql]

And backward_hooks_ may not only used for Allreduce/Reduce, we should keep scalability here

[zhwesky2010]

Ok

[zhwesky2010]

Here call backward_hooks_, how about when AccumulateCompleted, first call_hooks_ , then gradient_accumulation between batch, last call backward_hooks_ .
So We must have two function: CallPostHooks, and CallBackwardPostHooks. And this can changed after this PR merged.

[chenwhql]

yes, I agree

[zhwesky2010]

Ok

[zhwesky2010]

LGTM

[hutuxian]

LGTM for reduce hook part