Replace Tensor by LoDTensor in each operator.

[qingqing01]

The following comments are addressing three problems:

• Necessary change to LoDTensor
• Replace uses of Tensor into LoDTensor in operators who don't need LoD
• How to pass LoD from operators' inputs to outputs

LoDTensor

The definition of LoDTensor is as follows:

class LodTensor {
 public:
  LoDTensor() {}
  void set_tensor(Tensor* tensor) { tensor_ = tensor; }
 private
  Tensor* tensor_;
  Lod lod_;
};

The code snippet shows how we use it in operators:

void InferShape(const framework::InferShapeContext &ctx) const override {
    auto tensor_x = ctx.Input<LoDTensor>("X")->tensor();
    auto tensor_y = ctx.Input<LoDTensor>("Y")->tensor();
    PADDLE_ENFORCE_EQ(tensor_x.dims(),
                      tensor_y.dims(),
                      "Two inputs of Add Op's dimension must be same.");
    auto* lod_tensor_out =   ctx.Output<LodTensor>("Out");
    Tensor* out = new Tensor();
    out->Resize(tensor_x->dims());
    lod_tensor_out->set_tensor(out);
  }
};

We see that LoDTensor::set_tensor is kind of ugly because the caller needs to destruct the tensor at right time.

A solution is to make LoDTensor owns the tensor and delete it at destruction time:

class LodTensor {
 public:
  LoDTensor() : tensor_(new Tensor()) {}
  ~LoDTensor() { delete tensor_; }
  //  删掉set_tensor接口。
  // void set_tensor(Tensor* tensor) { tensor_ = tensor; } 
 private
  Tensor* tensor_;
  Lod lod_;
};

---

[qingqing01]

对不用感知LoD信息Operator替换Tensor

对于fc、mul、elementwise-mul/add等operators在写OpKernel时只需要用Tensor，而不需要LoDTensor，为了使得这些operators不感知LoDTensor，替换方法步骤：

1. Variable的Get接口可以从LoDTensor里获取到Tensor，对Get接口特化Tensor模板：

class Variable {
  template <>
  const Tensor& Get<Tensor>() const {
    if (IsType<LoDTensor>()) {
      auto lod_t = static_cast<const LoDTensor*>(holder_->Ptr());
      return *lod_t->tensor();
    } else {
      PADDLE_ENFORCE(IsType<Tensor>(),
                     "Variable must be type LoDTensor or Tensor");
      return *static_cast<const Tensor*>(holder_->Ptr());
    }
    return;
  }
}

2. 在Operator InferShape里input、output不同方式获取Tensor

class MulOp : public framework::OperatorWithKernel {
 protected:
  void InferShape(const framework::InferShapeContext &ctx) const override {
    // input依然通过 Input<Tensor>获取Tensor
    auto x_dims = ctx.Input<Tensor>("X")->dims();
    auto y_dims = ctx.Input<Tensor>("Y")->dims();
    // ...
    // output需要通过Output<LoDTensor>先得到LoDTensor, 并在第一次调用时在variable中创建LoDTensor.
    // 在通过tensor()接口得到Tensor
    ctx.Output<LoDTensor>("Out")->tensor()->Resize(
        {x_mat_dims[0], y_mat_dims[1]});
  }
};

3. 在class ExecutionContext里重写Output函数, 使得OpKernel的计算里不用感知LoDTensor，依然保持和原来的代码一致。

template <typename Place, typename T>
class MulKernel : public framework::OpKernel {
 public:
  void Compute(const framework::ExecutionContext& context) const override {
    const Tensor* x = context.Input<Tensor>("X");
    const Tensor* y = context.Input<Tensor>("Y");
    Tensor* z = context.Output<Tensor>("Out");
    // ...
  }
};

[qingqing01]

如何在Operator里传递LoD

1. LoD传递原因

用下面连接方式来举例，op1、op3是sequence计算op, op2是非sequence计算op。op2得传递输入的LoD信息到输出，否则op3获取不到LoD信息。

op1(sequence) -> op2(non_sequence) -> op3(sequence)

2. 如何传递

需要解决问题：

• forward op需要传递LoD, backward op一般来说不需要传递LoD。
• 由于Operator可能存在多个输入、多个输出，需要每个Op来决定传递哪个输入的LoD到哪个输出。
• 在哪里传递。

由于前两个问题，比较难把LoD的传递隐藏起来，不让非sequence op感知。下面方法通过显示的operator InferShape来传递，需要每个Op显示调用传递函数。

首先在class InferShapeContext里定义传递函数：

class InferShapeContext {
  void CopyLoD(const std::string& in_name, const std::string& out_name) {
    PADDLE_ENFORCE(InputVar(in_name)->IsType<LoDTensor>(),
                   "The Input(%s) must be LoDTensor.", in_name);
    PADDLE_ENFORCE(OutputVar(out_name)->IsType<LoDTensor>(),
                   "The Output(%s) must be LoDTensor.", out_name);
    Output<LoDTensor>(out_name)->set_lod(Input<LoDTensor>(in_name)->lod());
  }
}

然后在operator InferShape里调用：

class MulOp : public framework::OperatorWithKernel {
 protected:
  void InferShape(const framework::InferShapeContext &ctx) const override {
    auto x_dims = ctx.Input<Tensor>("X")->dims();
    auto y_dims = ctx.Input<Tensor>("Y")->dims();
    // ...
    ctx.Output<LoDTensor>("Out")->tensor()->Resize(
        {x_mat_dims[0], y_mat_dims[1]});
   // 将输入X的LoD传递给输出Out
    ctx.CopyLoD(/*in = */ "X", /* out = */ "Out");
  }
};

[qingqing01]

related to #3717

[wangkuiyi]

Two cents from my side:

Derivation solves the first two problems

class LoDTensor {
 private:
   LoD* lod_;
   Tensor* tensor_;
};

into

class LoDTensor : public Tensor {
 private:
  LoD* lod_;
};

In this way, we can save the deletion of tensor_ in the destructor of LoDTensor.

Also, we can use the return value of context.Input<LoDTensor>("X") as a Tensor pointer.

I was wrong before

I chose the embedding over the inheritance in the definition of LoDTensor for a reason. But that reason is wrong.

I thought that we will have multiple types of Tensors in the future -- SparseTensor, DenseTensor, RowSparseTensor, etc. If we derive LoDTensor from them, we will have LoDSparseTensor, LoDDenseTensor, LoDRowSparseTensor, etc. But I was wrong, because embedding doesn't save us from this. By embedding LoD and Tensor, we still need a type Tensor, which should be the parent class of SparseTensor, DenseTensor, etc. As long as we have such a base class, we can have only on LoDTensor -- the one derived from this base class Tensor. So embedding and inheritance are basically the same. The only difference is that in C++, inheritance allows us to access methods of Tensor from LoDTensor. This is the solution to two of three problems addressed in this PR.

[qingqing01]

Need to handle how to pass LoD from operators' inputs to outputs, so reopen this issue.

[Superjomn]

This works fine, and the CopyLoD can't be hidden from users, because which output copy which input's LoD should be specified by op developer, even the FC op.

[luotao1]

CopyLoD对大多数op可以隐藏，对FC op来说，output的LoD信息和input的完全一样。只有seq相关的op才会做修改。

[Superjomn]

那是只有一个 input, 一个 output 的时候的情况吧， 如果有多个 input 或者多个 output， 那对应关系总需要指定吧。

比如，即使不需要修改 LoD，也需要知道具体的一个output复制哪个input的LoD。

我觉得对于 op developer，这些不隐藏会更好些，不完善的自动化会让 developer 思维懒惰而忽略了 LoD 的机制，写出不完善的代码，出现问题也比较难 debug。

[luotao1]

对于目前的非seq Op来说，input的LoD信息都是一样的。目前还未碰到过不同input的LoD信息不一样的情况，除了Recurrent_op。所以只需要在Recurrent Op中特别指明即可。对大多数op来说，不需要感知这块。

[Superjomn]

add_two 接入两个 input, 有不同的LoD， output 复制哪个呢？

[luotao1]

add_two 接入两个 input, 有不同的LoD， output 复制哪个呢

这种情况目前在实际应用中未碰到过。这种情况应该出现么 @lcy-seso

且对于Recurrent op来说：

• 如果要拆循环，那么不同input的lod信息必须严格一样。因为如果不一样，那么也不能保证sample的数目一样，这样后续就没法接着做计算了。

对于Recurret Op来说，必须要加一些CHECK来保证程序的稳定性。目前制定的规则如下：
http://doc.paddlepaddle.org/develop/doc_cn/howto/deep_model/rnn/recurrent_group_cn.html#训练流程的使用方法

这种稳定性和灵活性在原来的版本中无法兼容。当时选择的是稳定性，限制input的lod信息必须严格一样，这样出错也比较好排查。一旦input的lod信息可以不一样，后续的排查工作会非常困难。