【multi precision】multi precision support (fp32 + fp16) by xingjing1 · Pull Request #9339 · PaddlePaddle/Paddle-Lite

xingjing1 · 2022-08-11T07:00:38Z

1.支持运行时混合精度
2.添加calib inplace算子
性能：

库体积影响：
+300KB

hong19860320 · 2022-08-23T06:47:18Z

cmake/configure.cmake

 if (LITE_ON_TINY_PUBLISH)
  add_definitions("-DLITE_ON_TINY_PUBLISH")
  add_definitions("-DLITE_ON_FLATBUFFERS_DESC_VIEW")
+  add_definitions("-DLITE_WITH_FLATBUFFERS_DESC")


可能影响模型加载耗时，需要评估下

目前测试的几个模型看来没有影响

shentanyue · 2022-08-23T08:30:04Z

lite/api/light_api.cc

-    const std::shared_ptr<const cpp::ProgramDesc>& program_desc) {
+    const std::shared_ptr<const cpp::ProgramDesc>& program_desc,
+    bool use_precision_low) {
  auto* exe_scope = &scope_->NewScope();


命名应该是 exec_scope

之前别人命名的，我这先不改了

shentanyue · 2022-08-23T08:32:12Z

lite/api/light_api.cc

+          if (op_type != "feed" && op_type != "fetch") {
+            if (place.precision == PRECISION(kFloat)) {
+              place.precision = PRECISION(kFP16);
+            } else if (place.precision == PRECISION(kAny)) {


判断可以合并
if (place.precision == PRECISION(kFloat) || place.precision == PRECISION(kAny)) {
function()...
}

shentanyue · 2022-08-23T08:38:12Z

lite/core/op_lite.cc

        op_type_, place.target, place.precision, place.layout);
-    VLOG(5) << "pick kernel for " << op_info()->Type() << " "
-            << place.DebugString() << " get " << ks.size() << " kernels";
+    // VLOG(5) << "pick kernel for " << op_info()->Type() << " "


可以清理下注释的代码

shentanyue · 2022-08-23T08:39:17Z

lite/core/op_lite.cc


+#ifdef LITE_ON_FLATBUFFERS_DESC_VIEW
+bool OpLite::Attach(const cpp::OpDescWrite &opdesc, lite::Scope *scope) {
+  // valid_places_.clear();


同，注释的代码不用的话，可以清理下

shentanyue · 2022-08-23T08:42:50Z

lite/core/program.cc

+  int low_precision = 1;
+  std::string old_op;
+
+  if (use_precision_low == true) {


这里的逻辑直接改成这样更直观点？
use_precision_low_ = use_precision_low;
low_precision = use_precision_low_ ? 1 : 0;

shentanyue · 2022-08-23T08:44:27Z

lite/core/program.cc

+#ifdef ENABLE_ARM_FP16
+      if (lite::DeviceInfo::Global().has_fp16() && low_precision == 1) {
+        if (op_type != "feed" && op_type != "fetch") {
+          if (place.precision == static_cast<PrecisionType>(1)) {


直接用PrecisionType::kFloat这类枚举值，不要用1、5这类hard code，下同

ok，这地方忘改了

shentanyue · 2022-08-23T08:49:16Z

lite/core/program.cc

+            place.precision = static_cast<PrecisionType>(5);
+          }
+        }
+        // transfer weight to fp16


下面的代码是不是跟WeightFP32ToFP16这个函数重复了？

是的，我看看调用一下

很多地方不一样，应该调用不了

shentanyue · 2022-08-23T08:50:24Z

lite/core/program.cc

+        //  kernels = op->CreateKernels({place});
+        //}
+        if (kernels.size() == 0 && place.target == TargetType::kARM) {
+          place.target = TargetType::kHost;


这里是为什么要改成TargetType::kHost呢？

因为有些fp16的kernal是host下的

虽然也是arm的代码

hong19860320 · 2022-08-23T08:52:11Z

lite/api/paddle_api.h

  // whether to load data from memory. Model data will be loaded from memory
  // buffer if model_from_memory_ is true.
  bool model_from_memory_{false};
+  PrecisionMode pre_mode_{LITE_PRECISION_NORMAL};


不用简写成pre_mode吧，直接用 precision_mode_

shentanyue · 2022-08-23T08:58:12Z

lite/kernels/arm/calib_inplace_compute.h

+
+template <DataLayoutType DLType>
+class CalibComputeFp32ToInt32
+    : public KernelLite<TARGET(kARM), PRECISION(kInt32), DLType> {


PRECISION(kInt32)->PRECISION(kFloat)

ok，这里用不上，之后删掉

shentanyue · 2022-08-23T08:58:32Z

lite/kernels/arm/calib_inplace_compute.h

+
+template <DataLayoutType DLType>
+class CalibComputeFp32ToInt64
+    : public KernelLite<TARGET(kARM), PRECISION(kInt64), DLType> {


PRECISION(kInt64)->PRECISION(kFloat)

ok，这里用不上，之后删掉

shentanyue · 2022-08-23T09:00:30Z

lite/operators/calib_inplace_op.h

+#ifdef LITE_ON_FLATBUFFERS_DESC_VIEW
+  bool AttachImpl(const cpp::OpDescWrite &opdesc, lite::Scope *scope) override;
+#endif
+  void *getparam() { return &param_; }


zhupengyang

LGTM

xingjing1 added 3 commits August 5, 2022 02:05

fix bugs

f25409e

fix bugs

9664807

fix bugs

4df8c61

xingjing1 requested review from hong19860320, liyancas and zhupengyang as code owners August 11, 2022 07:00

xingjing1 changed the title ~~Fix fp16 bugs~~ 【muti precision】fp16 and fp32 Aug 11, 2022

xingjing1 changed the title ~~【muti precision】fp16 and fp32~~ 【muti precision】muti precision support Aug 11, 2022

xingjing1 added 6 commits August 12, 2022 06:52

fix bugs

816cda1

fix bugs

683a21c

fix bugs

afbde41

fix bugs

a70815e

fix full_publish bugs

5a88ffb

fix bugd

4ee128d

hong19860320 reviewed Aug 23, 2022

View reviewed changes

xingjing1 added 2 commits August 23, 2022 07:22

fix bugs

c9c9dbb

fix bugs

abbec02

shentanyue reviewed Aug 23, 2022

View reviewed changes

hong19860320 reviewed Aug 23, 2022

View reviewed changes

shentanyue reviewed Aug 23, 2022

View reviewed changes

fix bugs

0f3da67

fix bugs

37f292f

zhupengyang approved these changes Sep 1, 2022

View reviewed changes

zhupengyang merged commit 5e35c5e into PaddlePaddle:develop Sep 1, 2022

zhupengyang changed the title ~~【muti precision】muti precision support~~ 【multi precision】multi precision support (fp32 + fp16) Sep 14, 2022

Conversation

xingjing1 commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhupengyang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xingjing1 commented Aug 11, 2022 •

edited

Loading