Improve mobilenetv2 INT8 performance by using INT8 relu as post-op by lidanqing-vv · Pull Request #17570 · PaddlePaddle/Paddle

lidanqing-vv · 2019-05-22T10:25:50Z

This PR improved mobilenetv2 INT8 performance with good accuracy.

We have to use relu instead of brelu as the post-op in INT8 conv2d kernel , because INT8 brelu as a post-op is not enabled in mkldnn v01.8. I add TODO and comments of what will be changed when v0.20 is enabled.

The performance of mobilenetv2 with this PR is follows:

INT8/FP32	Top1 Accuracy	Performance
FP32	71.90%	X
INT8	71.43%	1.92 X

test machine: Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz, 14 Cores
paddle_num_threads 14

Performance of all the int8 models on CLX and SKX will be delivered by this Friday.

test=develop

lidanqing-vv · 2019-05-22T10:29:06Z

Because I wanted to remove the conflicted commits in PR17546 and I did something like renaming. Somehow previous PR was closed automatically.
I am refering to previous PR's reviews and commit new changes according to that PR17546 reviews.

luotao1 · 2019-05-22T10:39:52Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

        }
      } else if (!force_fp32_output) {
-        if (fuse_relu) {
+        if (fuse_relu || fuse_brelu) {


How about using a variable to compute fuse_relu || fuse_brelu at first?
bool xxx = fuse_relu || fuse_brelu

How about using a variable to compute fuse_relu || fuse_brelu at first?
bool xxx = fuse_relu || fuse_brelu

Yes. I agree with you. I will change. It is about using uint8_t or int8_t

luotao1 · 2019-05-22T10:42:17Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

-                                       output_shift_scale, sum_scale, is_test);
+        conv_pd = ConvFwdPrimitiveDesc(
+            src_md, weights_md, dst_md, strides, paddings, mkldnn_engine,
+            fuse_relu || fuse_brelu, fuse_residual_conn, false, 0.0,


false, 0.0, please see 'It will be removed once the int8 is enabled.' #17130 (comment)

false, 0.0, please see 'It will be removed once the int8 is enabled.' #17130 (comment)

Hi, @luotao1 I changed but do not merge and do not do further review. Previous PR worked, but after merging some updates, there seems some error. Let me test locally first.

I just found CreatePostOps made problems. As in https://github.com/lidanqing-intel/Paddle/commits/develop-mobilnetv2, only last commit makes the accuracy 0. All previous commits are giving good accuracy. I will fix during the day, also will talk with @wojtuss.

test=develop change the "fuse_relu||fuse_brelu" to "unsigned_output" test=develop

…nabled in mkldnn v0.18 test=develop

test=develop

luotao1 · 2019-05-28T09:04:39Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

-            mkldnn_engine, fuse_relu, fuse_residual_conn, false /*fuse_brelu*/,
-            0.0 /*fuse_brelu_threshold*/, output_shift_scale, sum_scale,
-            is_test);
+            mkldnn_engine, fuse_relu || fuse_brelu /*fuse_relu*/,


470: fuse_relu || fuse_brelu-》unsigned_output and why notes /*fuse_relu*/

@luotao1 Hi, as in comments line 460. When mkldnn v0.20 is enabled, the INT8 brelu post-ops will be supported, I will substitute with the code noted in /**/.

I think we should not substitute fuse_relu || fuse_brelu with unsigned_output, in fact, we should use fuse_relu only at this position. We use fuse_relu || fuse_brelu now because in INT8 inference and achieve good accuracy, just because relu and brelu both give unsigned in inference. But the correct way is the code in /**/.

Got it, when we can update to mkldnn v0.20?

Got it, when we can update to mkldnn v0.20?

mkldnn v0.20 code freeze is 7th of June, but they need time for testing and release date is by 28th of June.

luotao1 · 2019-05-28T09:05:13Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

-                                       output_shift_scale, sum_scale, is_test);
+        conv_pd = ConvFwdPrimitiveDesc(
+            src_md, weights_md, dst_md, strides, paddings, mkldnn_engine,
+            fuse_relu || fuse_brelu /*fuse_relu*/, fuse_residual_conn,


477: fuse_relu || fuse_brelu-》unsigned_output and why notes /*fuse_relu*/

luotao1 · 2019-05-28T09:06:06Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

+        conv_pd = ConvFwdPrimitiveDesc(
+            src_md, weights_md, dst_md, strides, paddings, mkldnn_engine,
+            fuse_relu || fuse_brelu /*fuse_relu*/, fuse_residual_conn,
+            false /*fuse_brelu*/, fuse_brelu_threshold, output_shift_scale,


do you need false /*fuse_brelu*/ again since you have fuse_relu || fuse_brelu before?

luotao1 · 2019-05-28T09:07:00Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

-            0.0 /*fuse_brelu_threshold*/, output_shift_scale, sum_scale,
-            is_test);
+            mkldnn_engine, fuse_relu || fuse_brelu /*fuse_relu*/,
+            fuse_residual_conn, false /*fuse_brelu*/, fuse_brelu_threshold,


do you need false /*fuse_brelu*/ again since you have fuse_relu || fuse_brelu before?

luotao1 · 2019-05-28T09:12:38Z

paddle/fluid/operators/mkldnn/conv_mkldnn_op.cc

+            src_md, weights_md, dst_md, strides, paddings, mkldnn_engine,
+            fuse_relu || fuse_brelu /*fuse_relu*/, fuse_residual_conn,
+            false /*fuse_brelu*/, fuse_brelu_threshold, output_shift_scale,
+            sum_scale, is_test);


Could you combine ConvFwdPrimitiveDesc two functions, if (bias) = false, pass bias_md=nullptr into it.

Could you combine ConvFwdPrimitiveDesc two functions, if (bias) = false, pass bias_md=nullptr into it.

Ok, I will do like this.

I merge your pr at first, and you can refine in next PR!

I merge your pr at first, and you can refine in next PR!

Thank you! I start to refine now and will check the refined code through tests. I just update that it could be mkldnn v1.0 that will be used by baidu. Both support this op and both will be released after paddle release 1.5.

Got it. But the refined code PR could be created before paddle release 1.5?

Got it. But the refined code PR could be created before paddle release 1.5?

Yes, will be before 1.5

add INT8 conv+relu6 fuse and enbale mobilentv2 INT8 test

14e594b

test=develop

luotao1 reviewed May 22, 2019

View reviewed changes

luotao1 added int8 Intel labels May 22, 2019

lidanqing-vv changed the title ~~Add INT8 fuse+relu6 support and mobilenetv2 INT8 test~~ [WIP] Add INT8 fuse+relu6 support and mobilenetv2 INT8 test May 22, 2019

lidanqing-vv force-pushed the int8-mobilenetv2-updated-17130 branch from a19b7b1 to c02b869 Compare May 22, 2019 13:36

change fasle and 0.0 to fuse_brelu and brelu_threshold

50e9491

test=develop change the "fuse_relu||fuse_brelu" to "unsigned_output" test=develop

lidanqing-vv force-pushed the int8-mobilenetv2-updated-17130 branch from 3a606f4 to 50e9491 Compare May 22, 2019 21:34

Use relu instead of brelu as INT8 post-op because INT8 brelu is not e…

ca74a8a

…nabled in mkldnn v0.18 test=develop

lidanqing-vv changed the title ~~[WIP] Add INT8 fuse+relu6 support and mobilenetv2 INT8 test~~ Improve mobilenetv2 INT8 performance by using INT8 relu as post-op May 27, 2019

lidanqing-vv added 2 commits May 28, 2019 00:02

Merge branch 'develop' into int8-mobilenetv2-updated-17130

ade1c17

continuous-integration fix

ed226ff

test=develop

wojtuss approved these changes May 28, 2019

View reviewed changes

wojtuss added this to the v1.5 for Intel milestone May 28, 2019

luotao1 reviewed May 28, 2019

View reviewed changes

luotao1 merged commit 04b6c29 into PaddlePaddle:develop May 28, 2019

lidanqing-vv mentioned this pull request Jun 6, 2019

refactor the function ConvFwdPrimitiveDesc #17897

Merged

wozna mentioned this pull request Aug 2, 2019

Replace Relu with bounded Relu in MobileNetV2 quantization mkldnn #18988

Merged

Conversation

lidanqing-vv commented May 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidanqing-vv commented May 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidanqing-vv May 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lidanqing-vv commented May 22, 2019 •

edited

Loading

lidanqing-vv May 22, 2019 •

edited

Loading