create test for quantized resnet50 by sfraczek · Pull Request #16399 · PaddlePaddle/Paddle

sfraczek · 2019-03-22T13:48:23Z

No description provided.

luotao1 · 2019-03-22T14:32:03Z

paddle/fluid/inference/tests/api/CMakeLists.txt

TIMEOUT 1200 needs 20 minutes?

Once I ran randomly into timeout problem and I found somewhere default is 600 so i doubled it. I didn't realize it was in seconds 😆

i will leave it to default

luotao1 · 2019-03-22T14:33:31Z

paddle/fluid/inference/tests/api/CMakeLists.txt

why not use

inference_analysis_api_test(test_analyzer_int8_resnet50 ${INT8_RESNET50_INSTALL_DIR} analyzer_int8_resnet50_tester.cc SERIAL)

~~How do I pass it --infer_data=${INT8_RESNET50_INSTALL_DIR}/data.txt ?~~
its a legacy thing. I had problem passing some parameters. I can do as you point out now :)

luotao1 · 2019-03-22T14:34:28Z

paddle/fluid/inference/tests/api/analyzer_int8_resnet50_tester.cc

why should SetPasses manually here?

The default order of passes wasn't working

i tried without it and it works now. I can remove this

No, actually it gets wrong without this:

The default order of passes wasn't working

If default order of passes wasn't working, should you fix the default order? Otherwise, it is confused for users, since they should not care about the order.

I will see what can be done about this.

@luotao1 , @sfraczek , I have fixed the order of mkldnn passes: #16448
and removed use of SetPasses() here.

@luotao1 , @sfraczek , sorry, I have abandoned #16448 and reverted the changes here, as it does not work well in all cases

sfraczek · 2019-03-25T08:13:20Z

Hi @luotao1 , this build failed because it requires quantization core PR to work. How should we proceed?

luotao1 · 2019-03-25T08:57:05Z

paddle/fluid/inference/tests/api/CMakeLists.txt

Does resnet50_int8v2_data.txt.tar.gz contain 50000 images?

No, it has only 100. I Set the same data for calibration as for testing.

So, how do we test for 50000 images?

We can create a file like this one but with 5000 images and change the code a little to do that.

you can create resnet50_int8v2_full_data.txt.tar.gz with 50000 images, and change the code a litter.

luotao1 · 2019-03-25T08:58:23Z

paddle/fluid/inference/tests/api/analyzer_int8_resnet50_tester.cc

line23-34 GetPaddleDType could be put in tester_helper.h

ok good idea. Then I can also remove it from analyzer_bert_tester.cc.

luotao1 · 2019-03-25T08:58:59Z

paddle/fluid/inference/tests/api/analyzer_int8_resnet50_tester.cc

why batch_size is 100 here? The default batch_size is 1.

I used 100 images as calibration dataset and the same for the test set both with batch 100. I didn't consider what batch size should I use for prediction because the result should be equivalent. For quantization we only support one iteration so I had to use batch bigger than 1. Should I change something here? Is it important for a simple test?

it is not important for this simple test. But we need to test 50000 images on batch_size=1 on local machine. Thus, please enhance the test (maybe a new PR).

Ok, should I close this one then and open a new one with 50000 images?
Should I use first 100 of them for calibration and reuse them during testing of all 50000 images?

No, you could go ahead this PR, since 50000 images are too large for our ci to download. Thus, we will only test 50000 images on the local machine.

luotao1 · 2019-03-25T09:00:22Z

paddle/fluid/inference/tests/api/analyzer_int8_resnet50_tester.cc

The default order of passes wasn't working

If default order of passes wasn't working, should you fix the default order? Otherwise, it is confused for users, since they should not care about the order.

luotao1 · 2019-03-25T09:07:25Z

paddle/fluid/inference/tests/api/tester_helper.h

top1 acc quantized-》top1 Int8 accuracy
top1 acc reference-》top1 FP32 accuracy

luotao1 · 2019-03-25T09:09:41Z

paddle/fluid/inference/tests/api/analyzer_int8_resnet50_tester.cc

This TEST(Analyzer_int8_resnet50, quantization) is only for CompareQuantizedNativeAndAnalysis, how could we do profiler?

I will add profiler.

luotao1 · 2019-03-25T09:10:56Z

How should we proceed

@sfraczek You should proceed to merge core PR at first.

sfraczek · 2019-03-25T12:58:29Z

@luotao1 I have a problem with both my CMAKE instructions:

inference_download_and_uncompress(${INT8_RESNET50_INSTALL_DIR} "http://paddle-inference-dist.cdn.bcebos.com/int8" "resnet50_int8v2_data.txt.tar.gz")
inference_download_and_uncompress(${INT8_RESNET50_INSTALL_DIR} "http://paddle-inference-dist.cdn.bcebos.com/int8" "resnet50_int8_model.tar.gz" )

After I run cmake and make the files are nowhere to be found. Do you know why that may be happening?

luotao1 · 2019-03-25T13:04:28Z

cdn has some problem today. The file http://paddle-inference-dist.cdn.bcebos.com/int8/resnet50_int8v2_data.txt.tar.gz exists.

sfraczek · 2019-03-25T13:07:20Z

But I can download it with wget. I had the same problem last week.

luotao1 · 2019-03-25T13:11:16Z

We fix it in #16423 temporary, you can update like #16423 do. (change cdn to bj)

sfraczek · 2019-03-25T18:39:01Z

For now I have merged quantization code because I want to see if CI would pass.
@luotao1 I think there is a problem with setting any default order of passes currently. This is because of separation of mkldnn passes and cpu passes. For example in mobilenet, among other passes, we need the below three passes in mixed order:

depthwise_covn_mkldnn_pass
conv_bn_fuse_pass
conv_relu_mkldnn_fuse_pass

I don't want to make a hasty decision on how to resolve this.

luotao1 · 2019-03-26T08:20:52Z

For example in mobilenet, among other passes, we need the below three passes in mixed order

This is confused for users, and hard to use, could you fix them in default order?

lidanqing-vv · 2019-03-26T12:37:51Z

Hi, @luotao1 I have two questions:

Does this PR need to be merged before 29th?
We decided to store the image and labels in binary file which is smaller. For 50 000 images and label, the binary file size is 29G. We plan to implement the test as other analyzer_test in input_slots_all like in analyzer_transformer_tester.cc line 195. Load 29G into ram, do you think it will be ok for baidu validation team?
Thanks!

luotao1 · 2019-03-26T13:00:26Z

Does this PR need to be merged before 29th?

Yes, it should be merged before 29th. You could use the current resnet50_int8v2_data.txt.tar.gz, which is OK for CI test.

We decided to store the image and labels in a binary file which is smaller. For 50 000 images and label, the binary file size is 29G. We plan to implement the test as other analyzer_test in input_slots_all like in analyzer_transformer_tester.cc line 195. Load 29G into ram, do you think it will be ok for Baidu validation team?

The ILSVRC2012_img_val.tar.gz used in int8 V1 is 6.3G, the binary file (with tar.gz) is 29G for int8 v2?
29G is too big to upload and download from cdn. Could you provide a python script for converting it from ILSVRC2012_img_val.tar.gz?
Load 29G into ram, do you mean loading all the dataset at first? It needs a lot of memory for the machine. Our test machine is

free -g
             total       used       free     shared    buffers     cached
Mem:           376         38        337          0          0         33
-/+ buffers/cache:          4        371
Swap:            0          0          0

sfraczek · 2019-03-26T14:26:54Z

@luotao1 that archive contains image files and val_list.txt and they read and process the dataset here https://github.intel.com/AIPG/paddle/blob/7c5319ba121a6d73aeba0f06ce158680b160dcc2/python/paddle/fluid/contrib/tests/test_calibration.py#L67

So I can just copy a reader (and transformer with opencv) from our C-api app or we can save a file that is roughly 4 times smaller than previously (because we save them as floats now but we can save them as uchar and convert them to float later and finish the preprocessing left which is mean subtraction and division by stddev).

Sand3r- · 2019-03-26T15:27:09Z

@luotao1

For example in mobilenet, among other passes, we need the below three passes in mixed order

This is confused for users, and hard to use, could you fix them in default order?

I have refined the default pass order, by adding conv batch norm passes to the list of mkl-dnn passes. That way:

The correct order of pass execution is preserved and the fusings behave as intended
There is no need for user to manually set passes for his/her application

sfraczek · 2019-03-27T11:19:21Z

@luotao1 I've edited the code to read data.bin file, and meanwhile @lidanqing-intel has created the data.bin file with the test 100 images. It fixes the 2% accuracy diff on mobilenet because the dataset is the same as @hshen14 . I will soon push the fixes and please accept the data.bin from @lidanqing-intel and put on the server so I can modify cmake with the correct path.
I've replaced the fake dataset with the small data.bin too.

luotao1 · 2019-03-27T15:47:17Z

please update this PR after #16396 and #16490 merged.

test=develop

sfraczek · 2019-03-27T22:01:16Z

it is good for CI now. Works with small dataset and profiler should work with custom batch_size on 50000 images.
However, reporting accuracy of a full 50000 images dataset will require some further work because it only checks it for one batch currently (100) in quantizer test.

luotao1 · 2019-03-28T02:16:07Z

[22:07:51]	149/609 Test #165: test_analyzer_int8_resnet50 .....................***Exception: SegFault  2.71 sec
[22:07:51]	[==========] Running 2 tests from 1 test case.
[22:07:51]	[----------] Global test environment set-up.
[22:07:51]	[----------] 2 tests from Analyzer_int8_resnet50
[22:07:51]	[ RUN      ] Analyzer_int8_resnet50.quantization
[22:07:51]	/paddle/paddle/fluid/inference/tests/api/analyzer_int8_image_classification_tester.cc:114: Failure
[22:07:51]	Failed
[22:07:51]	Couldn't open file: /root/.cache/inference_demo/int8/data.bin
[22:07:51]	
[22:07:51]	        Start 167: test_analyzer_bert
[22:07:52]	150/609 Test #166: test_analyzer_int8_mobilenet ....................***Exception: SegFault  2.72 sec
[22:07:52]	[==========] Running 2 tests from 1 test case.
[22:07:52]	[----------] Global test environment set-up.
[22:07:52]	[----------] 2 tests from Analyzer_int8_resnet50
[22:07:52]	[ RUN      ] Analyzer_int8_resnet50.quantization
[22:07:52]	/paddle/paddle/fluid/inference/tests/api/analyzer_int8_image_classification_tester.cc:114: Failure
[22:07:52]	Failed
[22:07:52]	Couldn't open file: /root/.cache/inference_demo/int8/data.bin

test=develop

sfraczek · 2019-03-28T07:33:16Z

@luotao1 what is the current address of the file? I changed it to http://paddle-inference-dist.bj.bcebos.com/int8/imagenet_val_100.bin.tar.gz I thought it was shared with you.

luotao1 · 2019-03-28T08:04:41Z

https://paddle-inference-dist.bj.bcebos.com/int8/imagenet_val_100.tar.gz @sfraczek

lidanqing-vv · 2019-03-28T11:10:20Z

Hi @luotao1 Where should I put the python script for preprocessing and generating bin file?

luotao1 · 2019-03-28T11:12:35Z

please put in the same location with #16515

luotao1 · 2019-03-28T11:55:10Z

126: ---  detected 12 subgraphs
126: Fused graph 16
126: --- Running IR pass [depthwise_conv_mkldnn_pass]
126: --- Running IR pass [conv_bn_fuse_pass]
126: --- Running IR pass [conv_eltwiseadd_bn_fuse_pass]
126: --- Running IR pass [conv_bias_mkldnn_fuse_pass]
126: --- Running IR pass [conv3d_bias_mkldnn_fuse_pass]
126: --- Running IR pass [conv_relu_mkldnn_fuse_pass]
126: ---  detected 16 subgraphs
126: --- Running IR pass [conv_elementwise_add_mkldnn_fuse_pass]
126: Fused graph 0
126: --- Running IR pass [depthwise_conv_mkldnn_pass]
126: --- Running IR pass [conv_bn_fuse_pass]
126: --- Running IR pass [conv_eltwiseadd_bn_fuse_pass]
126: --- Running IR pass [conv_bias_mkldnn_fuse_pass]
126: --- Running IR pass [conv3d_bias_mkldnn_fuse_pass]
126: --- Running IR pass [conv_relu_mkldnn_fuse_pass]
126: --- Running IR pass [conv_elementwise_add_mkldnn_fuse_pass]
126: Fused graph 0
126: --- Running IR pass [cpu_quantize_placement_pass]
126: --- Running IR pass [depthwise_conv_mkldnn_pass]
126: --- Running IR pass [conv_bn_fuse_pass]
126: --- Running IR pass [conv_eltwiseadd_bn_fuse_pass]
126: --- Running IR pass [conv_bias_mkldnn_fuse_pass]
126: --- Running IR pass [conv3d_bias_mkldnn_fuse_pass]
126: --- Running IR pass [conv_relu_mkldnn_fuse_pass]
126: --- Running IR pass [conv_elementwise_add_mkldnn_fuse_pass]
126: Fused graph 0
126: --- Running IR pass [cpu_quantize_placement_pass]

There are duplicated passes, similar problem with #16174. We can fix it after May 29th.

lidanqing-vv · 2019-03-28T12:19:24Z

please put in the same location with #16515

Thanks. I will put in the same position. But could I push this file to this PR #16399 ?

luotao1 · 2019-03-28T12:24:46Z

please do not push the script to this PR. @lidanqing-intel

lidanqing-vv · 2019-03-28T12:27:44Z

please do not push the script to this PR. @lidanqing-intel

ok I will not, it will cause CI restart

luotao1 · 2019-03-28T14:28:10Z

paddle/fluid/inference/tests/api/tester_helper.h

+    const PaddlePredictor::Config *qconfig,
+    const std::vector<std::vector<PaddleTensor>> &inputs) {
+  PrintConfig(config, true);
+  std::vector<PaddleTensor> analysis_outputs, quantized_outputs;


Could you add some log like LOG(INFO) << "FP32 start..." before line470 andLOG(INFO) << "INT8 start..." before line471?

Can we do that in next PR?

luotao1

I merge this PR at first, please refine #16532 later.

$@sfraczek$ sfraczek added Intel int8 labels Mar 22, 2019

$@sfraczek$ sfraczek requested review from luotao1 and wojtuss March 22, 2019 13:48

luotao1 reviewed Mar 22, 2019

View reviewed changes

luotao1 reviewed Mar 25, 2019

View reviewed changes

$@sfraczek$

This comment has been minimized.

Sign in to view

$@sfraczek$ sfraczek changed the title ~~create test for quantized resnet50~~ [wip] create test for quantized resnet50 Mar 25, 2019

luotao1 mentioned this pull request Mar 26, 2019

Fix order of mkldnn passes #16448

Closed

Sand3r- mentioned this pull request Mar 27, 2019

Refine default MKL-DNN Pass order #16490

Merged

create test for quantized resnet50

fe21578

test=develop

$@sfraczek$ sfraczek changed the title ~~[wip] create test for quantized resnet50~~ create test for quantized resnet50 Mar 27, 2019

fixed url to dataset

8ece7a9

test=develop

luotao1 mentioned this pull request Mar 28, 2019

MKLDNN INT8 v2 readme.md #16515

Merged

lidanqing-vv mentioned this pull request Mar 28, 2019

preprocess with PIL the full val dataset and save binary #16529

Merged

luotao1 reviewed Mar 28, 2019

View reviewed changes

wojtuss mentioned this pull request Mar 28, 2019

create a test for quantized resnet50 - extended #16532

Closed

luotao1 approved these changes Mar 29, 2019

View reviewed changes

luotao1 merged commit 5b24002 into PaddlePaddle:develop Mar 29, 2019

wojtuss mentioned this pull request Mar 29, 2019

fix dataset reading and add support for full dataset #16559

Merged

Conversation

sfraczek commented Mar 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfraczek Mar 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfraczek Mar 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfraczek commented Mar 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfraczek Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfraczek Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfraczek Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luotao1 commented Mar 25, 2019

Uh oh!

This comment has been minimized.

sfraczek commented Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

$@sfraczek$ sfraczek commented Mar 22, 2019

$@sfraczek$ sfraczek Mar 22, 2019 •

edited

Loading

$@sfraczek$ sfraczek Mar 22, 2019 •

edited

Loading

$@sfraczek$ sfraczek Mar 25, 2019 •

edited

Loading

$@sfraczek$ sfraczek Mar 25, 2019 •

edited

Loading

$@sfraczek$ sfraczek Mar 25, 2019 •

edited

Loading

sfraczek commented Mar 25, 2019 •

edited

Loading

sfraczek commented Mar 25, 2019 •

edited

Loading

lidanqing-vv commented Mar 26, 2019 •

edited

Loading

sfraczek commented Mar 26, 2019 •

edited

Loading

Sand3r- commented Mar 26, 2019 •

edited

Loading

sfraczek commented Mar 27, 2019 •

edited

Loading