create new api to indicate detect thread usage

[LeoZhao-Intel]

This PR solves 2 issues:

1. When Predictor.run is called in changed thread, it will make memory leak due to threadid is inserted into key, while each time key is different.
2. For detect model, input dims are dynamic, not fixed, it will make conv/pool/concat mkldnn op memory leak due to each time key is different.

The solve method is to disable cache in this case, we extend EnableMKLDNN in AnaysisConfig, use parameter to control if cache is needed.

related #17611

[luotao1]

Please add the description of this PR.

[LeoZhao-Intel]

@jczaja @jianhang-liu

[LeoZhao-Intel]

This PR solves 2 issues:

1. When Predictor.run is called in changed thread, it will make memory leak due to threadid is inserted into key, while each time key is different.
2. For detect model, input dims are dynamic, not fixed, it will make conv/pool/concat mkldnn op memory leak due to each time key is different.

The solve method is to disable cache in this case, we extend EnableMKLDNN in AnaysisConfig, use parameter to control if cache is needed.

[luotao1]

[19:20:47]	[100%] Linking CXX executable test_analyzer_small_dam
[19:20:48]	../../../../../third_party/install/ngraph/lib/libngraph.so: file not recognized: File truncated
[19:20:48]	collect2: error: ld returned 1 exit status
[19:20:48]	make[2]: *** [paddle/fluid/inference/tests/api/test_analyzer_mobilenet_depthwise_conv] Error 1
[19:20:48]	paddle/fluid/inference/tests/api/CMakeFiles/test_analyzer_mobilenet_depthwise_conv.dir/build.make:641: recipe for target 'paddle/fluid/inference/tests/api/test_analyzer_mobilenet_depthwise_conv' failed
[19:20:48]	make[1]: *** [paddle/fluid/inference/tests/api/CMakeFiles/test_analyzer_mobilenet_depthwise_conv.dir/all] Error 2
[19:20:48]	CMakeFiles/Makefile2:108240: recipe for target 'paddle/fluid/inference/tests/api/CMakeFiles/test_analyzer_mobilenet_depthwise_conv.dir/all' failed
[19:20:48]	make[1]: *** Waiting for unfinished jobs....

http://ci.paddlepaddle.org/viewLog.html?buildId=115069&buildTypeId=Paddle_PrCi&tab=buildLog&_focus=7893

[jczaja]

Those are very core changes , have you run extensive performance tests to check if inference of our models (CAPI tests of our validation) is unharmed ?

[LeoZhao-Intel]

This is just WA to avoid memory leak issue due to dynamic shape and run calling from different threads, and does harm performance. mkldnn reuse does improve performance much.
This is perf data using reuse.

This is perf data after disable cache.

[jczaja]

@LeoZhao-Intel I just want to make sure that reusing is disabled only in situation when we expect it to be disabled. I will copy this PR branch to internal repo and run automatic tests. Results should be available tonight

[LeoZhao-Intel]

should be able to pass CI test, but it will fail if EnableMKLDNN(1) is called in multiple instances case and SetMkldnnthreadid() is not set properly, so as I said before it is just WA. Need re-factor whole stack including API.

[jczaja]

@LeoZhao-Intel Ok, performance (C-API) tests passed eg. no impact visible. My understanding is that you are still working on this PR ?

[LeoZhao-Intel]

this PR gonna be dropped since it impacts performance much and can't reach target, but some requirements may be leveraged into formal solution. e.g. support cache disabled case, use API to indicate scene.

[ghost]

Thanks Leo for big efforts on this PR. The investigation in this PR had given us much insight of this issue.

[ghost]

Closed. But some code will be cloned to a new PR from Leo. Similar as PR #18217 .