use static variable to do cache instead of thread local in thread frequent switching case

[LeoZhao-Intel]

This case is special, NativeExecutor Predictor.run is called in a thread frequent switching case, for mkldnn, TransferScope cache creation is unavoidable. While current cache mechanism is using thread local variable to store cache, if execution thread is changed every iteration, it will always create new memory which makes big memory leak.

The solution is to use a global static variable to handle this case by explicitly indicating from user. Here I reuse SetMkldnnThreadid.

fix #18372 (comment)

[LeoZhao-Intel]

Using static variable solution and API is to make it simple, otherwise if we use single globe structure, it makes more complicated, it need thread sync etc, and efficiency will drop.

[luotao1]

@LeoZhao-Intel, Discussed with @Superjomn :

1. Do you have some comments for "-1", or do you have some unit-test to ensure this "-1"?
2. Do you need add lock for static_transfer_scope_cache? When multi-instance use "-1" mode, it will fail on this.

[LeoZhao-Intel]

@LeoZhao-Intel, Discussed with @Superjomn :

1. Do you have some comments for "-1", or do you have some unit-test to ensure this "-1"?

-1 is just specific for mkldnn single instance but with thread frequent switching, so far we don't have unit-test.

2. Do you need add lock for static_transfer_scope_cache? When multi-instance use "-1" mode, it will fail on this.

In mkldnn single instance but with thread frequent switching, it only supports one instance, yes, it will fail if with multi-instances. We do have plan to refine this solution in the future.

[jczaja]

@luotao1 , @Superjomn Why GPU execution needs to store TransferScope in TLS, why each thread has to keep its own copy of scopes and data caches eg. what is wrong with having only one global data and scope cache?

[luotao1]

Why GPU execution needs to store TransferScope in TLS? why each thread has to keep its own copy of scopes and data caches eg. what is wrong with having only one global data and scope cache?

@jczaja CUDADeviceContext is thread_local, i.e. one GPU-card have one CUDADeviceContext.

[LeoZhao-Intel]

I guess, it is similar with mkldnn cache, to support multi-instance, thread_local is used to simply the implementation.

[luotao1]

LGTM

[luotao1]

[14:29:15]	/home/teamcity/work/e84e6e698a3f913d/paddle/fluid/inference/tests/api/analyzer_bert_tester.cc:256:32: error: no member named 'set_cur_mkldnn_session_id' in namespace 'paddle::platform'
[14:29:15]	      if (is_static) platform::set_cur_mkldnn_session_id(1);
[14:29:15]	                     ~~~~~~~~~~^

In mac_ci

[luotao1]

LGTM

[luotao1]

@LeoZhao-Intel Could you revert this PR since #18578 merge and there is no random fail CI on MKLDNN?

[LeoZhao-Intel]

@LeoZhao-Intel Could you revert this PR since #18578 merge and there is no random fail CI on MKLDNN?

ok, will revert it soon.

[LeoZhao-Intel]

see new PR #18879 for revert,