[WIP] Solve mklml memory leakage by luotao1 · Pull Request #23314 · PaddlePaddle/Paddle

luotao1 · 2020-03-30T12:15:03Z

Solve memory leakage by removing one intel libiomp5 library and keeping gnu libgomp5.
fix #22827

test=develop

luotao1 · 2020-03-30T12:51:57Z

@yinghu5 根据 #22827 (comment) ，9da98cd 将libiomp5去掉，同时用libmklml_gnu.so代替libmklml_intel.so。但依然存在内存泄漏问题。

使用pmap命令看，可以发现没有链接libmklml_intel.so和libiomp5，同时正确链接了libmklml_gnu.so。但还是有内存泄漏，且泄漏曲线和Issue中的图类似，隔一段时间会阶段性跳跃。

$ pmap -x 4091
4091:   ./build/mul_demo --model_dir=mul_model --thread_num=3 --num=-1
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000     376     268       0 r-x--  mul_demo
000000000065e000       8       8       8 rw---  mul_demo
0000000000660000      64      16      16 rw---    [ anon ]
0000000000d71000    3684    3564    3564 rw---    [ anon ]
00007ff1f0bfe000  512008    3740    3740 rw---    [ anon ]
00007ff210000000    1224     336     336 rw---    [ anon ]
00007ff210132000   64312       0       0 -----    [ anon ]
00007ff214000000    1928    1640    1640 rw---    [ anon ]
00007ff2141e2000   63608       0       0 -----    [ anon ]
00007ff218000000    1352    1064    1064 rw---    [ anon ]
00007ff218152000   64184       0       0 -----    [ anon ]
00007ff21c3e4000       4       0       0 -----    [ anon ]
00007ff21c3e5000   10240      20      20 rw---    [ anon ]
00007ff21cde5000       4       0       0 -----    [ anon ]
00007ff21cde6000   10240      24      24 rw---    [ anon ]
00007ff21d7e6000       4       0       0 -----    [ anon ]
00007ff21d7e7000   10240      20      20 rw---    [ anon ]
00007ff21e1e7000      56      36       0 r-x--  libgomp.so.1.0.0
00007ff21e1f5000    2044       0       0 -----  libgomp.so.1.0.0
00007ff21e3f4000       4       4       4 rw---  libgomp.so.1.0.0
00007ff21e3f5000    1676     544       0 r-x--  libc-2.18.so
00007ff21e598000    2048       0       0 -----  libc-2.18.so
00007ff21e798000      16      16      16 r----  libc-2.18.so
00007ff21e79c000       8       8       8 rw---  libc-2.18.so
00007ff21e79e000      16      12      12 rw---    [ anon ]
00007ff21e7a2000      84      44       0 r-x--  libgcc_s.so.1
00007ff21e7b7000    2048       0       0 -----  libgcc_s.so.1
00007ff21e9b7000       4       4       4 rw---  libgcc_s.so.1
00007ff21e9b8000    1032      68       0 r-x--  libm-2.18.so
00007ff21eaba000    2044       0       0 -----  libm-2.18.so
00007ff21ecb9000       4       4       4 r----  libm-2.18.so
00007ff21ecba000       4       4       4 rw---  libm-2.18.so
00007ff21ecbb000     916     576       0 r-x--  libstdc++.so.6.0.19
00007ff21eda0000    2044       0       0 -----  libstdc++.so.6.0.19
00007ff21ef9f000      32      32      32 r----  libstdc++.so.6.0.19
00007ff21efa7000       8       8       8 rw---  libstdc++.so.6.0.19
00007ff21efa9000      84      16      16 rw---    [ anon ]
00007ff21efbe000     100      76       0 r-x--  libpthread-2.18.so
00007ff21efd7000    2044       0       0 -----  libpthread-2.18.so
00007ff21f1d6000       4       4       4 r----  libpthread-2.18.so
00007ff21f1d7000       4       4       4 rw---  libpthread-2.18.so
00007ff21f1d8000      16       4       4 rw---    [ anon ]
00007ff21f1dc000      12       8       0 r-x--  libdl-2.18.so
00007ff21f1df000    2044       0       0 -----  libdl-2.18.so
00007ff21f3de000       4       4       4 r----  libdl-2.18.so
00007ff21f3df000       4       4       4 rw---  libdl-2.18.so
00007ff21f3e0000      28      16       0 r-x--  librt-2.18.so
00007ff21f3e7000    2044       0       0 -----  librt-2.18.so
00007ff21f5e6000       4       4       4 r----  librt-2.18.so
00007ff21f5e7000       4       4       4 rw---  librt-2.18.so
00007ff21f5e8000      96      16       0 r-x--  libz.so.1.2.8
00007ff21f600000    2044       0       0 -----  libz.so.1.2.8
00007ff21f7ff000       4       4       4 rw---  libz.so.1.2.8
00007ff21f800000   23596    3196       0 r-x--  libdnnl.so.1
00007ff220f0b000    2048       0       0 -----  libdnnl.so.1
00007ff22110b000     336     336     336 r----  libdnnl.so.1
00007ff22115f000       4       4       4 rw---  libdnnl.so.1
00007ff221160000     196     192     192 rw---    [ anon ]
00007ff221191000  126264    2460       0 r-x--  libmklml_gnu.so
00007ff228cdf000    2044       0       0 -----  libmklml_gnu.so
00007ff228ede000     148     148     148 r----  libmklml_gnu.so
00007ff228f03000     684     336     336 rw---  libmklml_gnu.so
00007ff228fae000     144      96      96 rw---    [ anon ]
00007ff228fd2000   43484   24796       0 r-x--  libpaddle_fluid.so
00007ff22ba49000    2044       0       0 -----  libpaddle_fluid.so
00007ff22bc48000     676     676     676 rw---  libpaddle_fluid.so
00007ff22bcf1000     116      68      68 rw---    [ anon ]
00007ff22bd0e000     128     112       0 r-x--  ld-2.18.so
00007ff22be22000    1072    1068    1068 rw---    [ anon ]
00007ff22bf2e000       4       4       4 r----  ld-2.18.so
00007ff22bf2f000       4       4       4 rw---  ld-2.18.so
00007ff22bf30000       4       4       4 rw---    [ anon ]
00007ffea3327000     132      16      16 rw---    [ stack ]
00007ffea334b000       8       4       0 r-x--    [ anon ]
ffffffffff600000       4       0       0 r-x--    [ anon ]
----------------  ------  ------  ------
total kB          969248   45744   13524

Issue中的命令，如果用单线程跑，也会出现内存泄漏。单线程命令为

./build/mul_demo --model_dir=mul_model --thread_num=1 --num=-1

备注：mul.zip的CMakeLists.txt需要将libiomp5去掉，同时用libmklml_gnu.so代替libmklml_intel.so

set(MATH_LIB ${MATH_LIB_PATH}/lib/libmklml_gnu${CMAKE_SHARED_LIBRARY_SUFFIX})

luotao1 · 2020-03-30T14:20:22Z

使用pprof工具，类似 https://github.com/PaddlePaddle/Paddle/pull/18372/files#issuecomment-506920962

$ /home/luotao/gperftools/bin/pprof --text ./build/mul_demo /tmp/mul_demo.17265.test_foo-end.heap Using local file ./build/mul_demo.
Using local file /tmp/mul_demo.17265.test_foo-end.heap.
Total: 0.3 MB
     0.3 100.0% 100.0%      0.3 100.0% mm_account_ptr_by_tid..0
     0.0   0.0% 100.0%      0.3 100.0% SGEMM
     0.0   0.0% 100.0%      0.3 100.0% __clone
     0.0   0.0% 100.0%      0.3 100.0% cblas_sgemm
     0.0   0.0% 100.0%      0.3 100.0% execute_native_thread_routine
     0.0   0.0% 100.0%      0.2  74.6% mkl_blas_avx_sgemm_get_bufs
     0.0   0.0% 100.0%      0.2  74.6% mkl_blas_avx_xsgemm
     0.0   0.0% 100.0%      0.2  74.6% mkl_blas_avx_xsgemm_par
     0.0   0.0% 100.0%      0.3 100.0% mkl_blas_sgemm
     0.0   0.0% 100.0%      0.2  74.6% mkl_blas_sgemm_omp_driver_v1
     0.0   0.0% 100.0%      0.2  74.6% mkl_serv_allocate
     0.0   0.0% 100.0%      0.1  25.4% mkl_serv_get_num_stripes
     0.0   0.0% 100.0%      0.1  25.4% mkl_serv_malloc
     0.0   0.0% 100.0%      0.3 100.0% paddle::AnalysisPredictor::ZeroCopyRun
     0.0   0.0% 100.0%      0.3 100.0% paddle::framework::NaiveExecutor::Run
     0.0   0.0% 100.0%      0.3 100.0% paddle::framework::OperatorBase::Run
     0.0   0.0% 100.0%      0.3 100.0% paddle::framework::OperatorWithKernel::RunImpl@215c8a0
     0.0   0.0% 100.0%      0.3 100.0% paddle::framework::OperatorWithKernel::RunImpl@215d4e0
     0.0   0.0% 100.0%      0.3 100.0% paddle::operators::MulKernel::Compute
     0.0   0.0% 100.0%      0.3 100.0% paddle::operators::math::Blas::MatMul
     0.0   0.0% 100.0%      0.3 100.0% paddle::work_thread
     0.0   0.0% 100.0%      0.3 100.0% start_thread
     0.0   0.0% 100.0%      0.3 100.0% std::_Bind_simple::_M_invoke
     0.0   0.0% 100.0%      0.3 100.0% std::_Bind_simple::operator
     0.0   0.0% 100.0%      0.3 100.0% std::_Function_handler::_M_invoke
     0.0   0.0% 100.0%      0.3 100.0% std::thread::_Impl::_M_run

luotao1 · 2020-04-01T05:57:20Z

在PR_CI_Inference中碰到undefined reference to symbol 'GOMP_loop_ull_static_next@@GOMP_2.0'

[2020-03-30 23:19:20]  [ 50%] Building CXX object CMakeFiles/simple_on_word2vec.dir/simple_on_word2vec.cc.o
[2020-03-30 23:19:21]  [100%] Linking CXX executable simple_on_word2vec
[2020-03-30 23:19:22]  /usr/local/bin/ld: /workspace/Paddle/build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.a(prior_box_op.cc.o): undefined reference to symbol 'GOMP_loop_ull_static_next@@GOMP_2.0'
[2020-03-30 23:19:22]  //usr/lib/x86_64-linux-gnu/libgomp.so.1: error adding symbols: DSO missing from command line
[2020-03-30 23:19:22]  collect2: error: ld returned 1 exit status
[2020-03-30 23:19:22]  CMakeFiles/simple_on_word2vec.dir/build.make:87: recipe for target 'simple_on_word2vec' failed

luotao1 · 2020-04-01T08:48:56Z

valgrind工具

使用valgrind3.15.0
运行命令：valgrind ./build/mul_demo

日志

mkl日志

    ==22770== Warning: set address range perms: large range [0x12807080, 0x31c07080) (undefined)
    ==22770==
    ==22770== HEAP SUMMARY:
    ==22770==     in use at exit: 526,301,226 bytes in 41,846 blocks
    ==22770==   total heap usage: 117,353 allocs, 75,507 frees, 542,454,597 bytes allocated
    ==22770==
    ==22770== LEAK SUMMARY:
    ==22770==    definitely lost: 224,536 bytes in 1,292 blocks
    ==22770==    indirectly lost: 1,756,318 bytes in 39,762 blocks
    ==22770==      possibly lost: 0 bytes in 0 blocks
    ==22770==    still reachable: 524,320,372 bytes in 792 blocks
    ==22770==                       of which reachable via heuristic:
    ==22770==                         stdstring          : 3,251 bytes in 58 blocks
    ==22770==         suppressed: 0 bytes in 0 blocks
    ==22770== Rerun with --leak-check=full to see details of leaked memory
    ==22770==
    ==22770== Use --track-origins=yes to see where uninitialised values come from
    ==22770== For lists of detected and suppressed errors, rerun with: -s
    ==22770== ERROR SUMMARY: 104 errors from 6 contexts (suppressed: 0 from 0)

openblas日志。openblas使用1.7.1版本

==5432== Warning: set address range perms: large range [0x2574b080, 0x44b4b080) (undefined)
==5432==
==5432== HEAP SUMMARY:
==5432==     in use at exit: 526,154,528 bytes in 38,240 blocks
==5432==   total heap usage: 109,017 allocs, 70,777 frees, 536,933,062 bytes allocated
==5432==
==5432== LEAK SUMMARY:
==5432==    definitely lost: 217,624 bytes in 1,270 blocks
==5432==    indirectly lost: 1,616,668 bytes in 36,180 blocks
==5432==      possibly lost: 0 bytes in 0 blocks
==5432==    still reachable: 524,320,236 bytes in 790 blocks
==5432==                       of which reachable via heuristic:
==5432==                         stdstring          : 3,251 bytes in 58 blocks
==5432==         suppressed: 0 bytes in 0 blocks
==5432== Rerun with --leak-check=full to see details of leaked memory
==5432==
==5432== Use --track-origins=yes to see where uninitialised values come from
==5432== For lists of detected and suppressed errors, rerun with: -s
==5432== ERROR SUMMARY: 32 errors from 6 contexts (suppressed: 0 from 0)

两者都有泄漏，但openblas泄漏的少

luotao1 · 2020-04-01T13:10:05Z

根据上一条的日志，使用valgrind -s ./build/mul_demo命令，修复未初始化的两个变量后，

    diff --git a/paddle/fluid/inference/api/analysis_config.cc b/paddle/fluid/inference/api/analysis_config.cc
    index 5ee2a54f43..af78051c00 100644
    --- a/paddle/fluid/inference/api/analysis_config.cc
    +++ b/paddle/fluid/inference/api/analysis_config.cc
    @@ -381,8 +381,8 @@ std::string AnalysisConfig::SerializeInfoCache() {
       ss << memory_pool_init_size_mb_;
       ss << use_tensorrt_;
    -  ss << tensorrt_workspace_size_;
    -  ss << tensorrt_max_batchsize_;
    +  // ss << tensorrt_workspace_size_;
    +  // ss << tensorrt_max_batchsize_;
       ss << tensorrt_min_subgraph_size_;
       ss << enable_memory_optim_;

能看到9da98cd 确实能修复iomp5泄漏的问题。修复前的日志如下：

    ==2799== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
    ==2799==
    ==2799== 1 errors in context 1 of 1:
    ==2799== Syscall param sched_setaffinity(mask) points to unaddressable byte(s)
    ==2799==    at 0x1092DFC9: syscall (in /home/opt/gcc-4.8.2.bpkg-r4/gcc-4.8.2.bpkg-r4/lib64/libc-2.18.so)
    ==2799==    by 0x10CE6E97: __kmp_affinity_determine_capable (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/third_party/install/mklml/lib/libiomp5.so)
    ==2799==    by 0x10CC2867: __kmp_env_initialize(char const*) (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/third_party/install/mklml/lib/libiomp5.so)
    ==2799==    by 0x10CABA34: _INTERNAL_25_______src_kmp_runtime_cpp_d89aedeb::__kmp_do_serial_initialize() (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/third_party/install/mklml/lib/libiomp5.so)
    ==2799==    by 0x10C9F27F: __kmp_get_global_thread_id_reg (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/third_party/install/mklml/lib/libiomp5.so)
    ==2799==    by 0x10C8C369: omp_set_num_threads@@VERSION (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/third_party/install/mklml/lib/libiomp5.so)
    ==2799==    by 0x5A2A12E: paddle::platform::SetNumThreads(int) (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
    ==2799==    by 0x59F1FF8: paddle::AnalysisPredictor::Init(std::shared_ptr<paddle::framework::Scope> const&, std::shared_ptr<paddle::framework::ProgramDesc> const&) (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
    ==2799==    by 0x59F2515: std::unique_ptr<paddle::PaddlePredictor, std::default_delete<paddle::PaddlePredictor> > paddle::CreatePaddlePredictor<paddle::AnalysisConfig, (paddle::PaddleEngineKind)2>(paddle::AnalysisConfig const&) (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
    ==2799==    by 0x59F3130: std::unique_ptr<paddle::PaddlePredictor, std::default_delete<paddle::PaddlePredictor> > paddle::CreatePaddlePredictor<paddle::AnalysisConfig>(paddle::AnalysisConfig const&) (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
    ==2799==    by 0x42289C: main (in /home/luotao/test/mul_weibo/build/mul_demo)
    ==2799==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
    ==2799==
    ==2799== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

luotao1 · 2020-04-01T13:45:33Z

使用valgrind --leak-check=full -s ./build/mul_demo将日志打印后，发现和#15339 以及 #18382 的错误一致，都是OpInfoFiller出错。

==3298== 10,212 (32 direct, 10,180 indirect) bytes in 1 blocks are definitely lost in loss record 36,409 of 36,410
==3298==    at 0x4C28583: operator new(unsigned long) (vg_replace_malloc.c:344)
==3298==    by 0x63095DE: paddle::framework::details::OpInfoFiller<paddle::operators::reader::CreateDoubleBufferReaderOpMaker, (paddle::framework::details::OpInfoFillType)1>::operator()(char const*, paddle::framework::OpInfo*) const (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
==3298==    by 0x630A273: paddle::framework::OperatorRegistrar<paddle::operators::reader::CreateDoubleBufferReaderOp, paddle::operators::reader::CreateDoubleBufferReaderOpMaker, paddle::operators::reader::DecoratedReaderInferShape, paddle::framework::EmptyGradOpMaker<paddle::framework::OpDesc>, paddle::framework::EmptyGradOpMaker<paddle::imperative::OpBase>, paddle::operators::reader::DecoratedReaderInferVarType>::OperatorRegistrar(char const*) (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
==3298==    by 0x70CAF35: ??? (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
==3298==    by 0x594B03E: ??? (in /home/luotao/Paddle/cpu_build/fluid_inference_install_dir/paddle/lib/libpaddle_fluid.so)
==3298==    by 0x1FFEFFFD6F: ???
==3298==    by 0x400E7CC: call_init.part.0 (in /home/opt/gcc-4.8.2.bpkg-r4/gcc-4.8.2.bpkg-r4/lib64/ld-2.18.so)
==3298==    by 0x400E8F2: _dl_init (in /home/opt/gcc-4.8.2.bpkg-r4/gcc-4.8.2.bpkg-r4/lib64/ld-2.18.so)
==3298==    by 0x40011C9: ??? (in /home/opt/gcc-4.8.2.bpkg-r4/gcc-4.8.2.bpkg-r4/lib64/ld-2.18.so)
==3298==    by 0x1: ???
==3298==    by 0x1FFF000016: ???
==3298==    by 0x1FFF000027: ???

luotao1 · 2020-04-02T02:20:49Z

@yinghu5 最新发现，使用develop分支，直接在mul_demo.cc里加入mkl_free_buffers()进行释放，即可解决内存泄漏问题。

GaoWei8 and others added 5 commits March 24, 2020 07:12

mklml memory leakage

f3051c4

Merge branch 'mkl_mem' of https://github.com/GaoWei8/Paddle into iomp

8e769a1

fix compile error

0ebefb7

fix dynamic_loader and some demo CMakeLists.txt

3e516ee

test=develop

remove iomp5.so in setup.py.in

9da98cd

test=develop

luotao1 mentioned this pull request Apr 7, 2020

solve mklml memory leak #23557

Merged

luotao1 closed this Apr 10, 2020

luotao1 deleted the iomp branch May 28, 2021 02:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Solve mklml memory leakage#23314

[WIP] Solve mklml memory leakage#23314
luotao1 wants to merge 5 commits intoPaddlePaddle:developfrom
luotao1:iomp

luotao1 commented Mar 30, 2020

Uh oh!

luotao1 commented Mar 30, 2020

Uh oh!

luotao1 commented Mar 30, 2020

Uh oh!

luotao1 commented Apr 1, 2020 •

edited

Loading

Uh oh!

luotao1 commented Apr 1, 2020 •

edited

Loading

Uh oh!

luotao1 commented Apr 1, 2020

Uh oh!

luotao1 commented Apr 1, 2020

Uh oh!

luotao1 commented Apr 2, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

luotao1 commented Mar 30, 2020

Uh oh!

luotao1 commented Mar 30, 2020

Uh oh!

luotao1 commented Mar 30, 2020

Uh oh!

luotao1 commented Apr 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luotao1 commented Apr 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

valgrind工具

日志

Uh oh!

luotao1 commented Apr 1, 2020

Uh oh!

luotao1 commented Apr 1, 2020

Uh oh!

luotao1 commented Apr 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

luotao1 commented Apr 1, 2020 •

edited

Loading

luotao1 commented Apr 1, 2020 •

edited

Loading

luotao1 commented Apr 2, 2020 •

edited

Loading