MKL: export all libs

[dpo]

Created in a bit of a haste. Not sure it's necessary to export all libs.

[giordano]

For reference, these are the libs I see on my Linux machine:

    LibraryProduct("libmkl_avx2", :libmkl_avx2),
    LibraryProduct("libmkl_avx512_mic", :libmkl_avx512_mic),
    LibraryProduct("libmkl_avx512", :libmkl_avx512),
    LibraryProduct("libmkl_avx", :libmkl_avx),
    LibraryProduct("libmkl_blacs_intelmpi_ilp64", :libmkl_blacs_intelmpi_ilp64),
    LibraryProduct("libmkl_blacs_intelmpi_lp64", :libmkl_blacs_intelmpi_lp64),
    LibraryProduct("libmkl_blacs_openmpi_ilp64", :libmkl_blacs_openmpi_ilp64),
    LibraryProduct("libmkl_blacs_openmpi_lp64", :libmkl_blacs_openmpi_lp64),
    LibraryProduct("libmkl_blacs_sgimpt_ilp64", :libmkl_blacs_sgimpt_ilp64),
    LibraryProduct("libmkl_blacs_sgimpt_lp64", :libmkl_blacs_sgimpt_lp64),
    LibraryProduct("libmkl_cdft_core", :libmkl_cdft_core),
    LibraryProduct("libmkl_core", :libmkl_core),
    LibraryProduct("libmkl_def", :libmkl_def),
    LibraryProduct("libmkl_gf_ilp64", :libmkl_gf_ilp64),
    LibraryProduct("libmkl_gf_lp64", :libmkl_gf_lp64),
    LibraryProduct("libmkl_gnu_thread", :libmkl_gnu_thread),
    LibraryProduct("libmkl_intel_ilp64", :libmkl_intel_ilp64),
    LibraryProduct("libmkl_intel_lp64", :libmkl_intel_lp64),
    LibraryProduct("libmkl_intel_thread", :libmkl_intel_thread),
    LibraryProduct("libmkl_mc3", :libmkl_mc3),
    LibraryProduct("libmkl_mc", :libmkl_mc),
    LibraryProduct("libmkl_pgi_thread", :libmkl_pgi_thread),
    LibraryProduct("libmkl_rt", :libmkl_rt),
    LibraryProduct("libmkl_scalapack_ilp64", :libmkl_scalapack_ilp64),
    LibraryProduct("libmkl_scalapack_lp64", :libmkl_scalapack_lp64),
    LibraryProduct("libmkl_sequential", :libmkl_sequential),
    LibraryProduct("libmkl_tbb_thread", :libmkl_tbb_thread),
    LibraryProduct("libmkl_vml_avx2", :libmkl_vml_avx2),
    LibraryProduct("libmkl_vml_avx512_mic", :libmkl_vml_avx512_mic),
    LibraryProduct("libmkl_vml_avx512", :libmkl_vml_avx512),
    LibraryProduct("libmkl_vml_avx", :libmkl_vml_avx),
    LibraryProduct("libmkl_vml_cmpt", :libmkl_vml_cmpt),
    LibraryProduct("libmkl_vml_def", :libmkl_vml_def),
    LibraryProduct("libmkl_vml_mc2", :libmkl_vml_mc2),
    LibraryProduct("libmkl_vml_mc3", :libmkl_vml_mc3),
    LibraryProduct("libmkl_vml_mc", :libmkl_vml_mc),

[dpo]

Yeah, I'm doing the same. Apart from the libmkl_blacs_mpich libs, the other libs the linux build is complaining about are definitely there?!

[dpo]

How do I define a list of products that depends on the platform?

[giordano]

Pass different products arguments to different build_tarballs calls. We have already the mechanism in place here.

[giordano]

the other libs the linux build is complaining about are definitely there?!

It's not that aren't there, they can't be dlopened. For example: https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=3800&view=logs&j=efce48f3-28a1-5321-70aa-c183c50cb11b&t=744a09f2-d2b5-5d6c-521a-0384a7fbb802&l=401

ERROR: could not load library "/agent/_work/1/s/M/MKL/build/x86_64-linux-gnu/1wNwU2sZ/destdir/lib/libmkl_avx.so"
/agent/_work/1/s/M/MKL/build/x86_64-linux-gnu/1wNwU2sZ/destdir/lib/libmkl_avx.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8
┌ Warning: lib/libmkl_avx.so cannot be dlopen()'ed
└ @ BinaryBuilder /depot/packages/BinaryBuilder/SPVAM/src/Auditor.jl:146
##[warning]lib/libmkl_avx.so cannot be dlopen()'ed

Looping over all the libraries shows that this symbol is defined in some of the mkl libraries

 for file in *.so; do echo $file; nm -D $file | grep mkl_sparse_optimize_bsr_trsm_i8; done
libmkl_avx2.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_avx512_mic.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_avx512.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_avx.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_blacs_intelmpi_ilp64.so
libmkl_blacs_intelmpi_lp64.so
libmkl_blacs_openmpi_ilp64.so
libmkl_blacs_openmpi_lp64.so
libmkl_blacs_sgimpt_ilp64.so
libmkl_blacs_sgimpt_lp64.so
libmkl_cdft_core.so
libmkl_core.so
libmkl_def.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_gf_ilp64.so
libmkl_gf_lp64.so
libmkl_gnu_thread.so
0000000000618860 T mkl_sparse_optimize_bsr_trsm_i8
libmkl_intel_ilp64.so
libmkl_intel_lp64.so
libmkl_intel_thread.so
0000000000ca52b0 T mkl_sparse_optimize_bsr_trsm_i8
libmkl_mc3.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_mc.so
                 U mkl_sparse_optimize_bsr_trsm_i8
libmkl_pgi_thread.so
00000000006f8fb0 T mkl_sparse_optimize_bsr_trsm_i8
libmkl_rt.so
libmkl_scalapack_ilp64.so
libmkl_scalapack_lp64.so
libmkl_sequential.so
0000000000613280 T mkl_sparse_optimize_bsr_trsm_i8
libmkl_tbb_thread.so
00000000008c84e0 T mkl_sparse_optimize_bsr_trsm_i8
libmkl_vml_avx2.so
libmkl_vml_avx512_mic.so
libmkl_vml_avx512.so
libmkl_vml_avx.so
libmkl_vml_cmpt.so
libmkl_vml_def.so
libmkl_vml_mc2.so
libmkl_vml_mc3.so
libmkl_vml_mc.so

but libmkl_avx.so doesn't link to any of them

% ldd libmkl_avx.so 
        linux-vdso.so.1 (0x00007fff1c569000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f0fd8d5d000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f0fd8b97000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f0fdbff5000)

[giordano]

I think this works only if building for one platform at the time (which happens to be the case with our CI system now), not if building for all them together. If the products are different for all platforms, you literally need separate calls to build_tarballs

[giordano]

I think this is not going to work anyway for Linux because libmkl_intel_lp64 can't be dlopened 😑

[giordano]

I'm trying to think how to go about this. My idea was to only add libmkl_intel_lp64 for macOS and Linux, just to make IpotMKL work, otherwise we'd enter an endless rabbit hole. Even so, I don't know how to fix libmkl_intel_lp64. I was thinking about using patchelf/install_name_tool to add a dependency to libmkl_intel_lp64 on whatever library provides the missing symbol, however the missing symbol is provided by multiple libraries:

sandbox:${WORKSPACE} # for file in *.so; do nm -D $file | grep "T \<mkl_blas_dsyrk\>" && echo "Symbol found in $file"; done
0000000000239a50 T mkl_blas_dsyrk
Symbol found in libmkl_gnu_thread.so
0000000000314b50 T mkl_blas_dsyrk
Symbol found in libmkl_intel_thread.so
000000000025ca00 T mkl_blas_dsyrk
Symbol found in libmkl_pgi_thread.so
000000000017b470 T mkl_blas_dsyrk
Symbol found in libmkl_sequential.so
000000000030af30 T mkl_blas_dsyrk
Symbol found in libmkl_tbb_thread.so

Honestly, I don't know which one we should pick up. Given the names of these libraries this looks like something that should be determined dynamically at runtime.

Just for the sake of trying it, I did

sandbox:${WORKSPACE} # patchelf --add-needed libmkl_gnu_thread.so libmkl_intel_lp64.so 

but this still failed to dlopen:

julia> using Libdl; dlopen("./libmkl_gnu_thread.so")
ERROR: could not load library "./libmkl_gnu_thread.so"
./libmkl_gnu_thread.so: undefined symbol: mkl_blas_dgemm_blk_info_bdz

so I looked for the missing symbol and patched libmkl_gnu_thread.so accordingly:

sandbox:${WORKSPACE} # for file in *.so; do nm -D $file | grep "T \<mkl_blas_dgemm_blk_info_bdz\>" && echo "Symbol found in $file"; done
000000000021ad90 T mkl_blas_dgemm_blk_info_bdz
Symbol found in libmkl_core.so
sandbox:${WORKSPACE} # patchelf --add-needed libmkl_core.so libmkl_gnu_thread.so

In this way I've been able to dlopen libmkl_intel_lp64:

julia> using Libdl; dlopen("./libmkl_core.so")
Ptr{Nothing} @0x0000000004b08ba0

julia> using Libdl; dlopen("./libmkl_gnu_thread.so")
Ptr{Nothing} @0x000000000423b250

julia> using Libdl; dlopen("./libmkl_intel_lp64.so")
Ptr{Nothing} @0x0000000002788f40

Again, I don't know which of the libraries we should use for libmkl_intel_lp64, I can't see a good general choice. Suggestions welcome.

[dpo]

My interpretation is that those symbols are missing intentionally and the libraries are not linked together intentionally because the user is expected to pick the right combination for their architecture and application.

I wonder if we should create "combined" shared libs for the various combinations, e.g., libmkl_avx_sequential, whose purpose is solely to link libmkl_avx and libmkl_sequential (and whatever else may be needed).

I worry that patching the libraries will break certain combinations.

[giordano]

My interpretation is that those symbols are missing intentionally and the libraries are not linked together intentionally because the user is expected to pick the right combination for their architecture and application.

Yes, that's my feeling too

[ViralBShah]

Is this necessary? I thought most libraries will link to mkl_rt or so, which in turn will pick the right platform variant. It would be nice to figure this out so that we can start linking other things to MKL.

[giordano]

In an ideal world we shouldn't do this, but MKL is a mess

[ViralBShah]

We are increasingly seeing issues with openblas performance (mostly with its multi-threading), hence I am paying more attention to making it easy to use MKL ecosystem wide.

[giordano]

But at least OpenBLAS comes with a single library, you load that and you're done. MKL has some dozens of libraries, different on each operating system, and you can't even load all of them because some will freak out in some cases. But then third-party libraries will need some of the not-yet-loaded MKL libraries, so what to do? 🤷 Without further intelligence in the library wrappers we are out of luck, or the users will need to sort out loading of the library themselves, good luck with that

[dpo]

I'd love to get this to work but I'm not sure how. Some packages want to link against a bunch of the MKL libs, e.g., https://github.com/JuliaPackaging/Yggdrasil/pull/2144/files#diff-4601b0a333dde98a8dcb9e4897622fa2fe292d3940e4454e51f3653f89c22347R21.

I thought one way would be to create a "super-library" for each platform, that simply links to a consistent subset of the MKL, and export that.

[ViralBShah]

With LBT coming to base, we will have a clean way to use MKL (hopefully).

[dpo]

What's LBT?

[giordano]

libblastrampoline