`realloc(): invalid pointer` error with a particular combination of package dependencies

[marius311]

I'm using PackageCompiler to compile a large package with many dependencies, and thus far it had worked flawlessly (🙏). A relevant thing is that my package lazily loads PyPlot via Requires. This also has worked totally fine, in that I create_sysimage without PyPlot, and then in sessions which use the custom system image I can using PyPlot and everything works right.

Recently I added one more seemingly unrelated dependency, SparseArrays. The system image still builds, but now when I use it and try to using PyPlot I get a:

realloc(): invalid pointer

signal (6): Aborted
in expression starting at /home/cosmo/CMBLensing/src/plotting.jl:3

where plotting.jl:3 is

using .PyPlot

from inside the lazily loaded code (full stack trace below).

I have confirmed that simply removing the using SparseArrays from my package fixes the issue.

I tried coming up with a MWE but none of the simple things I tried still reproduce the error. However I have a Docker container with which this is 100% reproducible:

docker run marius311/cmblensing.jl:pkgcompilerissue julia -e "using PyPlot"

and the exact source which built this container available at this commit (built with --build-arg RUNDOCS=0 fwiw). Among other things, its PackageCompiler v1.1.0, PyPlot v2.9.0, and julia 1.4.0, and python 3.7.3 compiled via PyEnv with a dynamically linked libpython.

I'm happy to give any more info that I can. Any ideas how to track this down further or possible fixes? Thanks.

$ docker run marius311/cmblensing.jl:pkgcompilerissue julia -e "using PyPlot"
realloc(): invalid pointer

signal (6): Aborted
in expression starting at /home/cosmo/CMBLensing/src/plotting.jl:3
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fc501ec538d)
unknown function (ip: 0x7fc501ecd4db)
realloc at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
arraylist_grow at /buildworker/worker/package_linux64/build/src/support/arraylist.c:58 [inlined]
arraylist_push at /buildworker/worker/package_linux64/build/src/support/arraylist.c:69
jl_module_using at /buildworker/worker/package_linux64/build/src/module.c:484
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:669
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:872
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:872
include at ./Base.jl:377
include at /home/cosmo/CMBLensing/src/CMBLensing.jl:1
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:369
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:458
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:409 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:817
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:911
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:814
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:843
eval at ./boot.jl:331 [inlined]
eval at /home/cosmo/CMBLensing/src/CMBLensing.jl:1 [inlined]
#316 at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:85
unknown function (ip: 0x7fc4c4d1e9ac)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
err at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:42
#315 at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:84
unknown function (ip: 0x7fc4c4d1cafc)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
withpath at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:32
#314 at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:83
unknown function (ip: 0x7fc4c4d1ca8c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:643
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:693
#invokelatest#1 at ./essentials.jl:712 [inlined]
invokelatest at ./essentials.jl:711 [inlined]
foreach at ./abstractarray.jl:1919 [inlined]
loadpkg at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:22
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:643
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:693
#invokelatest#1 at ./essentials.jl:712 [inlined]
invokelatest at ./essentials.jl:711 [inlined]
require at ./loading.jl:930
require at ./loading.jl:922
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:399 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:436
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:656
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:843
eval at ./boot.jl:331
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
exec_options at ./client.jl:264
_start at ./client.jl:484
jfptr__start_20851 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
unknown function (ip: 0x401931)
unknown function (ip: 0x401533)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 47837620 (Pool: 47829434; Big: 8186); GC: 48

signal (11): Segmentation fault
in expression starting at /home/cosmo/CMBLensing/src/plotting.jl:3
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fc501ec538d)
unknown function (ip: 0x7fc501ecd4db)
realloc at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
arraylist_grow at /buildworker/worker/package_linux64/build/src/support/arraylist.c:58 [inlined]
arraylist_push at /buildworker/worker/package_linux64/build/src/support/arraylist.c:69
jl_module_using at /buildworker/worker/package_linux64/build/src/module.c:484
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:669
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:872
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:872
include at ./Base.jl:377
include at /home/cosmo/CMBLensing/src/CMBLensing.jl:1
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:369
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:458
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:409 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:817
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:911
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:814
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:843
eval at ./boot.jl:331 [inlined]
eval at /home/cosmo/CMBLensing/src/CMBLensing.jl:1 [inlined]
#316 at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:85
unknown function (ip: 0x7fc4c4d1e9ac)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
err at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:42
#315 at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:84
unknown function (ip: 0x7fc4c4d1cafc)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
withpath at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:32
#314 at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:83
unknown function (ip: 0x7fc4c4d1ca8c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:643
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:693
#invokelatest#1 at ./essentials.jl:712 [inlined]
invokelatest at ./essentials.jl:711 [inlined]
foreach at ./abstractarray.jl:1919 [inlined]
loadpkg at /home/cosmo/.julia/packages/Requires/qy6zC/src/require.jl:22
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:643
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:693
#invokelatest#1 at ./essentials.jl:712 [inlined]
invokelatest at ./essentials.jl:711 [inlined]
require at ./loading.jl:930
require at ./loading.jl:922
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:399 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:436
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:656
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:843
eval at ./boot.jl:331
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
exec_options at ./client.jl:264
_start at ./client.jl:484
jfptr__start_20851 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
unknown function (ip: 0x401931)
unknown function (ip: 0x401533)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 47837620 (Pool: 47829434; Big: 8186); GC: 48

[marius311]

I've boiled it down to a much simpler MWE (~50LOC) and made a repo for it: https://github.com/marius311/PkgCompiler_Issue379

There's two ways to fix the error in this MWE. Either delete any of the using statements (I checked deleting any of a bunch of them, it doesn't seem tied to a specific package), or do the code swap mentioned on these lines (FWIW this second way doesn't work in my non-MWE but figured it might be a hint).

I put a copy of the compiled container also on Docker Hub so you can see the error with:

docker run --rm -it marius311/pkgcompiler_issue379 julia -e "using PyPlot"

Let me know if there's anything else I might do to help track this down, its been really hard to find any workaround in my real case so I'm motivated to help find a solution here.

[KristofferC]

I tried this now (with Julia 1.6.2) and things seem to work well for me:

julia> create_sysimage([:Foo],cpu_target="generic", sysimage_path="sys.dylib")
...


❯ julia -q --project --sysimage="sys.dylib"

julia> @time using Foo
  0.000353 seconds (1.11 k allocations: 82.469 KiB, 619.63% compilation time)

so please try again and report back.