Juju: statically compiling the static compiler!

[NHDaly]

@lucatrv let's branch out the conversation from #76 to here!

[NHDaly]

--- EDIT: fixed withenv invocation. ---

So there's this cool bit of logic at the bottom of juliac.jl, which conditions running julia_main on whether or not it's being statically compiled, so that you can statically compile juliac and it won't be executed in the process (and error because there's nothing being compiled! 🤯):

if get(ENV, "COMPILE_STATIC", "false") == "false"
    julia_main(ARGS)
end

I don't know how that was intended to be used, but we can use it by simply setting that environment variable before compiling juliac.jl!

julia> withenv(()->build_executable("juliac.jl"; builddir="bin"), "COMPILE_STATIC"=>true)

However, when I try that, i run into this error:

ERROR: LoadError: LoadError: LoadError: GCC wasn't found. Please make sure that gcc is on the path and run Pkg.build("PackageCompiler")
Stacktrace:
 [1] error(::String) at ./error.jl:21
 [2] include_from_node1(::String) at ./loading.jl:576
 [3] include(::String) at ./sysimg.jl:14
 [4] include_from_node1(::String) at ./loading.jl:576
 [5] eval(::Module, ::Any) at ./boot.jl:235
 [6] _require(::Symbol) at ./loading.jl:490
 [7] require(::Symbol) at ./loading.jl:405
 [8] include_from_node1(::String) at ./loading.jl:576
 [9] include(::String) at ./sysimg.jl:14
while loading /Users/daly/.julia/v0.6/PackageCompiler/src/static_julia.jl, in expression starting on line 11
while loading /Users/daly/.julia/v0.6/PackageCompiler/src/PackageCompiler.jl, in expression starting on line 27
while loading /Users/daly/.julia/v0.6/PackageCompiler/juliac.jl, in expression starting on line 1
ERROR: failed process: Process(`/Applications/Julia-0.6.app/Contents/Resources/julia/bin/julia -Cgeneric -J/Applications/Julia-0.6.app/Contents/Resources/ju
lia/lib/julia/sys.dylib --compile=yes --depwarn=yes --compilecache=no --startup-file=no --output-o /Users/daly/.julia/v0.6/PackageCompiler/bin/juliac.o -e '
  empty!(Base.LOAD_CACHE_PATH) # reset / remove any builtin paths
  push!(Base.LOAD_CACHE_PATH, "/Users/daly/.julia/v0.6/PackageCompiler/bin/cache_ji_v0.6.3") # enable usage of precompiled files
  Sys.__init__(); Base.early_init(); # JULIA_HOME is not defined, initializing manually
  include("/Users/daly/.julia/v0.6/PackageCompiler/juliac.jl") # include Julia program file
  empty!(Base.LOAD_CACHE_PATH) # reset / remove build-system-relative paths'`, ProcessExited(1)) [1]
Stacktrace:
 [1] pipeline_error(::Base.Process) at ./process.jl:682
 [2] run(::Cmd) at ./process.jl:651
 [3] build_object(::String, ::String, ::Bool, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void
) at /Users/daly/.julia/v0.6/PackageCompiler/src/static_julia.jl:225
 [4] #static_julia#1(::String, ::Bool, ::Bool, ::String, ::String, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Void, ::Void, ::Void, ::Void, :
:Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::String, ::Void, ::PackageCompiler.#static_julia, ::String) at /Users/daly/.
julia/v0.6/PackageCompiler/src/static_julia.jl:121
 [5] (::PackageCompiler.#kw##static_julia)(::Array{Any,1}, ::PackageCompiler.#static_julia, ::String) at ./<missing>:0
 [6] #build_executable#12(::Void, ::String, ::Bool, ::Bool, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void, ::Void,
::Void, ::Void, ::PackageCompiler.#build_executable, ::String, ::String, ::String) at /Users/daly/.julia/v0.6/PackageCompiler/src/api.jl:87
 [7] (::PackageCompiler.#kw##build_executable)(::Array{Any,1}, ::PackageCompiler.#build_executable, ::String, ::String, ::String) at ./<missing>:0 (repeats
2 times)

I think this code is the problem:
https://github.com/JuliaLang/PackageCompiler.jl/blob/master/src/static_julia.jl#L3-L15

[lucatrv]

Yes, and if you comment using PackageCompiler in juliac.jl you should be able to statically compile it... but the output executalbe would not be very useful... :)

[NHDaly]

haha yes agreed :p

[NHDaly]

Actually, i think the try ... catch is hiding the actual error! It looks like for some reason you can't call out to a shell program from within a script you're compiling, during the compilation.

Trying to compile a script that simply calls to the shell fails:

julia> fname = mktemp()[1]
>> "/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmpqQXJpr"

julia> write(fname, "run(`ls`)")
>> 9

julia> build_executable(fname)
Julia program file:
  "/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmpqQXJpr"
C program file:
  "/Users/daly/.julia/v0.6/PackageCompiler/examples/program.c"
Build directory:
  "/Users/daly/Documents/developer/code/games/pongClone/builddir"
ERROR: LoadError: UndefVarError: uv_eventloop not defined
Stacktrace:
 [1] #spawn#375(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{RawFD,RawFD,RawFD}) at ./process.jl:509
 [2] run(::Cmd) at ./process.jl:650
 [3] include_from_node1(::String) at ./loading.jl:576
 [4] include(::String) at ./sysimg.jl:14
while loading /var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmpqQXJpr, in expression starting on line 1

Is that a bug?
@SimonDanisch, have you seen this before?

[lucatrv]

@NHDaly good catch!! 👍

[NHDaly]

Aha! So, the issue is that, in addition to the Sys.__init__(); Base.early_init() lines, we need to also call Base.__init() I think!!

I got there from this chain:
-- EDIT: oops, these links are to HEAD, instead of v0.6.3.. see #78 (comment) below for updated thoughts.. --

• I think the error comes when spawning a process, when it tries to call eventloop, here:
  https://github.com/JuliaLang/julia/blob/aa301aa60bb7097182c55248572c861361a40b53/base/process.jl#L347
• which returns uv_eventloop.
• uv_eventloop is defined in reinit_stdio(), here:
  https://github.com/JuliaLang/julia/blob/c0a278c3aa4149b86724bf0f22176ddaf3a3756d/base/libuv.jl#L115
• reinit_stdio() is normally called from `Base.init(), here:
  https://github.com/JuliaLang/julia/blob/89c86c148c4e773d3384be40cb2d14ce69849d96/base/sysimg.jl#L476

---

And look, we can confirm that Base.__init__() isn't being called, by checking another thing that function does -- seed the random number generator! As is, during compilation, we're running with Libc unseeded, so it produces the same random numbers every time! D:

julia> mktemp() do f,_  write(f, "println(Libc.rand())");  build_executable(f);  end
Julia program file:
  "/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmp8mE6zx"
C program file:
  "/Users/daly/.julia/v0.6/PackageCompiler/examples/program.c"
Build directory:
  "/Users/daly/.julia/v0.6/PackageCompiler/builddir"
0.8236475079774124
...
julia> mktemp() do f,_  write(f, "println(Libc.rand())");  build_executable(f);  end
...
0.8236475079774124
...
julia> mktemp() do f,_  write(f, "println(Libc.rand())");  build_executable(f);  end
...
0.8236475079774124
...

‼️‼️ Haha oops! :)

This problem is only during compilation, though, since program.c initializes everything correctly, so if the actual program uses a random number or calls an external process, that all works correctly:

julia> mktemp() do f,_
           write(f, "Base.@ccallable julia_main()::Cint = (println(Libc.rand()); return 0)")
           build_executable(f, "rand")
       end
...

julia> run(`./builddir/rand`)
WARNING: redefining constant JULIA_HOME
0.4199228285093486

julia> run(`./builddir/rand`)
WARNING: redefining constant JULIA_HOME
0.19340718496025144

julia> run(`./builddir/rand`)
WARNING: redefining constant JULIA_HOME
0.5302445693273514

So I'll open a PR for that now.

[NHDaly]

hmm, so unfortunately Sys.__init__(), Base.__init__() and Base.early_init() aren't playing well together.. this will require more investigation..
Simply adding Base.__init__() has the following problem:

julia> mktemp() do f,_
           write(f, "Base.__init__(); println(Libc.rand())")
           build_executable(f)
       end
Julia program file:
  "/var/folders/5j/c6kd481j4l71m802wh5qp6vc0000gn/T/tmpVPAAA2"
C program file:
  "/Users/daly/.julia/v0.6/PackageCompiler/examples/program.c"
Build directory:
  "/Users/daly/.julia/v0.6/PackageCompiler/builddir"
WARNING: redefining constant JULIA_HOME
0.5758873948500367
fatal: error thrown and no exception handler available.
ErrorException("Task cannot be serialized")
rec_backtrace at /Users/osx/buildbot/slave/package_osx64/build/src/stackwalk.c:84
record_backtrace at /Users/osx/buildbot/slave/package_osx64/build/src/task.c:245 [inlined]
jl_throw at /Users/osx/buildbot/slave/package_osx64/build/src/task.c:564
jl_error at /Users/osx/buildbot/slave/package_osx64/build/src/rtutils.c:41
jl_serialize_value_ at /Users/osx/buildbot/slave/package_osx64/build/src/dump.c:1040
jl_serialize_value_ at /Users/osx/buildbot/slave/package_osx64/build/src/dump.c:874
...
jl_serialize_value_ at /Users/osx/buildbot/slave/package_osx64/build/src/dump.c:1037
jl_save_system_image_to_stream at /Users/osx/buildbot/slave/package_osx64/build/src/dump.c:2506
jl_create_system_image at /Users/osx/buildbot/slave/package_osx64/build/src/dump.c:2562
jl_write_compiler_output at /Users/osx/buildbot/slave/package_osx64/build/src/precompile.c:61
jl_atexit_hook at /Users/osx/buildbot/slave/package_osx64/build/src/init.c:238
main at /Applications/Julia-0.6.app/Contents/Resources/julia/bin/julia (unknown line)
ERROR: failed process: Process(`/Applications/Julia-0.6.app/Contents/Resources/julia/bin/julia -Cgeneric -J/Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --compilecache=no --startup-file=no --output-o /Users/daly/.julia/v0.6/PackageCompiler/builddir/tmpVPAAA2.o -e '

It did fix the random number, but it somehow breaks serialization...

[NHDaly]

Oh, huh, oops, also i just realized that i was looking at HEAD code, i think... Here's the v0.6.3 Base.init, which does significantly less...:
https://github.com/JuliaLang/julia/blob/v0.6.3/base/sysimg.jl#L395

Actually, i can't find where Libc.srand happens now, but it must be something similar considering the behavior seen above.

[NHDaly]

Okay, i understand this somewhat more after reading the manual:
https://docs.julialang.org/en/v0.6.3/manual/modules/#Module-initialization-and-precompilation-1
In particular:

In order to make your module work with precompilation, however, you may need to change your module to explicitly separate any initialization steps that must occur at runtime from steps that can occur at compile time. For this purpose, Julia allows you to define an __init__() function in your module that executes any initialization steps that must occur at runtime. This function will not be called during compilation (--output-* or __precompile__()). You may, of course, call it manually if necessary, but the default is to assume this function deals with computing state for the local machine, which does not need to be – or even should not be – captured in the compiled image. It will be called after the module is loaded into a process, including if it is being loaded into an incremental compile (--output-incremental=yes), but not if it is being loaded into a full-compilation process.
[Emphasis mine]

So, since we're compiling (output-o), julia is not calling the modules' __init__() functions. The argument is that, quite possibly, they shouldn't be called.

So I think there's a judgement call here for whether it's appropriate for us to be manually initializing modules, and, if so, which ones. It seems reasonable to me that we would manually initialize some things like Sys, but I can see how it could be problematic. This seems like a semi-important design decision, actually.

---

For the particular issues we saw here, we can simply add Base.Random.srand() and Base. reinit_stdio() to the manual initialization we do. But the question I suppose is:

• should we be doing any manual initialization?
• if so, are there more things we should be manually initializing besides what we're currently doing? How can we find the set of all those things?

[lucatrv]

It would be interesting to see how Julia v0.7-alpha behaves, as it executes Base.init_depot_path() and Base.init_depot_path() in place of Sys.__init__() and Base.early_init(), however now it fails on me with the following message:

ERROR: LoadError: ArgumentError: Module Sys not found in current path.
Run `Pkg.add("Sys")` to install the Sys package.

Does it happen also to you?

Some time ago I asked information on proper initialization calls, but did not received an answer yet, maybe we should open an issue on Julia's repository asking for insight?

[NHDaly]

Hey, sorry i didn't respond for so long.. :[

Huh, no, I don't see that particular error. That's weird. Maybe the code has changed since then, but I get different errors:

ERROR: UndefVarError: LOAD_CACHE_PATH not defined

I'm going to try working on julia v0.7 support today as part of the friday #upgradathons! :)