How to get parallelism to work?

[ranjanan]

This is the file I passed to build_executable

Base.@ccallable function julia_main(ARGS::Vector{String})::Cint
     addprocs(2)
     pmap(x -> println(myid()), 1:2)
     return 0
 end

When I run the executable, I get:

➜  static ./build/buzz
ERROR (unhandled task failure): could not spawn setenv(`/Users/ranjan/graph/static/build/julia -Cgeneric -J/Users/ranjan/graph/static/build/buzz.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker XJ4ngyA8sbBGqZpR`; dir="/Users/ranjan/graph/static"): no such file or directory (ENOENT)
Stacktrace:
 [1] _jl_spawn(::String, ::Array{String,1}, ::Ptr{Void}, ::Base.Process, ::Base.DevNullStream, ::Base.PipeEndpoint, ::Base.TTY) at ./process.jl:360
 [2] #376 at ./process.jl:512 [inlined]
 [3] setup_stdio(::Base.##376#377{Cmd}, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:499
 [4] #spawn#375(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:511
 [5] (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./<missing>:0
 [6] #spawn#372(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:392
 [7] spawn(::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:392
 [8] open(::Base.CmdRedirect, ::String, ::Base.DevNullStream) at /Users/ranjan/graph/static/build/buzz.dylib:?
 [9] launch(::Base.Distributed.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at /Users/ranjan/graph/static/build/buzz.dylib:?
 [10] (::Base.Distributed.##31#34{Base.Distributed.LocalManager,Dict{Any,Any},Array{WorkerConfig,1},Condition})() at ./event.jl:73
fatal: error thrown and no exception handler available.
Base.UVError(prefix="could not spawn setenv(`/Users/ranjan/graph/static/build/julia -Cgeneric -J/Users/ranjan/graph/static/build/buzz.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker XJ4ngyA8sbBGqZpR`; dir="/Users/ranjan/graph/static")", code=-2)
rec_backtrace at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
jl_throw at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/libjulia.dylib (unknown line)
wait at /Users/ranjan/graph/static/build/buzz.dylib (unknown line)
#addprocs_locked#30 at /Users/ranjan/graph/static/build/buzz.dylib (unknown line)
#addprocs#29 at /Users/ranjan/graph/static/build/buzz.dylib (unknown line)
#addprocs#243 at /Users/ranjan/graph/static/build/buzz.dylib (unknown line)
julia_main at /Users/ranjan/graph/static/build/buzz.dylib (unknown line)
julia_main at /Users/ranjan/graph/static/build/buzz.dylib (unknown line)
main at /Users/ranjan/graph/static/./build/buzz (unknown line)

[NHDaly]

Huh, good find...

So I'm not sure exactly why, but it looks like the failure is coming from here:
https://github.com/JuliaLang/julia/blob/v0.6.3/base/distributed/managers.jl#L324

---- EDIT:----------
Oh, jk, the no such file error is coming from the way it's attempting to spawn julia: /Users/ranjan/graph/static/build/julia. That's not a real file. The below thoughts still hold though.

--

It looks like the LocalManager is trying to spawn another julia process, but with flags such that the spawned julia process is set to be compiling (--compile=yes). My guess, therefore, is that maybe the worker it's specifying was created during the static compilation, and no longer exists when the compiled file is executed? (--worker XJ4ngyA8sbBGqZpR)?

Either way, it looks to me like that spawn command is not what we'd want; we wouldn't want to be spawning julia again from the compiled exe, but rather we'd want to be spawning the exe itself, i'd guess...

I would think this will probably require adding a custom ClusterManager, something that is aware-of and specific-to statically-compiled binaries... But maybe I'm wrong, and there are some built-in julia tricks we could use to get that to work!

wdyt?

[NHDaly]

Oh, whoops, no, the actual failure is that it's got the wrong name for the julia process it's trying to spawn:
/Users/ranjan/graph/static/build/julia, which doesn't exist.

I've edited the above comment to reflect that.

For some reason, addprocs isn't seeing the updated JULIA_HOME, which should still reflect the location of the julia exe used to compile the code. The simplest way to "make this work" would be to fix that, and then at least parallelism wouldn't be broken (hopefully), but it's still probably not what we want, since it would still be spawning the julia process, rather than the compiled binary.

The problem is here -- somehow this code isn't seeing the updated JULIA_HOME. Maybe it's somehow getting compiled before we overwrite JULIA_HOME?:
https://github.com/JuliaLang/julia/blob/v0.6.3/base/distributed/cluster.jl#L380-L385

You can see that here:

shell> cd ~/tmp/
julia> mktemp() do f,_
                  write(f, "Base.@ccallable function julia_main(ARGS::Vector{String})::Cint
                              println(Base.JULIA_HOME)
                              println(Base.julia_exename())
                              println(Base.julia_cmd())
                              println(Base.Distributed.default_addprocs_params())
                              return 0
                            end")
                  build_executable(f)
                  run(`$(joinpath("builddir", basename(f)))`)
       end
...
All done
WARNING: redefining constant JULIA_HOME
/Applications/Julia-0.6.app/Contents/Resources/julia/bin
julia
`/Applications/Julia-0.6.app/Contents/Resources/julia/bin/julia -Cgeneric -J/Users/daly/tmp/builddir/tmpAsWEFE.dylib --compile=yes --depwarn=yes`
Dict{Any,Any}(Pair{Any,Any}(:topology, :all_to_all),Pair{Any,Any}(:exeflags, ``),Pair{Any,Any}(:enable_threaded_blas, false),Pair{Any,Any}(:exename, "/Users/daly/tmp/builddir/julia"),Pair{Any,Any}(:dir, "/Users/daly/tmp"))

For some reason, it calculates :exename to be "/Users/daly/tmp/builddir/julia" (which reflects the original value of JULIA_HOME, the binary's location).

---

So there are really two issues here:

1. Why is this code still seeing a wrong value for JULIA_HOME? (I've opened Why is this code _still_ seeing a wrong value for JULIA_HOME?  #85)
   • Originally reported here: juliac fails to compile Blink.jl: JULIA_HOME not defined #47
   • I thought this was supposed to be fixed by Fix JULIA_HOME manually for all builds. #55?
2. Can we get parallelism to work with statically compiled julia code, without invoking the system julia? (we can keep discussing that here!)

[ranjanan]

@vtjnash suggested the following fix.

On Mac:

• Copy the julia binary (just the julia file) to your build folder.
• Change its rpath to the current executable path by doing: install_name_tool -rpath '@executable_path/../lib' '@executable_path' julia

On Windows:

• Just copy the julia binary (just the julia file)

[dwfmarchant]

Thanks for the suggestion - I'm trying to deal with the same issue.

Where you able to get this fix to work? I end up with this (on Mac):

dyld: Library not loaded: @rpath/libjulia.dylib
  Referenced from: /static-julia/addprocs/builddir/julia
  Reason: image not found
dyld: Library not loaded: @rpath/libjulia.dylib
  Referenced from: /static-julia/addprocs/builddir/julia
  Reason: image not found

Haven't been able to try on Windows yet.

[artt]

Any updates on this? :(