Consider using zig cc to make prebuilt binaries

[creationix]

https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html

I think this might enable us to build all the prebuilt luvi binaries using a single linux host such as our CI servers.

[rphillips]

Looks extremely promising.

[squeek502]

Was thinking about this as well. Would be interesting to try.

[zhaozg]

+1

[squeek502]

Played around with it a bit. Was able to get luv to compile but the resulting LuaJIT fails to load the resulting luv.so.

git clone https://github.com/luvit/luv.git
cd luv
export CC="zig cc"
export ASM="zig cc"
make
./build/luajit tests/run.lua
PANIC: unprotected error in call to Lua API (error loading module 'luv' from file './luv.so':
        ./luv.so: undefined symbol: lua_status
stack traceback:
        [C]: at 0x002998b0
        [C]: in function 'require'
        ./lib/tap.lua:1: in main chunk
        [C]: in function 'require'
        tests/run.lua:6: in main chunk
        [C]: at 0x00218770)

zig cc is super new so I'd guess something from ziglang/zig#4784 is the cause.

---

EDIT: zig cc-compiled LuaJIT does not include any symbols:

$ nm -gD ./buildgcc/luajit | grep " T " | wc -l
153
$ nm -gD ./build/luajit | grep " T " | wc -l
0

(./buildgcc/luajit is built with gcc, ./build/luajit is built with zig cc)

---

EDIT#2: My guess is that its the unfinished "passing args to the linker" bullet point, as the end-result LuaJIT compile command ends up being:

/usr/local/bin/zig  cc -O2 -g -DNDEBUG  -Wl,--export-dynamic -rdynamic CMakeFiles/luajit.dir/deps/luajit/src/luajit.c.o CMakeFiles/luajit.dir/deps/luajit/src/ljamalg.c.o CMakeFiles/luajit.dir/lj_vm.S.o  -o luajit  -lm

so I would bet that --export-dynamic is not being passed to the linker.

---

EDIT#3: Running the above directly gives the output:

warning: unsupported linker arg: --export-dynamic

coming from here, presumably.

---

EDIT#4: Manually just added --export-dynamic to the linker args in link.cpp just to get past the error above. Now I'm running into a problem with mismatched glibc versions (zig cc is compiling against a newer one than my system glibc; specifically it's trying to use fnctl64 which my glibc doesn't have). Zig has the ability to target specific glibc versions (e.g. -target native-native-gnu.2.23) but I'm not sure it affects zig cc.

This is worth noting because it will potentially affect the portability of zig cc-compiled luvi binaries (right now we use holy-build-box to maximize the portability of our Linux binaries).

[andrewrk]

I'm happy to work with y'all on this, give me a couple days to fix up that flag integration like @squeek502 noted above and then let's coordinate on this use case.

For glibc versions, as long as you know which glibc version you want to target and we add some way to communicate that info to zig cc then it should work. I'm sure we can figure it out.

holy-build-box readme says:

Binaries work on pretty much any glibc-2.12-or-later-based x86 and x86-64 Linux distribution released since approx 2011. A non-exhaustive list:

So you should be able to use -target x86_64-linux-gnu.2.12. That's zig-specific -target syntax, which is actually not supposed to work; I need to change it to be llvm syntax, but before making that change I'll make sure there's a way to specify the glibc version. It seems that two wrongs do make a right after all :)

[squeek502]

I have a feeling the tough part will be getting OpenSSL to compile. We use a CMake ExternalProject for it and call out to make/nmake and let OpenSSL do its (fairly complicated) thing. Our tiny builds (without OpenSSL and a few other things) will probably be pretty easy to get working by comparison.

[andrewrk]

One problem with cmake I learned this week is that it does not have a separate concept of a "host compiler" and a "target compiler". So unless the upstream project adds support for it, there will be an issue when a project builds a binary and then tries to run it. A better build system would allow you to choose which binaries were native and which were for the target, so that you could build your native tools natively, and build your target code for the target.

[creationix]

Instead of using holy-build-box libc, I'd prefer if we could do a static build with musl. Then it will work on even more linux versions than our old binaries. I had trouble before trying to build static luajit binaries, but I seem to remember that was exactly one of the examples in @andrewrk's blog post.

[squeek502]

Was actually able to build a static musl binary of the luvi tiny variant by doing:

export CC="zig cc -target native-native-musl -fno-sanitize=undefined"
export ASM="zig cc"
make tiny
make

(the -fno-sanitize=undefined is to get around Illegal instruction errors from generating vm_arch.h during the LuaJIT build process, see this comment).

However, when running the resulting luvi binary, I'm getting:

PANIC: unprotected error in call to Lua API ([string "return require('init')(...)"]:1: module 'init' not found:

meaning that something is going wrong with the embedding of src/lua/init.lua via CMake here.

[squeek502]

Oh, just remembered that I tried making a musl build of luv/luajit and ran into:

PANIC: unprotected error in call to Lua API (error loading module 'luv' from file './luv.so':
        Dynamic loading not supported

That is, when compiled statically, musl only has a stub for dlopen (reasoning for this can be found in various places, e.g. here, here). This means that fully static builds are probably untenable for Luvi, since dynamically loading modules is definitely necessary.

[SinisterRectus]

Also https://github.com/phusion/holy-build-box/blob/master/README.md#why-statically-linking-to-glibc-is-a-bad-idea

[creationix]

This means that fully static builds are probably untenable for Luvi, since dynamically loading modules is definitely necessary.

True, anybody needing to load native modules will need this. I wonder if it's worth the effort to have a static build that disables loading native modules. Even ffi use cases often use dlopen to call the libraries.

[andrewrk]

The other option zig cc provides you is "static except glibc" builds, and if you pick an old enough glibc version (e.g. matching holy-build-box) then your binary tarballs will work on any linux distro that uses glibc and standard dynamic linker path (which is most). This is the $arch-linux-gnu target.

The nice thing about static tarballs is they work on all linux distros, even ones like NixOS which have non-standard dynamic linker paths, and alpine linux which uses musl instead of glibc.

[andrewrk]

Alright, I've done some polishing on this feature, and it's about 1 week until the 0.6.0 release. If you're still exploring this use case, now would be a good time to try it out and report issues that you run into. Let me know if I can help.