Multi-stage incremental builds

[Byrth]

https://discourse.julialang.org/t/unable-to-find-compatible-target-in-system-image/52879

The idea here is that I'm trying to build a Julia sysimage in multiple stages so the code I am developing can be quickly deployed to staging / built for release. I have one fairly static stage that contains my dependencies and then have another stage that contains the application that I am developing.

Both stages are built on the same machine in the same multi-stage dockerfile (using DOCKER_BUILDKIT=1)

#23 347.7 [ Info: PackageCompiler: creating system image object file, this might take a while...
#23 347.7 ERROR: Unable to find compatible target in system image.
#23 348.8 ERROR: LoadError: failed process: Process(`/root/julia/bin/julia --color=yes --startup-file=no '--cpu-target=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)' --sysimage=/root/julia/lib/julia/sys.so --project=/root/myproj.jl/inner.jl --output-o=/tmp/jl_UGJKKB.o -e '...'`, ProcessSignaled(11))

As you can see, it is using the default cpu-target for x86-64, which is appropriate for the VM it is building on. When I force that as was recommended on discourse, I correspondingly get the same result. When I try to force generic,-cx16, I get a different error about my VM not supporting cx16 which is unexpected because I excluded it.

Julia 1.5.3
PackageCompiler 1.2.4
CentOS 7 docker file using devtoolkit-9 to get a modern gcc version.

[Byrth]

To add more information, I have problems when I build a sysimage on a Cascade Lake Xeon processor using default_app_cpu_target() (generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)) and then try to use it on a Broadwell Xeon processor.

Looking at the different flags available on each processor (from less /proc/cpuinfo):
Cascade Lake only - invpcid_single, ibrs_enhanced, avx512f, avx512dq, clflushopt, clwb, avx512bw, avx512vl, xsavec, pku, ospke
Broadwell only - hle, rtm

Based on some docs, I take it that the default is building for:

• generic for no AVX
• sandybridge (without xsaveopt and explicitly enforcing the default non-cloning behavior) for functions that benefit from AVX
• haswell (without rdrand and based on the sandybridge image rather than generic) for functions that benefit from AVX2
• No support in the default for AVX512

I am guessing that the -feature options are provided to prevent excessive duplication for minor operational benefit (and I see in the notes that rnrnd and xsaveopt are disabled in llvm anyway). Regardless, the Broadwell processor should be using the Haswell version of the functions (falling back to the other two). The fact that it fails to load with "Unable to find compatible target in system image" means that it does not recognize Haswell as applicable when it is compiled on my Cascade Lake VM and run on the Broadwell VM. Notably, compiling it with cpu_target="generic" on Cascade Lake and then moving to Broadwell does work, so cx16 is not the problem this time. I cannot see from the processor_x86.cpp code how I would print either the flags for the system image or the flags it sees as the target. I don't see the structure defined there in the first few hundred bytes of the system image (I was hoping it would be in a header.)

If I wanted to add explicit support for Broadwell to the x86_64 default compiled on cascade lake, perhaps I could use cpu_target="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1);broadwell,-rtm,base(2)"?

Reading all this documentation, I am only increasingly convinced that the host architecture for compilation should not matter (in theory) for relocatability because we are defining the targets. Perhaps this is a julia issue rather than a PackageCompiler one?

[KristofferC]

Perhaps this is a julia issue rather than a PackageCompiler one?

I think so. The only thing PackageCompiler does is to pass along the cpu_target to Julia which builds the sysimage for us:

PackageCompiler.jl/src/PackageCompiler.jl

Lines 226 to 228 in 4965668

	cmd = `$(get_julia_cmd()) --cpu-target $cpu_target
	--sysimage=$tmp_corecompiler_ji
	-g1 -O0 --output-ji=$tmp_sys_ji $new_sysimage_source_path`

So you should get the same error in case you try to build Julia itself with this cpu_target. I would try that and open an issue on Julia.

[Byrth]

I changed jobs and no longer have access to those servers, but good to know anyway.