Running PackageCompiler from the package itself

[GunnarFarneback]

Let me first say that I have a non-typical use case and expect to fix the problem myself, but could use some guidance to find a good solution. Also all my experiments have been made on the sd-notomls branch. My insights into master and earlier history are negligible.

Use case: I have a C++ application where embedded Julia is used as an optional computation engine for some experimental algorithms. Most (but not all) of the people who run the application with these algorithms have Julia installed solely for this purpose and don't otherwise run Julia themselves. The application has an interface to check out experimental code from a git server, where the user chooses repository and branch. In the Julia case this is a package. After retrieving the code, the application calls an install script within the package, which today basically does activate, instantiate, precompile.

Since there's a minute-long startup time (not quite a deal breaker but very annoying) I want to extend the install script with generation of a custom system image. This does work beautifully when running PackageCompiler manually from "outside" the package and reduces the startup time to the order of seconds, which is entirely satisfactory. For the automation I want to have a subdirectory, e.g. compile, which contains Project.toml and Manifest.toml that include the PackageCompiler dependency, as well as a build script that runs PackageCompiler and a snoopfile.jl that exercises the parts of the code that should go into the system image. I have sort of got this working, but (at least) two problems remain.

1. I had to modify relative_snoop_locations in pkg.jl so it could find compile/snoopfile.jl, as commented in remove complex toml based snooping #204. I might have misunderstood the intention of snoopfile.jl but I interpreted it as an alternative to test/runtests.jl when that for some reason is not good for the purpose. It seems reasonable that this could be placed somewhere else than in the list in relative_snoop_locations, which then needs some way to be specified.

2. When PackageCompiler spawns off new Julia processes, those don't know anything about the environment where compile_incremental was run from but assumes that e.g. LoweredCodeUtils is available in the default environment. In my scenario this assumption doesn't hold and the spawned Julia processes should run with the same active project as was used when calling compile_incremental. Does this sound reasonable in general or is there some better solution? (For snoopfile.jl I could work around this by activating at the top of the file, but for the compilation step this isn't an option.)

[asinghvi17]

Arbitrary kwarg propagation could be a solution?

[KristofferC]

This issue has a lot of references to the old PackageCompiler and it is kind of hard to know what is still relevant. I'll close this but feel free to open an updated version of it.