Skip to content

improve documentation #604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 13, 2021
Merged

improve documentation #604

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/make.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ makedocs(
"Home" => "index.md",

"Manual" => [
"sysimages.md"
"libs.md"
"apps.md"
"sysimages.md",
"apps.md",
"libs.md",
],

"Examples" => [
Expand All @@ -34,5 +34,5 @@ makedocs(

deploydocs(
repo = "github.com/JuliaLang/PackageCompiler.jl.git",
push_preview = true,
push_preview = false,
)
205 changes: 110 additions & 95 deletions docs/src/apps.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,71 +12,17 @@ original Julia source code for apps since everything gets baked into the
sysimage.


## [Relocatability](@id relocatability)

Since we want to send the app to other machines the app we create must be
"relocatable". With an app being relocatable we mean it does not rely on
specifics of the machine where the app was created. Relocatability is not an
absolute measure, most apps assume some properties of the machine they will run
on, like what operating system is installed and the presence of graphics
drivers if one wants to show graphics. On the other hand, embedding things into
the app that is most likely unique to the machine, such as absolute paths to
libraries, means that the application almost surely will not run properly on
another machine.

For something to be relocatable, everything that it depends on must also be
relocatable. In the case of an app, the app itself and all the Julia packages
it depends on must also relocatable. This is a bit of an issue because the
Julia package ecosystem has rarely given much thought to relocatability
since creating "apps" has not been common.

The main problem with relocatability of Julia packages is that many packages
are encoding fundamentally non-relocatable information *into the source code*.
As an example, many packages tend to use a `build.jl` file (which runs when the
package is first installed) that looks something like:

```julia
lib_path = find_library("libfoo")
write("deps.jl", "const LIBFOO_PATH = $(repr(lib_path))")
```
## Creating an app

The main package file then contains:
The source of an app is a package with a project and manifest file.
You can easily create this using the package manager:

```julia
module Package

if !isfile("../build/deps.jl")
error("run Pkg.build(\"Package\") to re-build Package")
end
include("../build/deps.jl")
function __init__()
libfoo = Libdl.dlopen(LIBFOO_PATH)
end

...

end # module
using Pkg
Pkg.generate("MyApp")
```

The absolute path to `lib_path` that `find_library` found is thus effectively
included into the source code of the package. Arguably, the whole build system
in Julia is inherently non-relocatable because it runs when the package is
being installed which is a concept that does not make sense when distributing
an app.

Some packages do need to call into external libraries and use external binaries
so the question then arises: "how are these packages supposed to do this in a
relocatable way?" The answer is to use the "artifact system" introduced in
Julia 1.3, and described in the following [blog
post](https://julialang.org/blog/2019/11/artifacts). The artifact system is a
declarative way of downloading and using "external files" like binaries and
libraries. How this is used in practice is described later. Another useful
tool is the Julia package [RelocatableFolders.jl](https://github.com/JuliaPackaging/RelocatableFolders.jl).

## Creating an app

The source of an app is a package with a project and manifest file.
It should define a function with the signature
In the code for the package, there should be a function with the signature

```julia
Base.@ccallable function julia_main()::Cint
Expand All @@ -85,14 +31,10 @@ Base.@ccallable function julia_main()::Cint
end
```

which will be the entry point of the app (the function that runs when the
defined, which will be the entry point of the app (the function that runs when the
executable in the app is run). A skeleton of an app to start working from can
be found [here](https://github.com/JuliaLang/PackageCompiler.jl/tree/master/examples/MyApp).

Regarding relocatability, PackageCompiler provides a function
[`audit_app(app_dir::String)`](@ref) that tries to find common problems with
relocatability in the app.

The app is then compiled using the [`create_app`](@ref) function that takes a
path to the source code of the app and the destination where the app should be
compiled to. This will bundle all required libraries for the app to run on
Expand All @@ -112,18 +54,21 @@ julia> create_app("MyApp", "MyAppCompiled")
julia> exit()

~/PackageCompiler.jl/examples
❯ MyAppCompiled/bin/MyApp
❯ MyAppCompiled/bin/MyApp foo bar --julia-args -t4
ARGS = ["foo", "bar"]
Base.PROGRAM_FILE = "MyAppCompiled/bin/MyApp"
...
Hello, World!
Base.PROGRAM_FILE = "./bin/MyApp"
DEPOT_PATH = ["/home/kc/JuliaPkgs/PackageCompiler.jl/MyAppCompiled/share/julia"]
LOAD_PATH = ["/home/kc/Juli

...
Running the artifact
The result of 2*5^2 - 10 == 40.000000
unsafe_string((Base.JLOptions()).image_file) = "/Users/kristoffer/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp.dylib"
Example.domath(5) = 10
```

Note that the arguments passed to the executable are available in the global variable `ARGS`.
Standard julia arguments (like how many threads should be used) are passed in after the
`--julia-args` argument.

The resulting executable is found in the `bin` folder in the compiled app
directory. The compiled app directory `MyAppCompiled` could now be put into an
archive and sent to another machine or an installer could be wrapped around the
Expand All @@ -146,17 +91,17 @@ REPL that it provides. For apps, this is no longer the case, the sysimage is
not meant to be used when working interactively, it only needs to be
specialized for the specific app. Therefore, by default, `incremental=false` is
used for `create_app`. If, for some reason, one wants an incremental sysimage,
`incremental=true` could be passed to `create_app`. With the example app, a
`incremental=true` could be passed to `create_app`. With the example app, a
non-incremental sysimage is about 70MB smaller than the default sysimage.

### Filtering stdlibs

By default, all standard libraries are included in the sysimage. It is
By default, all standard libraries are included in the sysimage. It is
possible to only include those standard libraries that the project needs. This
is done by passing the keyword argument `filter_stdlibs=true` to `create_app`.
This causes the sysimage to be smaller, and possibly load faster. The reason
This causes the sysimage to be smaller, and possibly load faster. The reason
this is not the default is that it is possible to "accidentally" depend on a
standard library without it being reflected in the Project file. For example,
standard library without it being reflected in the Project file. For example,
it is possible to call `rand()` from a package without depending on Random,
even though that is where the method is defined. If Random was excluded from
the sysimage that call would then error. As another example, matrix
Expand All @@ -168,12 +113,75 @@ just loading those packages can cause code to change behavior.
Nevertheless, the option is there to use. Just make sure to properly test the
app with the resulting sysimage.


### Custom binary name

By default, the binary in the `bin` directory take the name of the project,
as defined in `Project.toml`. If you want to change the name, you can pass
`app_name="some_app_name"` to `create_app`.


## [Relocatability](@id relocatability)

Since we want to send the app to other machines the app we create must be
"relocatable". With an app being relocatable we mean it does not rely on
specifics of the machine where the app was created. Relocatability is not an
absolute measure, most apps assume some properties of the machine they will run
on, like what operating system is installed and the presence of graphics
drivers if one wants to show graphics. On the other hand, embedding things into
the app that is most likely unique to the machine, such as absolute paths to
libraries, means that the application almost surely will not run properly on
another machine.

For something to be relocatable, everything that it depends on must also be
relocatable. In the case of an app, the app itself and all the Julia packages
it depends on must also relocatable. This is a bit of an issue because the
Julia package ecosystem has rarely given much thought to relocatability
since creating "apps" has not been common.

The main problem with relocatability of Julia packages is that many packages
are encoding fundamentally non-relocatable information *into the source code*.
As an example, many packages tend to use a `build.jl` file (which runs when the
package is first installed) that looks something like:

```julia
lib_path = find_library("libfoo")
write("deps.jl", "const LIBFOO_PATH = $(repr(lib_path))")
```

The main package file then contains:

```julia
module Package

if !isfile("../build/deps.jl")
error("run Pkg.build(\"Package\") to re-build Package")
end
include("../build/deps.jl")
function __init__()
libfoo = Libdl.dlopen(LIBFOO_PATH)
end

...

end # module
```

The absolute path to `lib_path` that `find_library` found is thus effectively
included into the source code of the package. Arguably, the whole build system
in Julia is inherently non-relocatable because it runs when the package is
being installed which is a concept that does not make sense when distributing
an app.

Some packages do need to call into external libraries and use external binaries
so the question then arises: "how are these packages supposed to do this in a
relocatable way?" The answer is to use the "artifact system" introduced in
Julia 1.3, and described in the following [blog
post](https://julialang.org/blog/2019/11/artifacts). The artifact system is a
declarative way of downloading and using "external files" like binaries and
libraries. How this is used in practice is described later. Another useful
tool is the Julia package [RelocatableFolders.jl](https://github.com/JuliaPackaging/RelocatableFolders.jl).

### Artifacts

The way to depend on external libraries or binaries when creating apps is by
Expand Down Expand Up @@ -214,46 +222,53 @@ This is a problem that the Julia standard libraries themselves have:

```julia-repl
julia> @which rand()
rand() in Random at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Random/src/Random.jl:256
rand() in Random at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/1.7/Random/src/Random.jl:256
```

#### Using reflection and finding lowered code

There is nothing preventing someone from starting Julia with the sysimage that
comes with the app. And while the source code is not available one can read
comes with the app. In fact, to support distributed computing that needs to
start up new Julia processes, there is also a `julia` executable in the `bin` folder
that uses the custom sysimage. While the source code is not available one can read
the "lowered code" and use reflection to find things like the name of fields in
structs and global variables etc:

```julia-repl
~/PackageCompiler.jl/examples/MyAppCompiled/bin kc/docs_apps*
❯ julia -q -JMyApp.so
julia> MyApp = Base.loaded_modules[Base.PkgId(Base.UUID("f943f3d7-887a-4ed5-b0c0-a1d6899aa8f5"), "MyApp")]
MyApp
~/PackageCompiler.jl/examples/MyAppCompiled/bin
❯ ./julia -q

julia> names(MyApp; all=true)
10-element Array{Symbol,1}:
18-element Vector{Symbol}:
Symbol("#1#3")
Symbol("#2#4")
Symbol("#eval")
Symbol("#fooifier_path")
Symbol("#include")
Symbol("#is_crayons_loaded")
Symbol("#julia_main")
Symbol("#real_main")
:MyApp
:eval
:fooifier_path
:include
:is_crayons_loaded
:julia_main
:myrand
:o
:outputo
:real_main
:socrates

julia> @code_lowered MyApp.real_main()
CodeInfo(
1 ─ %1 = MyApp.ARGS
│ value@_2 = %1
│ %3 = Base.repr(%1)
│ Base.println("ARGS = ", %3)
│ value@_2
│ %6 = Base.PROGRAM_FILE
│ value@_3 = %6
│ %8 = Base.repr(%6)
│ Base.println("Base.PROGRAM_FILE = ", %8)
│ value@_3
│ %11 = MyApp.DEPOT_PATH
```
1 ─ Core.NewvarNode(:(#2))
│ Core.NewvarNode(:(n))
│ %3 = MyApp.ARGS
│ value@_17 = %3
│ %5 = Base.repr(%3)
│ Base.println("ARGS = ", %5)
│ value@_17
│ %8 = Base.PROGRAM_FILE
│ value@_16 = %8
│ %10 = Base.repr(%8)
...
10 changes: 9 additions & 1 deletion docs/src/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@ come across.

## Installation instructions

The package is installed using the standard way with the package manager:

```julia
using Pkg
Pkg.add("PackageCompiler")
```

!!! note
It is strongly recommended to use the official binaries that are downloaded from
https://julialang.org/downloads/. Distribution-provided Julia installations are
Expand All @@ -59,7 +66,8 @@ To use PackageCompiler a C-compiler needs to be available:
### macOS, Linux

Having a decently modern `gcc` or `clang` available should be enough to use PackageCompiler on Linux or macOS.
For macOS, this can be the built-in Xcode command line tools or `homebrew` and for Linux, the system package manager should work fine.
For macOS, this can be the built-in Xcode command line tools or `homebrew` and for Linux, using the system package
manager to get a compiler should work fine.

### Windows

Expand Down
3 changes: 1 addition & 2 deletions docs/src/libs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@ julia> create_library("MyLib", "MyLibCompiled";
lib_name="libinc",
precompile_execution_file="MyLib/build/generate_precompile.jl",
precompile_statements_file="MyLib/build/additional_precompile.jl",
header_files = ["MyLib/inc.h"])
┌ Warning: it is not recommended to create an app without a preexisting manifest
header_files = ["MyLib/build/mylib.h"])
└ @ PackageCompiler ~/.julia/dev/PackageCompiler/src/PackageCompiler.jl:903
[ Info: PackageCompiler: creating base system image (incremental=false)...
[ Info: PackageCompiler: creating system image object file, this might take a while...
Expand Down
Loading