Skip to content

Conversation

@NHDaly
Copy link
Member

@NHDaly NHDaly commented Mar 25, 2019

This fixes compile_package and compile_incremental for packages that have test-only dependencies.

For example, I was seeing this error when I tried to compile CSV:

julia> PackageCompiler.compile_package("CSV", force=false)
[ Info: Recompiling stale cache file /Users/nathan.daly/.julia/compiled/v1.1/CSV/HHBkp.ji for CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]
[ Info: Registered package CSV, using already given UUID: 336ed68f-0bac-5ca0-87d4-7b16caf5d00b
Project CSV v0.4.3
    Status `~/.julia/dev/PackageCompiler/packages/CSV/Project.toml`
  (empty environment)
 Resolving package versions...
  Updating `~/.julia/dev/PackageCompiler/packages/CSV/Project.toml`
  [324d7699] + CategoricalArrays v0.5.2
  [a93c6f00] + DataFrames v0.17.1
  [9a8bc11e] + DataStreams v0.4.1
  [7d512f48] + InternedStrings v0.7.0
  [e1d29d7a] + Missings v0.4.0
  [ea10d353] + WeakRefStrings v0.5.8
  [ade2ca70] + Dates
  [a63ad114] + Mmap
  Updating `~/.julia/dev/PackageCompiler/packages/CSV/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.3
  [324d7699] + CategoricalArrays v0.5.2
  [944b1d66] + CodecZlib v0.5.2
  [34da2185] + Compat v2.0.0
  [a93c6f00] + DataFrames v0.17.1
  [9a8bc11e] + DataStreams v0.4.1
  [864edb3b] + DataStructures v0.15.0
  [7d512f48] + InternedStrings v0.7.0
  [82899510] + IteratorInterfaceExtensions v0.1.1
  [e1d29d7a] + Missings v0.4.0
  [bac558e1] + OrderedCollections v1.0.2
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v0.5.2
  [a2af1166] + SortingAlgorithms v0.3.1
  [2913bbd2] + StatsBase v0.29.0
  [3783bdb8] + TableTraits v0.4.1
  [bd369af6] + Tables v0.1.18
  [3bb67fe8] + TranscodingStreams v0.9.3
  [ea10d353] + WeakRefStrings v0.5.8
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/.julia/dev/PackageCompiler/packages/CSV/Project.toml`
  [9b87118b] + PackageCompiler v0.6.3
  [44cfe95a] + Pkg
  Updating `~/.julia/dev/PackageCompiler/packages/CSV/Manifest.toml`
  [c7e460c6] + ArgParse v0.6.2
  [9e28174c] + BinDeps v0.8.10
  [e1450e63] + BufferedStreams v1.0.0
  [0862f596] + HTTPClient v0.2.1
  [b27032c2] + LibCURL v0.5.0
  [522f3ed2] + LibExpat v0.5.0
  [2ec943e9] + Libz v1.0.0
  [9b87118b] + PackageCompiler v0.6.3
  [b718987f] + TextWrap v0.3.0
  [30578b45] + URIParser v0.4.0
  [c17dfb99] + WinRPM v0.4.2
[ Info: activating new environment at ~/.julia/dev/PackageCompiler/packages/CSV/Project.toml.
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
CSV: Error During Test at /Users/nathan.daly/.julia/packages/CSV/tT4Xy/test/runtests.jl:19
  Got exception outside of a @test
  LoadError: ArgumentError: Package DecFP [55939f99-70c6-5e9b-8bb0-5071ed7d61fd] is required but does not seem to be installed:
   - Run `Pkg.instantiate()` to install all recorded dependencies.

Notice that it never adds DecFP to the new Project.toml, despite CSV.jl having it in its test REQUIRE:
https://github.com/JuliaData/CSV.jl/blob/v0.4.3/test/REQUIRE


This is fixed in this PR, and the final step includes one more output block from Pkg.resolve():

 Resolving package versions...
  Updating `~/.julia/dev/PackageCompiler/packages/CSV/Project.toml`
  [336ed68f] + CSV v0.4.3
  [55939f99] + DecFP v0.4.8
  Updating `~/.julia/dev/PackageCompiler/packages/CSV/Manifest.toml`
  [336ed68f] + CSV v0.4.3
  [55939f99] + DecFP v0.4.8
  [276daf66] + SpecialFunctions v0.7.2
  [9abbd945] + Profile
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
[ Info: activating new environment at ~/.julia/dev/PackageCompiler/packages/CSV/Project.toml.
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
testing test_utf8_with_BOM.csv
...
All done
┌ Info: Not replacing system image.
└ You can start julia with `julia -J /Users/nathan.daly/.julia/dev/PackageCompiler/sysimg/sys.dylib` at a posix shell to load the compiled files.
"/Users/nathan.daly/.julia/dev/PackageCompiler/sysimg/sys.dylib"

This fixes `compile_package` and `compile_incremental` for packages that
have test-only dependencies.
@bjarthur
Copy link

i can't compile DataStructures.jl for the same reason-- it has a test-only dependency on Primes. would be great to see this merged!

@NHDaly
Copy link
Member Author

NHDaly commented Apr 28, 2019

Hmm, for some reason this seems to have broken the tests (sorry i didn't follow-up earlier).

After this change, the "unregistered package with runtests.jl" fails. Based on the output, it seems to be failing during the Pkg.resolve() step I added.

This is the error message:

ERROR: LoadError: Unsatisfiable requirements detected for package TestPackage [d4c01e28]:
 TestPackage [d4c01e28] log:
 ├─TestPackage [d4c01e28] has no known versions!
 └─restricted to versions * by an explicit requirement — no versions left
Stacktrace:
 [1] check_constraints(::Pkg.GraphType.Graph) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/GraphType.jl:935
 [2] Pkg.GraphType.Graph(::Dict{Base.UUID,Set{VersionNumber}}, ::Dict{Base.UUID,Dict{Pkg.Types.VersionRange,Dict{String,Base.UUID}}}, ::Dict{Base.UUID,Dict{Pkg.Types.VersionRange,Dict{String,Pkg.Types.VersionSpec}}}, ::Dict{Base.UUID,String}, ::Dict{Base.UUID,Pkg.Types.VersionSpec}, ::Dict{Base.UUID,Pkg.Types.Fixed}, ::Bool) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/GraphType.jl:362
 [3] deps_graph(::Pkg.Types.Context, ::Dict{Base.UUID,String}, ::Dict{Base.UUID,Pkg.Types.VersionSpec}, ::Dict{Base.UUID,Pkg.Types.Fixed}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:307
 [4] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Nothing) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:371
 [5] resolve_versions! at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:316 [inlined]
 [6] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1244
 [7] #up#29(::Pkg.Types.UpgradeLevel, ::Pkg.Types.PackageMode, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:198
 [8] #up at ./none:0 [inlined]
 [9] #up#24 at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:163 [inlined]
 [10] #up at ./none:0 [inlined]
 [11] resolve at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:203 [inlined] (repeats 2 times)
 [12] top-level scope at none:0
 [13] include at ./boot.jl:317 [inlined]
 [14] include_relative(::Module, ::String) at ./loading.jl:1044
 [15] include(::Module, ::String) at ./sysimg.jl:29
 [16] exec_options(::Base.JLOptions) at ./client.jl:266
 [17] _start() at ./client.jl:425
in expression starting at /Users/travis/build/JuliaLang/PackageCompiler.jl/sysimg/run_julia_code.jl:2

I guess resolve is trying to install the unregistered package somehow? Is there something I should be doing instead of resolve? It's tough to get some of these corner cases right.

@KestutisMa
Copy link

Could this be merged now?

@KristofferC
Copy link
Member

Obsolete by #304

@KristofferC KristofferC closed this Feb 9, 2020
@NHDaly NHDaly deleted the nhdaly-pkg.resolve branch February 13, 2020 18:52
@NHDaly
Copy link
Member Author

NHDaly commented Feb 13, 2020

🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants