cuda artifacts are downloaded repeatedly per Julia session

[arash-banadaki]

I use Julia Version 1.7.0 (2021-11-30) on a machine with a CUDA-enabled card.
I see that Julia keeps downloading the same artifacts over and over whenever you submit a code that invokes CUDA functionalities on the GPU.
To clarify if you submit the following code and exit Julia and come back again and submit the same code you will have to wait again for Julia to download all CUDA artifacts. However, if you repeat the command within the same session (without exiting Julia) it will not download the artifacts again. The code runs successfully eitherway.
Is this behavior intended?

using CUDA, LinearAlgebra
function cuinv(m::Matrix{T}) where T
    A = CuArray(m)
    B = CuArray(Matrix{T}(I(size(A,1))))
    A, ipiv = CUDA.CUSOLVER.getrf!(A)
    Matrix{T}(CUDA.CUSOLVER.getrs!('N', A, ipiv, B))
end

A = rand(100,100)
B = cuinv(A)

I have already posted this issue on the CUDA.jl repository and it seems that the issue is not caused by CUDA.jl.

[IanButterworth]

May be related to #2929

Do you just see it for the CUDA_compat artifact?

[arash-banadaki]

Not sure if it is related to #2929. I don't see any SHA mismatch errors.
It also never gets to downloading CUDA_compat. It keeps downloading the CUDA artifacts every time. And when it is done it continues on executing the submitted code with no issues.
The problem is that I have to wait for these artifacts to finish downloading for minutes every time I exit Julia and come back. It only happens when using the CUDA package.
Oh. I should mention that I have limited storage in my ~ directory ~5GB and CUDA artifacts are large (250MBs each). Not sure if it has anything to do with what I am observing.

[maleadt]

Oh. I should mention that I have limited storage in my ~ directory ~5GB and CUDA artifacts are large (250MBs each). Not sure if it has anything to do with what I am observing.

Try freeing up some space just to be sure. It might be the extraction that fails; the contents can be well over 1GB

[arash-banadaki]

Oh. I should mention that I have limited storage in my ~ directory ~5GB and CUDA artifacts are large (250MBs each). Not sure if it has anything to do with what I am observing.

Try freeing up some space just to be sure. It might be the extraction that fails; the contents can be well over 1GB

Ok. I freed up some space. Now almost all 5GB is occupied by the ~/.julia/artifacts. The reported behavior did not change. My guess is that there are still more executable to download and I ran out of space.
Assuming neither of you experience this behavior, can one of you submit the following command and check how much space is occupied on your disk by the directory that this command returns?

julia> CUDA.ptxas()

[maleadt]

Around 1.4GB, but it needs additional space when extracting.

Can you simplify your MWE like here: JuliaGPU/CUDA.jl#542 (comment)? That will make it easier for other people to reproduce without needing a functional GPU/CUDA set-up.

[arash-banadaki]

Ok. Before moving on to simplifying the problem even further, can I ask you to clarify something?
When I asked you about the size of the CUDA.ptxas(), what I really meant was the size of the entire artifact directory belonging to CUDA and not just the bin directory in that artifact.
So to clarify, CUDA.ptxas() returns a path that should look like: ~/.julia/artifacts/0ed779f9e1d7b0c9905eb5be2c4349d32fbeb050/bin/ptxas. I want to know how big ~/.julia/artifacts/0ed779f9e1d7b0c9905eb5be2c4349d32fbeb050 is. Please also get a ls -l from the lib directory in your CUDA artifact directory. For me that path looks like: ls -l ~/.julia/artifacts/0ed779f9e1d7b0c9905eb5be2c4349d32fbeb050/lib. I have already posted the contents of mine here.

[maleadt]

Yes, that was a du -hs of the entire artifact directory. You can check your contents by downloading the source files here: https://github.com/JuliaBinaryWrappers/CUDA_loader_jll.jl/releases, but they generally shouldn't have been extracted if Julia wasn't been able to verify the entire tree (that's the git-tree-sha in the Artifacts.toml).

[Sixzero]

For me downloading CUDA_Runtime finished with its download (I think) at least 4 times and after the 5th time it didn't start again, but I guess then things started working, but on my internet which is not crazy fast all these took I think at least an hour.
If someone knows why downloads can repeat, then I would love to have a fix for this, it is quite frustrating to wait so long.

[maleadt]

I've seen this happen too when a download fails. Ideally Pkg would support resuming partial downloads of these large artifacts.

[Sixzero]

Also as a note: I am not sure if the downloads didn't finish or if it just restarted after a successful download(thinking it is still not downloaded).

[carstenbauer]

Just hit this on a system with terribly slow internet. It's super annoying... I don't know but my feeling is that the download(s) didn't fail. No idea why it still repeats.

[LIU-FIR]

The same situation occurred on me. I guess the artifact CUDA_Runtime downloading process repeated at least 4 times. At the last time, the downloading process became unbearable slow. I am not quite sure if it would success in the end. :-(

For me downloading CUDA_Runtime finished with its download (I think) at least 4 times and after the 5th time it didn't start again, but I guess then things started working, but on my internet which is not crazy fast all these took I think at least an hour. If someone knows why downloads can repeat, then I would love to have a fix for this, it is quite frustrating to wait so long.

[maleadt]

Ran into this once again. The issue now was that unpacking the artifact failed due to insufficient free space in /home. Presumably because of that failure Pkg.jl proceeded to download the 1GB artifact about 4 times, which on a slow internet connection would have been horrible.

[kh-abd-kh]

I hate myself everytime i use julia.
i am trying to download
using DynamicPolynomials
and julia downloads

Downloading artifact: CUDA_Runtime
Downloading [=> ] 0.3 % %

i have limited space and very limited internet quota/month

Please consider that there are some humans in this universe with very slow internet
or tell them, go away julia is not for you
it is a true torture.

I hate julia.

[vchuravy]

@kh-abd-kh I am sorry for your frustration. https://discourse.julialang.org is a better place to seek help than this issue tracker.

For this particular case I am confused why it even downloaded CUDA. If I install DynamicPolynomials, CUDA is not installed as a dependency.

vchuravy@odin ~> julia +beta
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-beta2 (2024-05-29)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> activate --temp
  Activating new project at `/tmp/jl_wDJryZ`

(jl_wDJryZ) pkg> add DynamicPolynomials
   Resolving package versions...
   Installed MultivariatePolynomials ─ v0.5.6
   Installed DynamicPolynomials ────── v0.6.0
   Installed MutableArithmetics ────── v1.4.5
    Updating `/tmp/jl_wDJryZ/Project.toml`
  [7c1d4256] + DynamicPolynomials v0.6.0
    Updating `/tmp/jl_wDJryZ/Manifest.toml`
  [d360d2e6] + ChainRulesCore v1.24.0
  [34da2185] + Compat v4.15.0
  [864edb3b] + DataStructures v0.18.20
  [7c1d4256] + DynamicPolynomials v0.6.0
  [102ac46a] + MultivariatePolynomials v0.5.6
  [d8a4904e] + MutableArithmetics v1.4.5
  [bac558e1] + OrderedCollections v1.6.3
  [189a3867] + Reexport v1.2.2
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [9fa8497b] + Future v1.11.0
  [b77e0a4c] + InteractiveUtils v1.11.0
  [8f399da3] + Libdl v1.11.0
  [37e2e46d] + LinearAlgebra v1.11.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [de0858da] + Printf v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization v1.11.0
  [2f01184e] + SparseArrays v1.11.0
  [fa267f1f] + TOML v1.0.3
  [8dfed614] + Test v1.11.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [e66e0078] + CompilerSupportLibraries_jll v1.1.1+0
  [4536629a] + OpenBLAS_jll v0.3.27+1
  [bea87d4a] + SuiteSparse_jll v7.6.0+0
  [8e850b90] + libblastrampoline_jll v5.8.0+1
Precompiling project...
  3 dependencies successfully precompiled in 7 seconds. 22 already precompiled.

(jl_wDJryZ) pkg> why CUDA
ERROR: The following package names could not be resolved:
 * CUDA (not found in project or manifest)
   Suggestions:

To your larger point we are aware that heavy dependencies like CUDA can be problemenatic and the ecosystem is slowly transitioning to package extensions so that only if you explicitly install CUDA, you will have to download it. This transition is ongoing and not yet complete since it requires every package author to decide how they want to handle the dependency.