Cache usage meta tracking issue

[ehuss]

This issue is to help provide an overview of the different issues around Cargo's excessive disk usage, and tangentially, reducing compile time by reusing artifacts in a shared cache.

Cleaning outdated artifacts

Cargo's target directory can grow substantially over time. It has limited capabilities to clean it with cargo clean. Also, in general, cargo clean has a fair number of bugs and is generally underwhelming.

Various issues and links of interest:

• cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026 — cargo ./target fills with outdated artifacts as toolchains are updated/changed
• How to effectively clean target folder for CI caching #5885 — How to effectively clean target folder for CI caching
• Have an option to make Cargo attempt to clean up after itself. #6229 — Have an option to make Cargo attempt to clean up after itself.
• cargo update removes uninstalled deps from target/, when detected they have been removed from Cargo.toml #6435 — Remove artifacts for deps removed from Cargo.lock.
• cargo-sweep — A tool to prune unused files.
• The -Z mtime-on-use flag is an experiment to have Cargo update the mtime of used files to make it easier for tools like cargo-sweep to detect which files are stale.

I think a way forward here is to experiment and investigate different ways for tracking artifacts and last-use timestamps. mtime-on-use has an issue with cached files in Docker. The filename hash is opaque and doesn't provide any insight into the metadata which would inform whether or not an artifact could be removed.

Cargo currently tracks a variety of things in different ways. It has a .json fingerprint file which is generally unused (only for debug logging). It also has an invoked.timestamp file used for some change tracking. And mtime information is used in a few different ways. It might be interesting to experiment with a different way to coordinate all this information. Perhaps a single, unified file tracking all artifacts, or changing the way the per-artifact .json file works. The key points is that it must be fast and reliable, and should work well in Docker.

Cleaning cargo's home

Cargo's home directory ~/.cargo grows without bounds. There is currently no built-in way to shrink it.

The cargo-cache package is the foremost way to manage it currently (besides rm -rf). Ideally some of this would be a built-in capability of Cargo.

The main issue tracking this is #3289 — cargo clean ~/.cargo.

There has not been much discussion about this. Ideally cargo would have this capability built in, perhaps with some of the easier/safer tasks automated on a periodic basis.

Reusing shared dependencies

sccache is the primary way to share artifacts across projects. It is also possible to share targets with setting the CARGO_TARGET_DIR environment variable.

Issues:

• Suggestion: re-use built dependencies across directories #4301 — Suggestion: re-use built dependencies across directories
• Cache compilations of everything from crates.io #4436 — Cache compilations of everything from crates.io
• Per-user compiled artifact cache #5931 — Per-user compiled artefact cache

Since this has the potential to use a substantial amount of disk space, it would be desirable to have better support for pruning as listed above.

There are a fairly large number of tools which dig into the target directory. They would all be broken by this change, so we would need to figure out a strategy for migration before doing this. I began this in #6668, but I have not finished. Ideally #6668 and #6577 would be finished before making this change.

[bugproof]

sccache is the primary way to share artifacts across projects

I don't think it works. I used sccache and artifacts are still stored per project instead of globally. Only setting CARGO_TARGET_DIR to something like ../.cargo-build-artifacts actually changed where artifacts are stored.

If you have ever used any C++ package managers like vcpkg and/or conan they also store dependencies globally. I think that behavior should be default for cargo as well. If I have 5 projects using tokio runtime I don't want duplicated tokio artifacts in all of them eating my disk space. That's a major oversight in cargo design.

[bjorn3]

I used sccache and artifacts are still stored per project instead of globally.

Sccache has a global cache, but when actually using a dependency, it is copied to the local target dir. Sccache reduces build time, but doesn't help with disk usage.

If you have ever used any C++ package managers like vcpkg and/or conan they also store dependencies globally.

That design doesn't allow different projects to use different versions or configurations of a dependency. It also requires that multiple compiler versions are abi compatible with each other which is not the case for rustc. See https://cor3ntin.github.io/posts/abi/ for all the pain C++'s stable abi causes. Furthermore it hurts reproducability by introducing global state. With cargo only Cargo.lock could be considered state and this file is per project and meant to be checked in. The target dir is just a cache, unlike with vcpkg and conan. There are tradeoffs between the designs of cargo and vcpkg/conan. Neither is strictly better than the other.

If I have 5 projects using tokio runtime I don't want duplicated tokio artifacts in all of them eating my disk space.

If those are different versions of tokio or different configurations, you have to duplicate them one way or another.

[bugproof]

That design doesn't allow different projects to use different versions or configurations of a dependency.

I think it does. There's just directory per version in a single shared location. If two crates use the same version those artifacts should be re-used. I currently use CARGO_TARGET_DIR to save up some space but don't know if there are any drawbacks to that approach.