ci: compare yarn vs pnpm (benchmark...)

[belgattitude]

Link to yarn discussion: yarnpkg/berry#4965 (cheksums)
Link to pnpm investigation about size: pnpm/pnpm#5504 (closed)

What does yarn compressionLevel brings on CI ?

Psst, I'm looking to reduce carbon emissions... install is the most common task I have. just curious about the results

• Yarn with cache 100 install/week with compressionLevel=0 saves ♻️ ~53 minutes of CI/month from compressionLevel=mixed (>25% speedup).
• Pnpm would give an extra ♻️ ~60 minutes of CI/month from yarn-cache-no-compress.
• Yarn pnp linker will beat the charts but couldn't make it work (yet). NPM has not support for workspace:^ yet.

CI Scenario	Install	@action/cache	Setup	Total	Projection	Save/mo	Save/mo
yarn4 no-cache-mixed	±113s	no-cache	0s	±113s	±12.5h/mo		reference
yarn4 cache-mixed	±42s	2s (190MB)	0s	±44s	±4.8h/mo	reference	♻️ - 7.5h
yarn4 cache-no-comp	±32s	4s (160MB)	0s	±36s	±4h/mo	♻️ - 0.9h	♻️ - 8.5h
pnpm7 cache	±17s	8s (245MB)	2s	±27s	±3h/mo	♻️ -1.8h	♻️ - 9.5h
pnpm7 no-cache	±61s	no-cache	2s	±61s	±6.6h/mo

Notes:

• The action benchmark. Latest output is here or you can check the history
• projection is made on a basis of 100 install a week or 400/month
• all benchmarks starts with a lock file and an action/cache (pnpm store, yarn cache) setup. All node_modules folder are cleared between benchmarks. yarn runs with node_module linker.
• Note that only yarn (enableGlobalCache:false) is able to efficiently keep the cache across lock file changes.
• As pnpm is not able to prune the cache on lock changes... the cache will grow over time (explicit rotation after 1 month). Yarn cache is more effective and is (almost) always warm (especially on multi-os/arch).
• Pnpm must be setup (2s - 50Mb network), yarn binary lives in the repo (3mb network).
• NPM 8+ is not yet able to install the workspace:^ dependencies.

Notes

About install

On github install times will slightly varies between runs. But here's an output:

About action/cache

When running an install decompression happens all the time. The post restore cache action only happens when the lock file change.

Decompress action/cache

With compressionLevel:0 the zips becomes compressible by zstd leading to better compression ratios. When compressionLevel:1 action/cache will detect the media as uncompressible. The reference pnpm seemss worse here as the cache is bigger, but it's not deduped the same way (see #2799 (comment))

Compress action/cache

Counter-intuitively no significant difference between compressionLevel in term of time. Of course with compressionLevel: 0 the action cache will save space (and network io)

By comparison with PNPM the restore is slower (deduplication is different, see #2799 (comment)) and has more files (symlinks) to deal with.

About Yarn compressionLevel

On multi-core high end cpu, no significant difference. If you want to replicate ci behavior locally, use this hyperfine/taskset command:

Command	Mean [s]	Min [s]	Max [s]	Relative
YARN_COMPRESSION_LEVEL=0 taskset -c 0 yarn install --immutable --mode=skip-build	8.430 ± 0.067	8.361	8.531	1.00
YARN_COMPRESSION_LEVEL=mixed taskset -c 0 yarn install --immutable --mode=skip-build	11.331 ± 0.086	11.249	11.458	1.34 ± 0.01

Hyperfine/Taskset cmd

hyperfine --runs=5 --export-markdown ".yarn/bench-compression-mixed-vs-zero-single-cores.md" \
          --prepare "YARN_COMPRESSION_LEVEL=0 yarn install; npx --yes rimraf '**/node_modules'" \
          "YARN_COMPRESSION_LEVEL=0 taskset -c 0 yarn install --immutable --mode=skip-build" \
          --prepare "YARN_COMPRESSION_LEVEL=mixed yarn install; npx --yes rimraf '**/node_modules'" \
          "YARN_COMPRESSION_LEVEL=mixed taskset -c 0 yarn install --immutable --mode=skip-build" 

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2f8051b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
monorepo-nextjs-app	❌ Failed (Inspect)		Oct 28, 2022 at 1:03PM (UTC)
monorepo-vite-app	✅ Ready (Inspect)	Visit Preview	Oct 28, 2022 at 1:03PM (UTC)

[qlty-cloud-legacy]

Code Climate has analyzed commit 2f8051b and detected 0 issues on this pull request.

View more on Code Climate.

[ghost]

👇 Click on the image for a new way to code review

• Make big changes easier — review code in small groups of related files

• Know where to start — see the whole change at a glance

• Take a code tour — explore the change with an interactive tour

• Make comments and review — all fully sync’ed with github

  Try it now!

Legend

[belgattitude]

Moved to https://github.com/belgattitude/compare-package-managers