Modernizes installs' output (#5509)

**What's the problem this PR addresses?**

I wasn't entirely happy with the Yarn output:

- When running `yarn add`, we have no way to know what packages are
actually added to the lockfile. The cache messages sometimes help, but
with zero-installs now being opt-in, in many cases the packages would
already be in the cache and thus wouldn't be displayed at all.

- The "can't be found in the cache and will be fetched from the remote
registry" messages were incredibly slow to print - the same install
would take 28s install for Gatsby on a local iTerm 2 without those logs,
vs almost 2 minutes with. They also weren't very useful: we were only
showing the last 5 of them, and with zero-installs being opt-in in many
cases they wouldn't be shown at all.

- The `node-gyp` warnings were for the most part unactionable (at best
the user would pin a fixed version in their `packageExtensions` field,
which I suspect no one ever did).

- The `deprecated` warnings were for the most part unactionable, and
only printed in very specific cases (the first time they were added to
the project).

- The "missing / invalid peer dependency" warnings were actionable, but
in practice no one really look at them except when something breaks -
and even then, it becomes unreadable when there are too many of them.

- The skipped build warnings were printed every single time you ran
`yarn install`. It's nice to know the first time, then it quickly
becomes redundant.

Fixes #4165

**How did you fix it?**

I went a little overboard and did everything in the same PR ... at first
I thought I wouldn't have to change unrelated parts, but then I finished
implementing the skipped build warnings duplicate removal and oh no 🙈

- Only peer dependency warnings caused by workspaces are now reporter.
They have also been moved inside the post-resolution validation step.
The resolution step is now expected to only do one of two things: either
report an error when the resolution fails, or report the packages which
got added / removed from the lockfile.

- The `node-gyp` warnings have been removed.

- The `deprecated` warnings have been removed from the install. The
`yarn npm audit` command now reports deprecated packages, although this
can be disabled using `--no-deprecations` (or any of the audit filtering
settings).

- The "can't be found in the cache" messages have been removed. Instead,
the fetch step will report the number of cache hits / cache misses once
it's finished (same behaviour as `preferAggregateCacheInfo`). The size
delta will also be reported.

- Packages whose build was skipped are now stored within
`Project#skippedBuilds`, which is stored within the install state file.
Warnings are only emitted if the install was skipped for the first time.
To see the messages again, users can run `yarn rebuild`.

- Added the Yarn version on installs. A bit because of branding when
screenshot are taken, but mostly so we easily know what version are
people using when they share screenshots to us. In a follow-up PR I'd
like to try to retrieve the latest version from the website, to let them
know once one is available.

**Checklist**
<!--- Don't worry if you miss something, chores are automatically
tested. -->
<!--- This checklist exists to help you remember doing the chores when
you submit a PR. -->
<!--- Put an `x` in all the boxes that apply. -->
- [x] I have read the [Contributing
Guide](https://yarnpkg.com/advanced/contributing).

<!-- See
https://yarnpkg.com/advanced/contributing#preparing-your-pr-to-be-released
for more details. -->
<!-- Check with `yarn version check` and fix with `yarn version check
-i` -->
- [x] I have set the packages that need to be released for my changes to
be effective.

<!-- The "Testing chores" workflow validates that your PR follows our
guidelines. -->
<!-- If it doesn't pass, click on it to see details as to what your PR
might be missing. -->
- [x] I will check that all automated PR checks pass before the PR gets
reviewed.

Author: arcanis

.yarn/versions/90cd4b88.yml

@@ -0,0 +1,34 @@
+releases:
+  "@yarnpkg/builder": major
+  "@yarnpkg/cli": major
+  "@yarnpkg/core": major
+  "@yarnpkg/plugin-essentials": major
+  "@yarnpkg/plugin-interactive-tools": major
+  "@yarnpkg/plugin-nm": major
+  "@yarnpkg/plugin-npm": major
+  "@yarnpkg/plugin-npm-cli": major
+  "@yarnpkg/plugin-pnp": major
+  "@yarnpkg/plugin-pnpm": major
+  "@yarnpkg/plugin-version": major
+  "@yarnpkg/plugin-workspace-tools": major
+
+declined:
+  - "@yarnpkg/plugin-compat"
+  - "@yarnpkg/plugin-constraints"
+  - "@yarnpkg/plugin-dlx"
+  - "@yarnpkg/plugin-exec"
+  - "@yarnpkg/plugin-file"
+  - "@yarnpkg/plugin-git"
+  - "@yarnpkg/plugin-github"
+  - "@yarnpkg/plugin-http"
+  - "@yarnpkg/plugin-init"
+  - "@yarnpkg/plugin-link"
+  - "@yarnpkg/plugin-pack"
+  - "@yarnpkg/plugin-patch"
+  - "@yarnpkg/plugin-stage"
+  - "@yarnpkg/plugin-typescript"
+  - "@yarnpkg/doctor"
+  - "@yarnpkg/extensions"
+  - "@yarnpkg/nm"
+  - "@yarnpkg/pnpify"
+  - "@yarnpkg/sdks"

CHANGELOG.md

@@ -10,30 +10,39 @@ Yarn now accepts sponsorships! Please give a look at our [OpenCollective](https:
 
 ### **Major Changes**
 
-- The `yarn set version` command will now skip generating the `yarnPath` configuration on new projects if it detects you're using [Corepack](https://nodejs.org/api/corepack.html)
+- Some important defaults have changed:
+  - `yarn set version` will now skip generating the `yarnPath` configuration on new projects if it detects you're using [Corepack](https://nodejs.org/api/corepack.html).
+  - `yarn init` will no longer use zero-install by default.
 - All official Yarn plugins are now included by default in the bundle we provide. You no longer need to run `yarn plugin import` for *official* plugins (you still need to do it for third-party plugins, of course).
   - This doesn't change anything to the plugin API we provide, which will keep being maintained (Yarn still has a modular architecture and uses the exact same APIs as contrib plugins; all that changes is how we distribute our own features).
+- Yarn's UI during installs has been greatly improved:
+  - Packages added and removed from the lockfile are now explicitly called out.
+  - Changes to the project cache are now reported as a single line.
+  - Unactionable warnings (`node-gyp` and transitive peer dependency errors) have been removed.
+  - Skipped builds are now reported during the first installs only (and subsequent `yarn rebuild` calls).
+  - The Yarn version is now displayed (to help us investigate issues [when reported as screenshots](https://discord.com/channels/226791405589233664/749942897071357954/935110539313565746)).
+  - Deprecations have been moved to `yarn npm audit`.
+- Some settings were renamed or removed:
+  - `caFilePath` is now `httpsCaFilePath`
+  - `preferAggregateCacheInfo` has been removed (it's now always on)
+  - `pnpDataPath` has been removed to adhere to our new [PnP specification](https://yarnpkg.com/advanced/pnp-spec). For consistency, all PnP files will now be hardcoded to a single value so that third-party tools can implement the PnP specification without relying on the Yarn configuration.
+- The `yarn npm audit` command has been reimplemented:
+  - A registry providing the `/-/npm/v1/security/advisories/bulk` endpoint is now expected. Make sure your registry supports it, and let them know you need it otherwise (in the meantime you can use `npmAuditRegistry` to send your audit queries to the npm registry).
+  - Deprecations are now returned by default. To silence them, use `--no-deprecations`.
 - Some legacy layers have been sunset:
   - Plugins cannot access the Clipanion 2 APIs anymore (upgrade to [Clipanion 3](https://github.com/arcanis/clipanion))
   - Plugins cannot access the internal copy of Yup anymore (use [Typanion](https://github.com/arcanis/typanion) instead)
-- The network settings configuration option has been renamed from `caFilePath` to `httpsCaFilePath`.
-- `yarn workspaces foreach` now automatically enables the `-v,--verbose` flag in interactive terminal environments.
-- `yarn npm audit` no longer takes into account publish registries. Use [`npmAuditRegistry`](https://yarnpkg.com/configuration/yarnrc#npmAuditRegistry) instead.
-- `yarn npm audit` has been fully rewritten to use the newer `/-/npm/v1/security/advisories/bulk` endpoint. Make sure your registry supports it, and let them know you need it otherwise (in the meantime you can use `npmAuditRegistry` to send your audit queries to the npm registry).
-- The `--assume-fresh-project` flag of `yarn init` has been removed. Should only affect people initializing Yarn 4+ projects using a Yarn 2 binary.
-- `yarn init` no longer enables zero-installs by default.
-- Yarn will no longer remove the old Yarn 2.x `.pnp.js` file when migrating.
-- The `pnpDataPath` option has been removed to adhere to our new [PnP specification](https://yarnpkg.com/advanced/pnp-spec). For consistency, all PnP files will now be hardcoded to a single value so that third-party tools can implement the PnP specification without relying on the Yarn configuration.
-- The `ZipFS` and `ZipOpenFS` classes have been moved from `@yarnpkg/fslib` to `@yarnpkg/libzip`. They no longer need or accept the `libzip` parameter.
-- Yarn now assumes that the `fs.lutimes` bindings are always available (which is true for all supported Node versions).
-- Some libzip bindings are no longer needed for `ZipFS` and have been removed:
-  - `open`
-  - `ZIP_CREATE` & `ZIP_TRUNCATE`
+  - Yarn will no longer remove the old Yarn 2.x `.pnp.js` file when migrating.
+  - The `--assume-fresh-project` flag of `yarn init` has been removed.
 
 ### **API Changes**
 
 The following changes only affect people writing Yarn plugins:
 
+- The `ZipFS` and `ZipOpenFS` classes have been moved from `@yarnpkg/fslib` to `@yarnpkg/libzip`. They no longer need or accept the `libzip` parameter.
+
+  - Reading the zip archives is now done on the Node.js side for performance; as a result, the `open`, `ZIP_CREATE`, and `ZIP_TRUNCATE` bindings are no longer needed for `ZipFS` and have also been removed.
+
 - The `dependencies` field sent returned by `Resolver#resolve` must now be the result of a `Configuration#normalizeDependencyMap` call. This change is prompted by a refactoring of how default protocols (ie `npm:`) are injected into descriptors. The previous implementation caused various descriptors to never be normalized, which made it difficult to know what were the descriptors each function should expect.
 
   - Similarly, the descriptors returned by `Resolve#getResolutionDependencies` are now expected to be the result of `Configuration#normalizeDependency` calls.
@@ -66,11 +75,21 @@ The following changes only affect people writing Yarn plugins:
 
 - `workspace.dependencies` has been removed. Use `workspace.anchoredPackage.dependencies` instead.
 
+- The `Installer` class must now return `BuildRequest` structures instead of `BuildDirective[]`. This lets you mark that the build must be skipped, and the reason why.
+
+- `startCacheReport` has been removed, and is now part of the output generated by `fetchEverything`.
+
+- `forgettableNames` & `forgettableBufferSize` have been removed (the only messages using them have been removed, making the forgettable logs implementation obsolete).
+
 ### Installs
 
 - The `pnpm` linker avoids creating symlinks that lead to loops on the file system, by moving them higher up in the directory structure.
 - The `pnpm` linker no longer reports duplicate "incompatible virtual" warnings.
 
+### Features
+
+- `yarn workspaces foreach` now automatically enables the `-v,--verbose` flag in interactive terminal environments.
+
 ### Bugfixes
 
 - `yarn dlx` will no longer report false-positive `UNUSED_PACKAGE_EXTENSION` warnings

packages/acceptance-tests/pkg-tests-core/sources/utils/exec.ts

@@ -52,3 +52,13 @@ export const execFile = (
     });
   });
 };
+
+export const execGitInit = async (
+  options: Options,
+) => {
+  await execFile(`git`, [`init`], options);
+  await execFile(`git`, [`config`, `user.email`, `[email protected]`], options);
+  await execFile(`git`, [`config`, `user.name`, `Your Name`], options);
+  await execFile(`git`, [`config`, `commit.gpgSign`, `false`], options);
+  await execFile(`git`, [`config`, `core.hooksPath`, `no-hooks`], options);
+};

packages/acceptance-tests/pkg-tests-core/sources/utils/makeTemporaryEnv.ts

@@ -49,7 +49,6 @@ const mte = generatePkgDriver({
         [`FORCE_COLOR`]: `0`,
         // Otherwise the output wouldn't be the same on CI vs non-CI
         [`YARN_ENABLE_INLINE_BUILDS`]: `false`,
-        [`YARN_PREFER_AGGREGATE_CACHE_INFO`]: `false`,
         // Otherwise we would more often test the fallback rather than the real logic
         [`YARN_PNP_FALLBACK_MODE`]: `none`,
         // Otherwise tests fail on systems where this is globally set to true

packages/acceptance-tests/pkg-tests-specs/sources/commands/__snapshots__/add.test.ts.snap

@@ -4,15 +4,17 @@ exports[`Commands add it should clean the cache when cache lives inside the proj
 {
   "code": 0,
   "stderr": "",
-  "stdout": "➤ YN0000: ┌ Resolution step
+  "stdout": "➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + no-deps@npm:2.0.0
+➤ YN0085: │ - no-deps@npm:1.0.0
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Fetch step
-➤ YN0013: │ no-deps@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
-➤ YN0019: │ no-deps-npm-1.0.0-cf533b267a-e0e60294c2.zip appears to be unused - removing
+➤ YN0013: │ A package was added to the cache, and one was removed.
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0000: └ Completed
-➤ YN0000: Done
+➤ YN0000: · Done
 ",
 }
 `;
@@ -21,14 +23,17 @@ exports[`Commands add it should not clean the cache when cache lives inside the
 {
   "code": 0,
   "stderr": "",
-  "stdout": "➤ YN0000: ┌ Resolution step
+  "stdout": "➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + no-deps@npm:2.0.0
+➤ YN0085: │ - no-deps@npm:1.0.0
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Fetch step
-➤ YN0013: │ no-deps@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
+➤ YN0013: │ A package was added to the cache (+ 0.68 KiB).
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0000: └ Completed
-➤ YN0000: Done
+➤ YN0000: · Done
 ",
 }
 `;

packages/acceptance-tests/pkg-tests-specs/sources/commands/__snapshots__/install.test.js.snap

@@ -4,7 +4,8 @@ exports[`Commands install displays validation issues of nested workspaces 1`] =
 {
   "code": 0,
   "stderr": "",
-  "stdout": "➤ YN0000: ┌ Project validation
+  "stdout": "➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Project validation
 ➤ YN0057: │ package-a-ddd35d: String bin field, but no attached package name
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Resolution step
@@ -13,7 +14,7 @@ exports[`Commands install displays validation issues of nested workspaces 1`] =
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0000: └ Completed
-➤ YN0000: Done with warnings
+➤ YN0000: · Done with warnings
 ",
 }
 `;
@@ -22,24 +23,28 @@ exports[`Commands install it should fetch only required packages when using \`--
 {
   "code": 0,
   "stderr": "",
-  "stdout": "➤ YN0000: ┌ Resolution step
+  "stdout": "➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + no-deps@npm:2.0.0
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Fetch step
-➤ YN0013: │ no-deps@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
+➤ YN0013: │ A package was added to the cache (+ 0.68 KiB).
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0073: │ Skipped due to mode=update-lockfile
 ➤ YN0000: └ Completed
-➤ YN0000: Done with warnings
+➤ YN0000: · Done with warnings
 ",
 }
 `;
 
 exports[`Commands install it should print the logs to the standard output when using --inline-builds 1`] = `
-"➤ YN0000: ┌ Resolution step
+"➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + no-deps-scripted@npm:1.0.0
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Fetch step
-➤ YN0013: │ no-deps-scripted@npm:1.0.0 can't be found in the cache and will be fetched from the remote registry
+➤ YN0013: │ A package was added to the cache (+ 2.07 KiB).
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0007: │ no-deps-scripted@npm:1.0.0 must be built because it never has been before or the last one failed
@@ -50,27 +55,30 @@ exports[`Commands install it should print the logs to the standard output when u
 ➤ YN0000: │ no-deps-scripted@npm:1.0.0 STDOUT postinstall out
 ➤ YN0000: │ no-deps-scripted@npm:1.0.0 STDERR postinstall err
 ➤ YN0000: └ Completed
-➤ YN0000: Done
+➤ YN0000: · Done
 "
 `;
 
 exports[`Commands install it should skip build scripts when using --mode=skip-build 1`] = `
-"➤ YN0000: ┌ Resolution step
+"➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + no-deps-scripted@npm:1.0.0
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Fetch step
-➤ YN0013: │ no-deps-scripted@npm:1.0.0 can't be found in the cache and will be fetched from the remote registry
+➤ YN0013: │ A package was added to the cache (+ 2.07 KiB).
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0000: └ Completed
-➤ YN0000: Done
+➤ YN0000: · Done
 "
 `;
 
 exports[`Commands install reports warning if published binary field is a path but no package name is set 1`] = `
 {
   "code": 0,
   "stderr": "",
-  "stdout": "➤ YN0000: ┌ Project validation
+  "stdout": "➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Project validation
 ➤ YN0057: │ root-workspace-0b6124: String bin field, but no attached package name
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Resolution step
@@ -79,7 +87,7 @@ exports[`Commands install reports warning if published binary field is a path bu
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0000: └ Completed
-➤ YN0000: Done with warnings
+➤ YN0000: · Done with warnings
 ",
 }
 `;
@@ -88,15 +96,17 @@ exports[`Commands install should not build virtual workspaces 1`] = `
 {
   "code": 0,
   "stderr": "",
-  "stdout": "➤ YN0000: ┌ Resolution step
+  "stdout": "➤ YN0000: · Yarn 0.0.0
+➤ YN0000: ┌ Resolution step
+➤ YN0085: │ + no-deps@npm:2.0.0
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Fetch step
-➤ YN0013: │ no-deps@npm:2.0.0 can't be found in the cache and will be fetched from the remote registry
+➤ YN0013: │ A package was added to the cache (+ 0.68 KiB).
 ➤ YN0000: └ Completed
 ➤ YN0000: ┌ Link step
 ➤ YN0007: │ foo@workspace:workspace must be built because it never has been before or the last one failed
 ➤ YN0000: └ Completed
-➤ YN0000: Done
+➤ YN0000: · Done
 ",
 }
 `;

packages/acceptance-tests/pkg-tests-specs/sources/commands/_entry.test.js renamed to packages/acceptance-tests/pkg-tests-specs/sources/commands/_entry.test.ts

@@ -1,20 +1,20 @@
-const semver = require(`semver`);
+export {};
 
 describe(`Entry`, () => {
   describe(`version option`, () => {
     test(
       `it should print the version from the package.json when given --version`,
       makeTemporaryEnv({}, async ({path, run, source}) => {
         const {stdout} = await run(`--version`);
-        expect(semver.valid(stdout.trim())).toBeTruthy();
+        expect(stdout.trim()).toEqual(`0.0.0`);
       }),
     );
 
     test(
       `it should print the version from the package.json when given -v`,
       makeTemporaryEnv({}, async ({path, run, source}) => {
         const {stdout} = await run(`-v`);
-        expect(semver.valid(stdout.trim())).toBeTruthy();
+        expect(stdout.trim()).toEqual(`0.0.0`);
       }),
     );
   });

packages/acceptance-tests/pkg-tests-specs/sources/commands/add.test.ts

@@ -630,59 +630,5 @@ describe(`Commands`, () => {
       expect(cacheContent.find(entry => entry.includes(`no-deps-npm-1.0.0`))).toBeDefined();
       expect(cacheContent.find(entry => entry.includes(`no-deps-npm-2.0.0`))).toBeDefined();
     }));
-
-    test(
-      `it shouldn't show deprecation warnings for non-deprecated packages`,
-      makeTemporaryEnv({}, async ({path, run, source}) => {
-        const {stdout} = await run(`add`, `[email protected]`);
-
-        // Check if the deprecation warning is shown
-        expect(stdout).not.toContain(`no-deps@npm:1.0.0 is deprecated`);
-      }),
-    );
-
-    test(
-      `it shouldn't show deprecation warnings for deprecated packages when the deprecation warning is an empty string`,
-      makeTemporaryEnv({}, async ({path, run, source}) => {
-        const {stdout} = await run(`add`, `[email protected]`);
-
-        // Check if the deprecation warning is shown
-        expect(stdout).not.toContain(`no-deps-deprecated-empty@npm:1.0.0 is deprecated`);
-      }),
-    );
-
-    test(
-      `it should show deprecation warnings for deprecated packages`,
-      makeTemporaryEnv({}, async ({path, run, source}) => {
-        const {stdout} = await run(`add`, `[email protected]`);
-
-        // Check if the deprecation warning is shown
-        expect(stdout).toContain(`no-deps-deprecated@npm:1.0.0 is deprecated: ¯\\_(ツ)_/¯\n`);
-
-        // Check if the package is installed successfully
-        await expect(xfs.readJsonPromise(ppath.join(path, Filename.manifest))).resolves.toMatchObject({
-          dependencies: {
-            [`no-deps-deprecated`]: `1.0.0`,
-          },
-        });
-      }),
-    );
-
-    test(
-      `it should show deprecation warnings for deprecated packages (deprecation warning only includes whitespace)`,
-      makeTemporaryEnv({}, async ({path, run, source}) => {
-        const {stdout} = await run(`add`, `[email protected]`);
-
-        // Check if the deprecation warning is shown
-        expect(stdout).toContain(`no-deps-deprecated-whitespace@npm:1.0.0 is deprecated\n`);
-
-        // Check if the package is installed successfully
-        await expect(xfs.readJsonPromise(ppath.join(path, Filename.manifest))).resolves.toMatchObject({
-          dependencies: {
-            [`no-deps-deprecated-whitespace`]: `1.0.0`,
-          },
-        });
-      }),
-    );
   });
 });