feat(coverage): v8 experimental AST-aware remapping

[AriPerkkio]

Description

• Adds support for experimental AST-aware V8 coverage remapping
• Vitest v3 will default to old v8-to-istanbul, but switches to ast-v8-to-istanbul when experimentalAstAwareRemapping: true is set.
• If all goes well, we can remove v8-to-istanbul in Vitest v4 (Enable V8 coverage's experimentalAstAwareRemapping by default and remove old remapping mode #7928)
• Right now the provider.ts is a bit of mess as it supports both remappers. Once v8-to-istanbul can be dropped, it will simplify quite much
• Tests start and run as fast as with previous @vitest/coverage-v8. Report accuracy is as good as with @vitest/coverage-istanbul. Report generation is slightly slower than with previous @vitest/coverage-v8, but faster than overall istanbul provider run.
• Fixes coverage-v8 issues with calculation of branch coverage #6300
• Fixes coverage-v8 reports incorrect coverage when using multiple environments #5783
• Fixes Missing function coverage when transform prepends code right before the function #7130
• Fixes Can we make `istanbul` the default coverage provider over `v8`? #7587

Testing:

• vuejs/core: 100% match with @vitest/coverage-istanbul, except branch coverage is 87.69% 12059/13751 vs 87.46% 12047/13773. There are couple of uncovered function default arguments that are not reported by V8 at all. These are marked as covered.
• vitejs/vite: Close to 100% match with @vitest/coverage-istanbul. Differences coming from V8 limitations, e.g. uncovered default parameters.
• Self-test: 100% match test: self test with vitest-dev/vitest#7736 AriPerkkio/ast-v8-to-istanbul#31

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	4070914
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/68177ab962d1870008ca7181
😎 Deploy Preview	https://deploy-preview-7736--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[pkg-pr-new]

@vitest/browser

npm i https://pkg.pr.new/@vitest/browser@7736

@vitest/coverage-istanbul

npm i https://pkg.pr.new/@vitest/coverage-istanbul@7736

@vitest/expect

npm i https://pkg.pr.new/@vitest/expect@7736

@vitest/coverage-v8

npm i https://pkg.pr.new/@vitest/coverage-v8@7736

@vitest/mocker

npm i https://pkg.pr.new/@vitest/mocker@7736

@vitest/pretty-format

npm i https://pkg.pr.new/@vitest/pretty-format@7736

@vitest/runner

npm i https://pkg.pr.new/@vitest/runner@7736

@vitest/snapshot

npm i https://pkg.pr.new/@vitest/snapshot@7736

@vitest/spy

npm i https://pkg.pr.new/@vitest/spy@7736

@vitest/ui

npm i https://pkg.pr.new/@vitest/ui@7736

@vitest/utils

npm i https://pkg.pr.new/@vitest/utils@7736

vite-node

npm i https://pkg.pr.new/vite-node@7736

vitest

npm i https://pkg.pr.new/vitest@7736

@vitest/web-worker

npm i https://pkg.pr.new/@vitest/web-worker@7736

@vitest/ws-client

npm i https://pkg.pr.new/@vitest/ws-client@7736

commit: 4070914

[AriPerkkio]

@hi-ogawa this approach should provide us an easier way to control how Vite's generated code is excluded from coverage reports. We no longer need to modify source maps - instead we ignore the nodes while remapping coverage.

https://github.com/AriPerkkio/ast-v8-to-istanbul?tab=readme-ov-file#ignoring-generated-code

[hi-ogawa]

I checked ast-v8-to-istanbul too and it looks all good to me. Awesome stuff!

Report generation is slightly slower than with previous @vitest/coverage-v8, but faster than overall istanbul provider run.

I'm curious how you understand the performance characteristic leading to this result.

I think ast-v8-to-istanbul being slower than v8-to-istanbul is probably obvious since it now parses and traverse AST and remap node by node while v8-to-istanbul just goes through v8 coverage once.

What about @vitest/coverage-istanbul? I guess istanbul's instrumentation transform can be slower than ast-v8-to-istanbul's ast manipulation? Also do you take into account instrumented code being running slower? Or maybe other aspect of coverage processing logic?

[AriPerkkio]

I think ast-v8-to-istanbul being slower than v8-to-istanbul is probably obvious since it now parses and traverse AST and remap node by node while v8-to-istanbul just goes through v8 coverage once.

Yes, exactly. It is expected that AST based remapping is slower than the original v8-to-istanbul. The original one is very naive and produces many false positives, but it's fast. It's not expected that ast-v8-to-istanbul would reach the speed of v8-to-istanbul, but I'm hoping we could get very close to that. Both of these tools process a lot of text (JSON, code, source maps) so there's plenty of optimization possibilities. Looking at the API of ast-v8-to-istanbul, I think in future all of this could even run on Rust based tooling like oxc.

What about @vitest/coverage-istanbul? I guess istanbul's instrumentation transform can be slower than ast-v8-to-istanbul's ast manipulation? Also do you take into account instrumented code being running slower? Or maybe other aspect of coverage processing logic?

This speed difference reproduces on vuejs/core repository really well. When using Istanbul's pre-instrumented coverage, it's actually visible for end-users to see that files remain in [queued] filename.test.ts state for a while when transform is running. Also executing this pre-instrumented code is a lot slower - I think there would be good optimization opportunities in istanbul-lib-instrumentation's code transforms. Pre-instrumenting increases transpiled code and combined source maps a lot.

When comparing AST parsing, visiting and analysing of istanbul-lib-instrument and ast-v8-to-istanbul, we should see similar speed results.

I'm curious how you understand the performance characteristic leading to this result.

I've been comparing @vitest/coverage-istanbul, @vitest/coverage-v8 and @vitest/coverage-v8 + ast-v8-to-istanbul in:

• vitejs/vite
• vuejs/core
• ci: set up tests with vitest-dev/vitest#7736 vitest-tests/coverage-large#1
• test: self test with vitest-dev/vitest#7736 AriPerkkio/ast-v8-to-istanbul#31

In these tests I've checked overall speed with time vitest --run --coverage and compared coverage summaries between each run. I'm also planning to add debug timer for coverage generation via DEBUG=vitest:coverage in separate PR at some point.

[sheremet-va]

This looks incredible! Thank you for working on it 🙏🏻